Practical Software Measurement

I am pleased to annouce that on Thursday, February 25 at 1:00 PM EST, QSM will be hosting a webinar based on our new High Performance Benchmark Consortium.

QSM has introduced a program specifically designed to help software development or acquisition organizations quantify and demonstrate performance improvement over time. The High Performance Benchmark Consortium is for clients who want to be best in class software producers and are willing to be active participants in the program. In today’s economic environment it is more important than ever for both suppliers and acquirers to compete more effectively and provide value to their customers. Members of the Consortium gain access to proprietary research that leverages the QSM historical benchmark database of over 8,000 validated software projects.

Presented by benchmarking expert and head of QSM Consulting, Joe Madden, this webinar will discuss:

QSM consultant Paul Below has posted some quick performance benchmarking tables for IT, engineering class, and real time software.

The tables contain average values for the following metrics at various size increments:

Schedule (months)

Effort (Person Months)

Average Staff (FTE)

Mean Time to Defect (Days)

SLOC / PM

Two insights that jump out right away:

1. Application complexity is a big productivity driver. IT (Business) software solves relatively straightforward and well understood problems. As algorithmic complexity increases, average duration, effort, team size increase rapidly when compared to IT systems of the same size.

When he created control charts in the 1920’s, Walter Shewhart was concerned with two types of mistakes:

Just before the holidays, we hosted our first in-house webinar, "Managing Productivity with Proper Staffing Strategies." Confronted with challenges presented by the current economy, we see more and more systems development groups trying to do more with less.  The ultimate goal is to maximize productivity and minimize defects, but many teams struggle to get there.  It is possible, but the most effective methods used to achieve maximum efficiency are counter-intuitive.  People always think more effort will produce more product.  The fact is using less effort is often more effective.  Presented by industry expert, John Bryant, this webinar explains and proves the correct way to maximize productivity while at the same time minimizing cost and defects. 

MTTD is Mean Time to Defect.  Basically, it means the average time between defects (mean is the statistical term for average).  A related term is MTTF, or Mean Time to Failure.  It is usually meant as the average time between encountering a defect serious enough to cause the system to fail.

Is MTTD hard to compute?  Does it require difficult metrics collection? Some people I have spoken to think so.  Some texts think so, too.  For example:

Gathering data about time between failures is very expensive.  It requires recording the occurrence time of each software failure.  It is sometimes quite difficult to record the time for all the failures observed during testing or operation.  To be useful, time between failures data also requires a high degree of accuracy.  This is perhaps the reason the MTTF metric is not widely used by commercial developers.

Building on some interesting research performed by QSM's Don Beckett, I take a look at how Brooks' Law stacks up against a sample of large projects from our database:

Does adding staff to a late project only make it later? It’s hard to tell. Large team projects, on the whole, did not take notably longer than average. For small projects the strategy had some benefit, keeping deliveries at or below the industry average, but this advantage disappeared at the 100,000 line of code mark. At best, aggressive staffing may keep a project’s schedule within the normal range of variability.

Managing risk for a single project is a fairly well understood problem. If the project contains multiple releases and they are independent, risk managment remains fairly straightforward:

A software development project is going to proceed concurrently with the development of a new piece of hardware required to implement the software. Scheduled completion dates for both developments have been determined and a project plan has been created. Both projects can proceed independently until their respective completions (probably an unwarranted assumption, but this is a simple example!). Both projects must succeed in order for overall success to be achieved.  

Several weeks ago I read an interesting study on finding bugs in giant software programs:

The efficiency of software development projects is largely determined by the way coders spot and correct errors.

But identifying bugs efficiently can be a tricky business, when the various components of a program can contain millions of lines of code. Now Michele Marchesi from the University of Calgiari and a few pals have come up with a deceptively simple way of efficiently allocating resources to error correction.

Sizing is arguably the most challenging part of any software estimate:

Without a notion of functional size, managers may find it difficult to negotiate realistic schedules based on their demonstrated ability to deliver software. They are unable to show empirically why the twelve person team that worked so well on a 150,000 ESLOC project over six months not only fails to deliver a 75,000 ESLOC project in half the time, but produces an error-ridden product that infuriates the customer. Unlike manufacturing shoes, software development is full of non-linear relationships between size, time, effort, and defects. What data driven estimation does successfully is arm managers with the ability to sanity check their current plans against past performance and negotiate achievable outcomes based on a realistic assessment of how much functionality can be built with a set time frame and resource profile.

Progress, far from consisting in change, depends on retentiveness. When change is absolute ...no direction is set for possible improvement... when experience is not retained, as among savages, infancy is perpetual. Those who cannot remember the past are condemned to repeat it.

- George Santayana, The Life of Reason

The French have a saying: "Plus ca change, plus c’est la meme chose".