Practical Software Measurement

It’s time to update QSM’s industry trends and we need your help! Contributing data ensures that the database reflects a wide spectrum of project types, languages, and development methods. It helps us conduct ground-breaking research and improve our suite of estimation, tracking, and benchmarking tools. Contributors benefit from the ability to sanity-check estimates, ongoing projects, and completed projects against the best industry trends in the business.

 We're validating over 400 new projects, but we can always use more – especially in the Real Time, Microcode, and Process Control application domains. So what do you need to do to ensure your firm is represented in the next trend line update? That’s easy! Simply send us your DataManager (.smp files) or completed SLIM-Control (.scw) workbooks. Here’s the recommended minimum data set:

Our recent webinar, "Introduction to the High Performance Benchmark Consortium," was a great success and we are already looking forward to planning our next presentation.  Joe Madden received a lot of insightful questions regarding our new consulting program.  We are aware that your time is valuable and scheduling can be a challenge, so we have recorded a replay, including Q&A, for anyone who was unable to attend the scheduled webinar.

I am pleased to annouce that on Thursday, February 25 at 1:00 PM EST, QSM will be hosting a webinar based on our new High Performance Benchmark Consortium.

QSM has introduced a program specifically designed to help software development or acquisition organizations quantify and demonstrate performance improvement over time. The High Performance Benchmark Consortium is for clients who want to be best in class software producers and are willing to be active participants in the program. In today’s economic environment it is more important than ever for both suppliers and acquirers to compete more effectively and provide value to their customers. Members of the Consortium gain access to proprietary research that leverages the QSM historical benchmark database of over 8,000 validated software projects.

Presented by benchmarking expert and head of QSM Consulting, Joe Madden, this webinar will discuss:

QSM consultant Paul Below has posted some quick performance benchmarking tables for IT, engineering class, and real time software.

The tables contain average values for the following metrics at various size increments:

Schedule (months)

Effort (Person Months)

Average Staff (FTE)

Mean Time to Defect (Days)

SLOC / PM

Two insights that jump out right away:

1. Application complexity is a big productivity driver. IT (Business) software solves relatively straightforward and well understood problems. As algorithmic complexity increases, average duration, effort, team size increase rapidly when compared to IT systems of the same size.

When he created control charts in the 1920’s, Walter Shewhart was concerned with two types of mistakes:

Just before the holidays, we hosted our first in-house webinar, "Managing Productivity with Proper Staffing Strategies." Confronted with challenges presented by the current economy, we see more and more systems development groups trying to do more with less.  The ultimate goal is to maximize productivity and minimize defects, but many teams struggle to get there.  It is possible, but the most effective methods used to achieve maximum efficiency are counter-intuitive.  People always think more effort will produce more product.  The fact is using less effort is often more effective.  Presented by industry expert, John Bryant, this webinar explains and proves the correct way to maximize productivity while at the same time minimizing cost and defects. 

MTTD is Mean Time to Defect.  Basically, it means the average time between defects (mean is the statistical term for average).  A related term is MTTF, or Mean Time to Failure.  It is usually meant as the average time between encountering a defect serious enough to cause the system to fail.

Is MTTD hard to compute?  Does it require difficult metrics collection? Some people I have spoken to think so.  Some texts think so, too.  For example:

Gathering data about time between failures is very expensive.  It requires recording the occurrence time of each software failure.  It is sometimes quite difficult to record the time for all the failures observed during testing or operation.  To be useful, time between failures data also requires a high degree of accuracy.  This is perhaps the reason the MTTF metric is not widely used by commercial developers.

Building on some interesting research performed by QSM's Don Beckett, I take a look at how Brooks' Law stacks up against a sample of large projects from our database:

Does adding staff to a late project only make it later? It’s hard to tell. Large team projects, on the whole, did not take notably longer than average. For small projects the strategy had some benefit, keeping deliveries at or below the industry average, but this advantage disappeared at the 100,000 line of code mark. At best, aggressive staffing may keep a project’s schedule within the normal range of variability.

Managing risk for a single project is a fairly well understood problem. If the project contains multiple releases and they are independent, risk managment remains fairly straightforward:

A software development project is going to proceed concurrently with the development of a new piece of hardware required to implement the software. Scheduled completion dates for both developments have been determined and a project plan has been created. Both projects can proceed independently until their respective completions (probably an unwarranted assumption, but this is a simple example!). Both projects must succeed in order for overall success to be achieved.  

Several weeks ago I read an interesting study on finding bugs in giant software programs:

The efficiency of software development projects is largely determined by the way coders spot and correct errors.

But identifying bugs efficiently can be a tricky business, when the various components of a program can contain millions of lines of code. Now Michele Marchesi from the University of Calgiari and a few pals have come up with a deceptively simple way of efficiently allocating resources to error correction.