Has Software Productivity Declined Over Time?
Posted By Kate Armel on Wed, 2011-03-09 11:59<< Technology Can Only Do So Much | Main | The Housework Theory of Productivity >>
Peter Hill of ISBGS poses an interesting question:
Has software productivity improved over the last 15 years? What do you think? Perhaps it doesn't matter as long as quality (as in defect rate) as improved?
Two widely used productivity measures are Function Points/Person Month and QSM's PI (or productivity index). To answer Peter's question, I took a quick look at 1000 medium and high confidence business systems completed between 1996 and 2011. Here's what I found:
Whether we look at PI or FP/PM, the story's the same - on average, productivity has actually decreased over time. What could be causing this? One possible explanation is the correlation between measured productivity and the volume of delivered functionality. As the next chart shows, regardless of the metric used average productivity increases with project size:
Which led me to wonder: what has happened to average project size over time? Again, regardless of whether the delivered functionality was measured in SLOC or Function Points, the story was the same: projects are getting smaller.
Peter's question is a good example of why we often need more than one metric to interpret the data. More on that topic coming up shortly!
Related Posts
Comments
Productivity Trends
Posted by on Mon, 2011-03-21 09:27The value of a delivered system is the quantity, quality, and usability of the knowledge contained in the system that is made available to the user. The key determinant of project effort and duration however is the knowledge that must be gained by the project team. These are often related, but they're not the same.
Since knowledge is not directly and empirically measurable, we always use a proxy for it.
I think what we are seeing here is the difference between the way modern systems' knowledge content and modern projects' knowledge-to-be-gained vary from the traditional proxy measures. With COTS implementation projects or when using high level languages, much of the knowledge needed is already embedded in the package or language, usually requiring less knowledge-to-be-gained. Such a project is "more productive" from the perspective of the user. But this approach usually also requires less code to be written (which is how most knowledge-to-be-gained is realized in projects), so by the code production proxy we appear less productive. Since less code is needed we should get the job done quicker for less effort, so by our duration and effort proxy measures we will appear more productive.
So we should see a decline in code-proxy productivity and an increase in effort and duration proxy productivity, which we do see. Measured by delivered value, productivity should increase (which is why we use packages and high level languages of course) so value-related proxies such as FP should show an increase. Since they do not, we should ask why.
It is likely that smaller projects is one factor. However, unless the industry is overall delivering less value, which is unlikely, the smaller size of projects should be offset by having more of them. Here we might be seeing some measurement artifact, specifically: smaller projects tend to carry less overhead to manage tasks like metrics collection so we are simply not measuring as many projects.
It might also mean that the FP proxy does not measure where some of the knowledge is going in modern systems. With modern SI projects, much of the work is in designing infrastructure and ensuring scalability. The classical (ISO/IEC 20926) FP approach is that implementation issues such as infrastructure are rolled into the FP counted metrics--basically that implementation exists only to support the user-visible functions and scales up and down with the size and complexity of those functions. This might not be the case now and the proxy measure of FP is not tracking with the actual knowledge needed.
Phil Armour
Software productivity vs. size over time
Posted by on Tue, 2011-03-15 15:26Now that we have seen a correlation between measured productivity and size, it would be interesting to examine the productivity trend over time for projects in the same size range.
Perhaps productivity is the same or even improving over time for projects in the same size bin?
Has Software Productivity Declined Over Time?
Posted by on Thu, 2011-03-10 23:00 Elisabeth, Thank you for sharing the findings based on the QSM data and for introducing the additional data on project size.
The ISBSG data shows that productivity (measured as hours to deliver a Functional Unit of functionality) has not improved over time, but quality (measured as number of defects delivered with a piece of software) has improved. As an aside, your findings that 'average productivity increases with project size' might raise a few eyebrows, but the ISBSG data produces the same finding. Of course we only have data on projects that were completed. I wonder about the incidence of abandonment of large projects.
www.isbsg.org
re: Simpson's Paradox
Posted by Kate Armel on Wed, 2011-03-09 13:14That's a great point (and something we often advise our SLIM-Metrics clients to keep in mind): wherever possible, try to make apples to apples comparisons.
Interestingly enough, this morning I did stratify the projects into small (under 20K) medium (20-40 K) and large (40-60K) projects, the idea being to compare relatively small projects to each other.
The trend over time was less pronounced but it still wasn't upward sloping. Not sure what would have happened if I'd looked at really enormous projects. When you get to the larger end of the size spectrum the sample sizes begin to affect your results!
Simpson's Paradox
Posted by on Wed, 2011-03-09 12:44In statistics, the term "Simpson's Paradox" is used to describe a situation where an trend or relationship visible in one group of data is reversed when the data is broken down into categories.
I think this is what we are seeing here, when you look at productivity versus time you see one thing, but when you break it into ranges of project size you will likely see something different.
I think one takeaway is that, to understand, we need to examine more than one metric.
Productivity
Posted by Kate Armel on Wed, 2011-03-09 12:40I think that's exactly what has been going on. Years ago I noticed that the projects making their way into the QSM database were getting smaller. At the time, I theorized that companies were learning that for the behemoth projects of the 1990s, project size was a significant risk and complexity factor. We saw a lot of projects fail during that time period and those failures were costly - and painful!
This made me wonder: are the actual projects (programs) getting smaller? Or have we just learned to eat the elephant one bite at a time (in other words, dividing large, complex tasks into smaller, more manageable pieces).
This issue raises an interesting point about software metrics - simple numbers aren't terribly good at providing context. That's a point I plan to explore further in my next post.
Productivity Trends
Posted by on Wed, 2011-03-09 12:26This dove tails nicely with some of the things we have been seeing with the increasing use of incremental, interative/agile methods. One of the advantages of these methods is to get to market faster which is a benefit. To your point we need more than metrics to put these trends in context.
