Ask Carol: No Free Lunch in Software Estimation and Benchmarking

Dear Carol: 

Given all your international experience, I’m hoping you can tell me where I can find a large, freely available industry database that project managers could use for software estimation and/or benchmarking.  After 5 decades of software development wouldn’t you think that we could put together a software estimation or benchmarking database that the world could use for free? 

- Hopeful in Hartford

Dear Hopeful:  

Great question — and the dream of many IT project managers.  It might seem like an easy concept (just collect actual effort and project size and use it for future estimates); in practice it’s not that simple.

What I know is that in software estimation and benchmarking, there is no free lunch -- you get what you pay for.  And I’ll explain why…

Certainly there are data — lots and lots of project data — but to make it into useful estimating and benchmarking “information” doesn’t just happen.  A solid estimating tool suite such as QSM’s SLIM is supported by a team of pros who make sure that the data, models and equations are consistent, reliable, and high quality.  Even the not-for-profit International Software Benchmarking Standards Group’s (ISBSG) Application Development and Enhancement database is supported by a team of PhD’s in Melbourne, Australia who scrub data submissions before they are added to the database.  And the investment in tools like SLIM are well worth it — if you’re not relying on “good, solid” data, you are better off guessing based on your own experience.

This is what the professionals take care of and it’s an investment in high quality, consistent data.  It might seem simple enough to say “all I need is effort, size and duration of a past project to predict a future one,” but reliable estimates and quality benchmarks take much more thought.

Let’s consider the fundamental issue: what is a “project”?   While the term is defined consistently in theory in the PMBOK^® (Project Management Body of Knowledge) as "a temporary endeavor resulting in a unique product or service," in practice, data collected about a “project” and what they mean are seldom consistent.  Consider the following “contextual” variables that one needs to make sure that one project is similar to another:

1. Project type: New software, enhancement, R&D, hardware selection, maintenance, help desk development, etc.
2. Subject area:  Avionics, banking, insurance, manufacturing, government, medical office, hospital billing, warehousing, reporting, etc.
3. Phases covered:  What did/does the project include?  If it is software development/enhancement, does it include the five major life cycle phases (agile vs waterfall vs spiral all define a project or release similarly in that “it” delivers a piece of working software) — requirements through to installation of the first site?
4. Project scope:  How big was/is the project using consistent units or convertible units of measure (e.g., function points, non-comment source lines of code, use case points, even t-shirt sizing).
5. Development language:  This is akin to using “power tools” in construction.  Dot net and java are considerably more productive (and take less time) than are COBOL or a myriad of other 3gl and 4gl technologies.  The language level of the development (or hybrid level) is important for estimation and comparison.
6. Project hours:  While every hour is sixty minutes and every minute is sixty seconds, what is included in the hours varies widely for projects.  Is overtime included/excluded, whose hours are included (FTE, contractors, PM hours, etc.) are often overlooked as consistency factors.
7. Costs: What is included in the cost of a project?  Hardware, software, labor, management, overhead, market research?  You need to know what is included before you can do projections or comparisons.
8. Many other factors such as development approach (agile vs waterfall vs spiral vs hybrid), platform (more hardware to synchronize takes more effort), constraints (is the project constrained to be done within a particular time frame, is there a preset budget, does the software need to adhere to six sigma or other quality requirements).

Certainly it would be nice to have a consistent, reliable, and solid industry database available at no charge (complete with definitions and explanations for any anomalies in the data) that project managers could use, but it just isn’t practical.  Estimating and benchmarking are much more than trivial exercises, and like everything else in life, if you don’t have the time to do them right, when will you have time to do them over?  An investment in good toolsets pays itself off quickly — and unfortunately, Hopeful, there is no free lunch in benchmarking (or in life.)

- Carol

QSM hosts a free advice column for software professionals who seek help to solve project management, communication and general software project issues. Carol Dekkers is a QSM consultant and IT measurement and project management expert who speaks internationally on topics related to software development. Send your questions to Ask Carol!