Software project benchmarking and estimating leverages the power of historical project data to do solid project estimates, yet the concepts behind such processes are often not well understood.  Benchmarking and estimating rely on productivity comparisons with completed (actual) projects in a historical database and on parametric equations that mimic real life.  I find that technical concepts such as software estimation or benchmarking often can be explained by using analogies that work in other industries.  As I was thinking about benchmarking and estimating this week, the popular children’s book, Dr. Seuss's Green Eggs and Ham, came to mind.

I was talking about data mining, benchmarking, and the SLIM Suite of software estimating tools with QSM’s research director, Kate Armel. It seems that many project estimators believe that creating microscopic slices of project data is the key to precision in estimating and benchmarking, when, in reality, bigger chunks of data take less time to assemble and provide greater value.  Projects are never exact duplicates of each other, however, there are valuable trends and patterns that come out of a few common characteristics.

How is baseball analysis like software project management?  One way is the ability to continually update estimates and forecasts, as the situation and our knowledge change.  As Larry Putnam Jr recently wrote, “project estimation should continue throughout the entire project lifecycle”. 

Walter Shewhart, the father of Statistical Process Control, explained it like this:

“…since we can make operationally verifiable predictions only in terms of future observations, it follows that with the acquisition of new data, not only may the magnitudes involved in any prediction change, but also our grounds for belief in it.”

Here is a baseball example that should appear very familiar to software estimators who are familiar with the often quoted cone of uncertainty.  The following graph is taken from Curve Ball: Baseball, Statistics, and the Role of Chance in the Game, by Jim Albert and Jay Bennett.

The above model is based upon only a few simple items:  The number of homeruns hit so far; the number of games played so far and number remaining; and the total number of games in a season.  We could try to improve the model, especially early in the season, by incorporating more information.  For example:  

A common misperception is that an estimator’s job is done after a software project’s parameters are set. On the contrary, software estimation should be conducted throughout the project lifecycle to reflect inevitable changes and to improve estimates on other projects. In this article, originally published in Projects at Work, Larry Putnam Jr. identifies three ways to maximize estimating efforts — before, during and after your project is complete.

Read the full article!

Since I work for a software metrics and estimation company, many people ask me questions regarding capacity planning and demand management. Most of the project managers that I speak with are using project portfolio management tools for very detailed, task-level resource capacity planning. They spend a lot of time planning the person hours for each task and then these task-level plans are prioritized and viewed across the organization. These are useful tools and methods and they usually require a sizable investment.

The problem is that many of these project managers don’t have a good way to support demand management. That is to say that they aren’t able to accurately estimate the key drivers that go into their PPM tools.  They need to be able to answer key questions like: Should we commit to 6 months or 9 for the project duration? Do we need 10 software developers or 20 to finish in 9 months? How much is the overall project going to cost? What alternatives do we have? Has anyone ever achieved that duration in the past? Should we even go forward with this project; what is the risk?

Oftentimes the project manager comes up with this information informally based on experience. Unfortunately, when they don’t use a scientific approach to estimating, they leave out key factors that affect the estimate and project success, like project complexity, team efficiency, and overall project uncertainty.

Client organizations who are considering investing in SLIM^® (a top-down, scope-based, parametric software estimation tool) often ask us for return on investment (ROI) case study examples which we gladly provide to help them with their business case. However, one question that has never been asked but I have always wondered is: does ROI accelerate with increased investment or does it follow the law of diminishing returns?

To answer this question, we looked at seven software estimation ROI case studies that included a variety of small, medium and large clients, from a single seat of SLIM all the way up to an enterprise Estimation Center of Excellence (ECoE).

On the above chart we plotted the investment in SLIM tools and consulting on the X axis vs. the return (actual cost savings) on the Y axis for each case study using a logarithmic scale. We then drew a trend line through the data points.

Following the trend line, small engagements (~$30K) had an average ROI of more than 13:1.  Medium engagements (~$300K) had an average ROI of 33:1.  Large ECoE engagements ($3M) had an average ROI of 67:1.  So not only is the ROI compelling, but it also accelerates with increased investment.

The actual cost savings (return) as reported by, or observed while working with, our clients include:

I am a professional software project estimator.  While not blessed with genius, I have put in sufficient time that by Malcolm Gladwell’s 10,000 hour rule, I have paid my dues to be competent.  In the past 19 years I have estimated over 2,000 different software projects.  For many of these, the critical inputs were provided and all I had to do was the modeling.  Other times, I did all of the leg work, too: estimating software size, determining critical constraints, and gathering organizational history to benchmark against.  One observation I have taken from years of experience is that there is overwhelming pressure to be more precise with our estimates than can be supported by the data.  

In 2010 I attended a software conference in Brasil.  As an exercise, the participants were asked to estimate the numerical ranges into which 10 items would fall.  The items were such disparate things as the length of coastline in the United States, the gross domestic product of Germany, and the square kilometers in the country of Mali: not things a trivia expert would be likely to know off hand.  Of 150 participants, one person made all of the ranges wide enough.  One other person (me) got 9 out of 10 correct.

The laws of flight have many commonalities with the “physics” of software development. Whereas the principles of aerodynamics are essentially about thrust, drag, lift and weight, developing software has always been about the relationship between size (scope), effort (cost), duration (schedule) and the often overlooked measurement of quality (reliability).  

In aviation parlance the expression “flying by the seat of your pants” literally meant that you felt the plane through your “seat”. Early aircraft had limited navigational aids, mainly a rudimentary compass (which worked only when flying level) and a simple string to assess airflow relative to the plane’s fuselage. Pilots could determine the degree of an ascent or descent by G-force sensation, or assess airspeed by the severity of the aircraft’s vibrations. Until the invention of the gyroscope it was quite dangerous to fly without a virtual horizon – the centerpiece of a modern dashboard.

Early aeronauts plotted their routes using individual skill, celestial or fixed landmarks and their own real time perceptions rather than depending on mechanical tools. However, fog or low visibility would render the limited instruments they had useless, with the flight itself remaining as a risky endeavor. A successful trip meant you landed at your destination in one piece, but it was largely dependent upon the talent and judgment of the aviator, visibility and weather, and perhaps no small amount of luck.    

Dear Carol: 

I’m a developer in our IT department and we know that project estimating is a big deal for our customers.  Somehow, no matter what we do, we can't seem to get it right.  We do know that project size is an important input to good estimating  and our gut feel is that if we get sizing right, we’ll do better estimates!  I know you recommend using function points, but I’ve also been reading a lot about use case points, story points, SLOC, sizing by analogy, T-shirt sizing, COSMIC and other sizing metrics.  We do a mix of waterfall, agile, iterative and even Kanban to do our projects so what’s the best choice for sizing to get the best results? 

- Size Challenged in Milwaukee

Dear Size Challenged:    

Sometimes I wonder if the internet and the proliferation of (mis)information is a good thing. Before the internet, our choices (for sizing or estimating or anything) were limited and we didn’t have such an overwhelming task to first sift through many options before taking action.  Your list of software sizing choices is an example of this.