Software project benchmarking and estimating leverages the power of historical project data to do solid project estimates, yet the concepts behind such processes are often not well understood.  Benchmarking and estimating rely on productivity comparisons with completed (actual) projects in a historical database and on parametric equations that mimic real life.  I find that technical concepts such as software estimation or benchmarking often can be explained by using analogies that work in other industries.  As I was thinking about benchmarking and estimating this week, the popular children’s book, Dr. Seuss's Green Eggs and Ham, came to mind.

I was talking about data mining, benchmarking, and the SLIM Suite of software estimating tools with QSM’s research director, Kate Armel. It seems that many project estimators believe that creating microscopic slices of project data is the key to precision in estimating and benchmarking, when, in reality, bigger chunks of data take less time to assemble and provide greater value.  Projects are never exact duplicates of each other, however, there are valuable trends and patterns that come out of a few common characteristics.

Dear Carol: 

Given all your international experience, I’m hoping you can tell me where I can find a large, freely available industry database that project managers could use for software estimation and/or benchmarking.  After 5 decades of software development wouldn’t you think that we could put together a software estimation or benchmarking database that the world could use for free? 

- Hopeful in Hartford

Dear Hopeful:  

Great question – and the dream of many IT project managers.  It might seem like an easy concept (just collect actual effort and project size and use it for future estimates); in practice it’s not that simple.

What I know is that in software estimation and benchmarking, there is no free lunch -- you get what you pay for.  And I’ll explain why…

Dear Carol: 

I’m a developer in our IT department and we know that project estimating is a big deal for our customers.  Somehow, no matter what we do, we can't seem to get it right.  We do know that project size is an important input to good estimating  and our gut feel is that if we get sizing right, we’ll do better estimates!  I know you recommend using function points, but I’ve also been reading a lot about use case points, story points, SLOC, sizing by analogy, T-shirt sizing, COSMIC and other sizing metrics.  We do a mix of waterfall, agile, iterative and even Kanban to do our projects so what’s the best choice for sizing to get the best results? 

- Size Challenged in Milwaukee

Dear Size Challenged:    

Sometimes I wonder if the internet and the proliferation of (mis)information is a good thing. Before the internet, our choices (for sizing or estimating or anything) were limited and we didn’t have such an overwhelming task to first sift through many options before taking action.  Your list of software sizing choices is an example of this. 

Dear Carol:

As a project manager who is new to formal project estimating, I’ve been hearing about the importance of having project histories available for accurate estimating.  We just purchased SLIM-Estimate but we don’t have any project history.  Can we still use SLIM, and how many projects do we need before we can get accurate estimates?

– PM in Atlanta

Dear PM:

You may have heard that “history repeats itself” and the adage is true in software development.  Completed projects where the actual software size, effort hours, duration and cost are often the best predictors of future performance on projects – and your own project history gives accurate indicators of how your corporation performs.  However, the majority of QSM clients who purchase SLIM-Estimate start out with little or none of their own project history.  The good news is that the SLIM tool comes preloaded with productivity, duration, staffing, and effort hours trend lines based on thousands of completed real life projects, and delineated by industry and type of project.  When you do an estimate using SLIM, the Monte Carlo simulation models are run, and the results are compared against trend line graphs so that you can see how your estimate of effort, duration, staffing and cost compare to the chosen industry.  This gives you the confidence to know where your estimate falls against comparable completed projects (of a given size.) If your estimates fall outside the bounds of a single standard deviation above or below the industry trend lines, you know that you may want to reassess the assumptions of your estimate.

Dear Carol: 

I work in IT and we do a lot of our software development projects based on pre-defined delivery dates (no one really knows where they come from).  Sometimes this works out, but usually we end up delivering the project months late because we had no idea the project was as big or as complex as anyone had originally thought. A friend says his company uses “t-shirt sizing” for software project estimates and I’ve never heard of it. What can you tell me about this new approach?

- I’m willing to learn from the fashion industry 

Dear I’m willing:

T-shirt sizing has been around in one form or another for a few years but it is not a widespread mainstream estimating method.  It is a quick and dirty way to estimate software size using ranges of size.  Once you have the relative size (e.g., XS, Small, Medium, Large, XL, XXL, or larger), you can enter it into several different estimating tools as the size on which to base the estimate. While this doesn’t give you a guaranteed size for the software to be delivered, it is better than not having any idea of the size.  Size (scope) is one of the three main tenets of the triple constraints (scope, budget and schedule) for project management.  Given one, you can estimate the other two.

Sometimes business life follows literature. Recently, I came across the following quote and I had to pause:

“Before we build a better mousetrap, we need to find
out if there are any mice out there.” - Yogi Berra

It reminded me of a conversation I had over lunch 15 years ago, when I was president of the International Function Point Users Group (IFPUG) and Charles Symons was president of the UK Software Metrics Association (UKSMA), where we were talking about the future of software sizing.  IFPUG is the inventor of the original method to size software using a measure called “Function Points.”  Charles is the creator of a similar UK method called Mark II function points and a co-creator of the Common Software Metrics International Consortium (COSMIC) sizing method that was, at the time, still in its infancy.  I’m paraphrasing with the words but I believe it captures the content of our conversation:

“The problem with function points,” Charles remarked, “is that they aren’t yet perfect enough.  What we need is a better mousetrap and the world will beat a path to our door.”

I disagreed saying “I don’t think that’s the problem at all – I think the problem is that world doesn’t yet see mice as a problem.”

QSM hosts a free advice column for software professionals who seek help to solve project management, communication and general software project issues. Carol Dekkers is a QSM consultant and IT measurement and project management expert who speaks internationally on topics related to software development. Send your questions to Ask Carol!

Dear Carol: 

My manager frequently says “You can’t manage what you can’t measure” and then tells us to collect more and more metrics, willy-nilly. We’ve done this for years now and it just doesn’t seem to make sense to collect all this data that’s never used, when we’ve got better things to do like developing software. What can we do to make him stop this unproductive exercise?

– Over Metricated

Dear Over: 

 This is a common situation in software development: management pursuit of metrics without a clear plan to use them.

The “you can’t manage what you can’t measure” is a paraphrase of Tom DeMarco’s quote, “You can't manage what you can't control, and you can't control what you don't measure.”¹ What is interesting is that DeMarco’s follow-up quote 13 years later is seldom cited: “Metrics cost a ton of money. It costs a lot to collect them badly and a lot more to collect them well... At its best... metrics can inform and guide developers, and help organizations to improve.  At its worst, it can do actual harm… And there is an entire range between the two extremes...”² 

QSM hosts a free advice column for software professionals who seek help to solve project management, communication and general software project issues. Carol Dekkers is a QSM consultant and IT measurement and project management expert who speaks internationally on topics related to software development. Send your questions to Ask Carol!

Dear Carol: 

I’ve been a member of many software development teams and I simply don’t understand the point of doing early project estimates before we know what are the requirements. It just causes problems once the project starts because the estimate becomes the budget and drives the schedule. Obviously, the estimates are wrong because they are based on flawed/incomplete data, so why would anyone even bother doing an estimate when the budget and schedule go awry as soon as the project starts? Estimates cause management and project managers to “stress out” trying to meet an artificial date and budget – we ought to abandon estimates and just get to work on the project! What am I missing here? 

- Looking for answers     

Dear Looking:

You’re not the first person to question the point of estimating before requirements are known; it can seem futile when it seems that they turn into budgets and schedules.  Even though there is uncertainty (and risk) with early estimates, there are reasons that companies do early estimates:

QSM hosts a free advice column for software professionals who seek help to solve project management, communication and general software project issues. Carol Dekkers is a QSM consultant and IT measurement and project management expert who speaks internationally on topics related to software development. Send your questions to Ask Carol!

Hi Carol:

Thanks for this excellent initiative. One of my key clients is planning to move away from FP counting as they think it’s expensive, takes time, does not measure non-functional work and also they do not want to invest in auditing the FP results. Instead, they are considering using LOC. We have tried explaining them all the shortcoming of LOC but no use. In fact we advised them to use SNAP along with FP but looks like they are just focusing on cost!

In your article on 'Size Matters', I noticed you had mentioned few other techniques to measure size like RICE Objects and Implementation Units. Would like to learn more about these and would like to understand if these are industry standards. Can you please share some insights?

– Sizing Enthusiast

Dear Sizing:

Because you are someone who knows the value of functional sizing (aka function points,) it is frustrating when management focuses on the cost of measurement rather than the value delivered.  I’m wondering whether there is a disconnect between the perceived value and the cost of FP counting. There are a couple of potential issues here that I’d like you to consider before we get into the sizing alternatives

QSM will be hosting a new free advice column for software professionals who seek help to solve project management, communication and general software project issues. The first few scenarios are based on questions we receive all the time. Carol Dekkers is a QSM consultant and IT measurement and project management expert who speaks internationally on topics related to software development. Send your questions to Ask Carol!

Dear Carol: 

I’m a systems analyst working for a telecom company and I have several projects on the go at the same time.  Our PMO (Project Management Office) told us that now we have to start tracking the size of our projects both at the beginning and at the end.  I can’t believe this – it’s just more metrics and (redundant) data.  I already track project size because I do time reporting (by project) and all they need to do is add up my hours at the end and voila, you’ve got the project size: the more hours it took - the bigger the project.  They don’t seem to get it and I’ve asked for an exemption from this overhead task. Their response was to enroll me in a two day sizing course!  What can I do to prove to them that they already have the size because I keep track of my hours? 

- Frustrated in Seattle

Dear Frustrated: