Bob Dale - standing in the corner of his Midtown office, face flushed with uncertainty, hands nervously cupping a re-gifted magic 8-ball - is about to make the worst decision of his life.
A week ago, Bob's group was asked to bid on a fixed-price contract for a top financial firm's marquee software project - six million lines of code and no margin for error. Trouble is: Bob's never handled a project like this. So there's no way of knowing whether his estimate - six months, staff of 1,300 - is reasonable or a recipe for disaster. Nervously, our hero sketches a few equations on the back of an envelope and turns the 8-ball on its head. "All signs point to yes."
Six months from now when the project lies in shambles, beset with cost overruns and unending delays, Bob will remember the precise instant that his professional life took a nose-dive into the Bermuda triangle.
Or maybe not. In an alternate universe, Bob does things slightly differently. Before signing off on the fixed-price bid, our hero consults his estimation software.
He enters a few inputs and sees that his chances for success are only slightly higher than the chances of an impromptu alternate universe forming in the office suite next door. Quickly, Bob brings his original plan into alignment with the latest financial industry performance data. He adjusts the schedule to 20 months and reduces the staff to 250 people. The software validates his plan against a sample of recently completed financial sector projects. He factors in uncertainty and his firm's risk tolerances, and he generates a revised estimate.
The probability of successfully executing the revised (i.e., data-driven) plan increases to 90%. "Thank heavens for technology," he thinks to himself. Catastrophe averted; 8-ball re-gifted once again.
Witchcraft, or the Science of Estimation?
"This is a story about the power of data," explains my colleague Kate Armel, Director of Research and Technical Support for project estimation authority at QSM. In other words, data-driven estimates are grounded in thousands of real project outcomes, not opinions (which can be biased or inaccurate) or marketing hopes. If learning from experience is the key to success, imagine what someone could do with real-time access to three decades of research, thousands of projects, and over 600 industry trends.
"Having relevant data on tap helps our clients understand and account for the complex and counterintuitive interactions between staff, schedule, defects, and productivity," says Armel. "Estimators can factor in the effects of using different technologies and development methods and quickly counter unrealistic expectations. For example, they can show - empirically - that using large teams delivers only modest schedule reduction while increasing cost and defects by an average of 300-400 percent.
"We know that software development is a moving target. In the past decade alone, incremental development and Agile methods continue to shrink the scope of the projects in our database, reducing failure rates by over 10% and improving average time-to-market by nearly 20 percent. For firms determined to stay competitive in a rapidly changing environment, this kind of continual real-time feedback is invaluable."
Clearly, there's no shortage of software stats and metrics for CIOs to ponder. The question is: how does access to accurate, timely information improve a company's bottom line?
Fear and Loathing in the C-Suite
In a 2007 survey, IBM's Scott Ambler found that 68.6% of IT professionals have been involved in projects that they knew would fail from the very start. Go ahead and read that sentence again.
"In other words, the people in a position to get the project on the right track, or at least in a position to influence the people who could do so, couldn't do anything about it," concluded Ambler.
A quick check of QSM's database of 10,000-plus completed projects reveals the impact of signing up for unachievable plans. A recent sample of more than 1400 projects that provided overrun or slippage data showed that one third overran their planned schedules by at least 20 percent.
How do project managers react when the schedule starts to slip? Typically, they pile on staff, driving up the cost and creating additional defects. (About 26 percent of projects exceeded their budgets by 20 percent or more.) As a last resort, stressed projects may cut features, delivering less functionality than planned. Only 0.6 percent of projects chose this (most drastic) option, probably because the "minimum releasable scope" constraint reared its ugly head.
Bad (or no) initial estimation forces projects to make unpalatable choices (a.k.a., the least bad option), often long after the window when timely intervention has the best chance of turning a project around. No matter how talented a team and its IT leadership may be, there's no painless way to combat faulty expectations or untenable parameters.
"We joke about the magic 8-ball, but that's really not so far from the truth," says my colleague Keith Ciocco, vice president of sales at QSM. "For instance, we're seeing a number of executives who confuse bottom-up planning with estimation. They'll divide a software project into its component tasks, and then try to match each task with the appropriate numbers."
So what's wrong with that?
"The problem is timing. Estimation has to come first, from the top-down. Schedules and budgets are usually locked-in early in the lifecycle - before we ever know what those detailed component tasks will look like."
In short, it's rearranging deck chairs on the Titanic, when the course for the iceberg has already been set.
What executives fail to realize, it seems, is that the majority of their IT challenges are tied to estimation - hitting profit margins, setting backlogs, utilizing staff. On top of that, the project data used to make accurate estimates can be essential for assessing productivity, comparing performance, and measuring improvement.
"CIOs know they need to do more with less, but they don't have any way to measure their progress," Ciocco says. "How does their productivity compare with competitors' teams'? Where are the bottlenecks happening?"
"When projects go awry, we tend to assume that the issue was our performance. But when you look at the data, it often shows that the team performed quite well. It's merely that the budget and the timeline were completely unrealistic."
How's that for music to a CIO's ears: You could drastically improve your company's performance and profitability without doing anything - just by making smarter, more informed decisions.
Estimation: Luxury or Necessity
CIOs are stressed, and with good reason. If you think making major decisions on massive IT projects is a nerve-wracking experience under normal circumstances, how about in a down economy?
"One of the biggest objections we hear to project estimation is that it's a luxury," Ciocco says. "Executives tend to think, sure, we'd love to have access to all that great data. But we just can't afford it right now."
The irony is that they can't afford to go without it. In fact, a tough economic climate only magnifies the staggering financial risks associated with bad decision-making at the outset. About this, the data is unequivocal: That first decision - that initial estimation - is the key to everything -productivity, profitability, and ultimately project success.
"So do whatever you need to do to make that first decision wisely," advises Ciocco. "No matter what your 8-ball is telling you."