- Capture some historical data on your projects and keep it simple. The more data, the better, but you can get a good start to your estimation program with just a few projects and a small amount of data from each of those projects. Focus on the core metrics: size, duration, reliability, productivity, and effort.
- Estimate at the release level before detailed planning takes place. This will enable you to tailor your detailed plan to goals that are reasonable. Many analysts spend hours laying out detailed plans for projects that end up over budget and late because they don’t figure out the big picture first.
- Use an empirically-based model that enables you to manage uncertainty. When making big decisions, it’s important to see the 90% chance compared to the 50%.
- Sanity-check your estimates with industry analytics. It’s always good to see typical cost and duration trends from projects that are similar to yours.
When the Honorable Ellen M. Lord, Undersecretary of Defense for Acquisition & Sustainment (USD/A&S) told the Senate Armed Services Committee on Dec. 7 that she intends to demand a higher level of accountability from program managers, you could feel mixed emotions from DoD acquisition professionals. Many are applauding the vocal prioritization on accountability. However, I’m sure struggling acquisition program managers and support contractors, are likely feeling they have a more focused target on their back. There will certainly be other major changes from the former Acquisition, Technology and Logistics (AT&L) office reorganization to two new USD-level offices of USD/A&S and Research & Engineering (USD/R&E). Each will surely be eager to show respective value to the Pentagon in their responsibilities to improve the DoD acquisition process. Particularly, as the DoD continues a focus on DoD business transformation priorities and ensuring that they are acquiring effective defense business systems with capabilities to support those priorities, I’d like to offer some firsthand observations that suggests there still remains a lack of consistency in how we manage that process.
Accountability Requires Consistency
Today more than ever we have access to large amounts of information. You've probably heard the term "big data," which in essence is having access to large amounts of data and examining the trends in that data. But many executives want to know how they can leverage this information to solve business problems, like lowering IT costs. One way is to use the data to do a better job of estimating IT projects.
Better estimating helps avoid signing up to schedules and budgets that are unrealistic; it helps avoid overstaffing a project or a portfolio of projects; and it helps calculate how much work can be completed within project constraints. In addition, it improves communication internally across the enterprise and externally between the vendor and the client. You can apply estimation to in-house projects and you can use it to generate better proposals or to do a better job of evaluating proposals. It can also help you negotiate more effectively.
To do a better job of estimating, you need to make good decisions regarding which metrics to leverage. You might have thousands of data points, but it's important to streamline the focus to the core release level metrics: cost, duration, effort, reliability, and productivity. Next, you need to find a centralized place to organize and store the data so you can analyze it. There are tools out there that can help you. In the view below, you can see a portfolio of projects stored in a centralized place with the ability to manage the access and security.
In any development methodology, we throw around the word “estimate” freely not really understanding how it’s interpreted by many. In many cases, an estimate, regardless of its content and process by which it was created, is received implicitly as a pin point number with the accuracy of multiple decimal points. This presents a problem for all parties involved.
I recently had a discussion with a gentleman who told me that prior to using our SLIM tool, the estimates in his organization were arrived at by casual hallway conversations, often started with, “how much do you think this will cost, and how long will it take?” A typical response is, “hmmm, I’d say about 6 months and $500K.” That innocent musing then becomes the information upon which business decisions are based, leadership bonuses may be won or lost, and the credibility of the dev team is on the line.
I’d strongly recommend adhering to some definitions when talking estimates. These definitions will help mitigate potential misunderstandings around the agreement of what makes an estimate:
Are you having a hard time collecting total effort for SLIM Phase 3 on a completed project?
Can you get a good handle on the peak staff?
Maybe we can still determine PI!
It is difficult and often time consuming to collect historical metrics on completed software projects. However, some metrics are commonly easier to collect than others, namely, peak staff, start and end dates of Phase 3 and the size of the completed project. Asking these questions can get things started:
- So, how many people did you have at the peak?
- When did you start design and when was integration testing done?
- Can we measure the size of the software?
That gives us the minimum set of metrics to dig up.
However, the PI (Productivity Index) formula also requires phase 3 effort. Can we use SLIM to generate a PI that is useful, using peak staff instead of total effort?
A statistical test on historical metrics can answer this question.
What are we comparing?
- Projects used in this study had all 4 of the following: actual reported effort; size; peak staff; duration.
- For each project, a derived effort is generated from peak staff, size and duration.
- A derived PI is generated from the derived effort, size and duration. This derived PI is then compared to the actual PI.
Definitions for terms:
In a world trending away from traditional waterfall and toward agile development methodologies, it would be understandable to assume that there is no longer a need for software project estimation. Many agile practitioners feel there’s no value in estimation, since they are already working with smaller increments and sprints and grooming their backlogs.
However, that assumption would be wrong.
In a recent interview, Ken Schwaber and Jeff Sutherland, the founders of Scrum, were asked about the #NoEstimate movement. Schwaber believes a more appropriate term may be #NoMeaningfulCommitments. He feels that people often confuse estimation with commitments and that, in fact, estimates should be used in making commitments. Sutherland mentioned a recent Rally (now CA) survey that asked members of 70,000 scrum teams about the estimation techniques they used and then correlated those techniques with speed of delivery. They found that those that eschewed estimates altogether yielded some of the slowest delivery times, while those that employed scope-based estimation delivered the fastest results.
Larry Putnam, Jr.'s latest article for InfoQ explains why estimation is still a very valuable practice, even in organizations that are dependent upon agile development methodologies. He outlines several best practices that stakeholders can use to get their software estimation processes back on track toward adding value to their organizations. Software estimation does not have to be difficult, onerous, or ineffective. Done right, it can be a highly effective tool that can help project managers provide value to their organizations.
The challenges surrounding software estimation are both well known and well documented. Most discussions on this topic center around the technical challenges estimators face: tight schedules, unclear scope, evolving requirements, and accounting for dependencies and risk. But there's a more fundamental challenge we don't hear so much about – educating stakeholders and making the business case for structured, yet practical estimation, and why it is a critical success factor.
Let's face it: process improvement is rarely cost-free. Businesses expect a visible return on investments made in estimation tools and training. The benefits of quality software estimation can be compelling, but moving decision makers from "open to the idea of estimation" to "willing to commit money and resources" can be difficult for busy analysts and managers juggling multiple roles and tasks.
In her latest video, QSM's Laura Zuber explains what makes a good software estimate and how empirically-based estimation helps projects:
Software estimation is no longer a solitary activity - as more organizations continue to move away from silo-driven development methodologies, the role of collaboration becomes increasingly essential. Organizations may have estimation experts within their companies, but there’s now a huge push towards bringing all stakeholders together throughout the estimation process. This movement is largely due to an increasingly-apparent correlation between collaboration and successful estimation.
When estimation experts create an environment of continuous collaboration between all stakeholders - from the technical to business level - estimation accuracy improves and expectations overall are better aligned across every stage of the software development lifecycle. That being said, it is critical that organizations establish an effective system for collaboration that appeals to all stakeholders.
QSM is pleased to announce the release of SLIM-Collaborate 3.0, the web-based, software-as-a-service version of our trusted software estimation, tracking and benchmarking suite. With more advanced workflow capabilities, the updated version of SLIM-Collaborate enables more efficient communication between stakeholders throughout the estimation process. Additionally, the demand resource capabilities added in SLIM-Collaborate 3.0 make it easier for users to identify staffing needs and allocate resources to a software project.
Highlights of the new capabilities in SLIM-Collaborate 3.0 include:
Larry Putnam, Jr. was recently quoted in 'Framework and Standards Are the "Essence" of Agile at Scale,' an article published in SD Times. The article consulted other industry experts such as Ivar Jacobson, Matt Schenck, Dean Leffingwell, and Ken France on best practices for implementing agile at scale. Larry's advice below.
Agile estimation is the key to a successful SAFe implementation
With all of the benefits of SAFe, getting it right is key. Using agile estimation can help.
“For organizations that are implementing SAFe, they’re really trying to coordinate a lot of different stakeholders within the organization and the real benefit they’re looking to get out of it is a much more nimble delivery,” said Larry Putnam Jr., co-CEO of QSM. “To be able to do that, we’ve got all these different stakeholders that we have to coordinate. That becomes really complicated and estimation is kind of the communication vehicle for these different stakeholders.”
Putnam explains that the highest level of an organization is the business and its stakeholders. The stakeholders or senior managers within an organization need to ask in what direction the business needs to go and how software will support that. These needs are usually articulated at a high level, said Putnam.
“Those are going to get apportioned across different value streams, and they’re really looking at the whole portfolio of what’s going on within the enterprise,” Putnam said. “In order to implement all those capabilities the business wants, you’ve got all these different cross-functional teams that are working on different pieces of the system.”