Organizations often come to us in the early stages of shopping for a software estimation tool and, oftentimes we find that they could be asking some additional questions. They often focus on the tool’s operating system, database structure, and architecture, when they could also be focusing on the quality of the data behind the tool. They also ask a lot of questions about inputting detailed information when really it would be in their best interest to focus on solid project-level information since detail-level inputs are often not available early in the planning lifecycle. Instead of focusing on the number of hours allotted to each individual person, it would be more beneficial to focus on how much work the overall team needs to finish.

In our 30+ years of experience in this industry, we've found that, no matter what tool an organization ultimately chooses, they need to be asking the right questions. Here is the criteria they should consider.

Tool Capability

As with any tool, it is important to match the tool with the job at hand. Using a screw driver to perform the task of a chisel will yield poor results. The same is true with trying to use a detailed planning tool in place of a software estimation tool. Make sure that you consider at what point in time formal estimates are required and how the resulting information is used to support negotiation and business decision making. Here are the main issues that should be taken into consideration when assessing an estimation tool.

Often within technology organizations there is a general belief that increasing staff increases the amount of production. But what if there were better options? Wouldn’t it be great to see some additional management options using predictive analytics? This type of analysis could save organizations millions of dollars by showing how to hit their goals by just planning more effectively.

Where do you start? First, we recommend centralizing your project data so your information can be easily accessed. These projects can be completed, in-progress, or getting ready to start. The best way to do this is with a tool that lets you store the data and that also lets you generate the forecasts, all in one convenient place.

The next step would be to run built-in forecasting models to see if you can complete the required amount of work with the existing number of resources.  These models also provide other options to consider, like adjusting the number of resources on a software release or extending a project schedule to save money. The best models are empirically-based and time-tested. To generate the analysis, you need to enter some basic project level goals and the models then leverage historical data to forecast a reliable duration, effort, cost, and staffing assessment for each release.

What if you could leverage summary level cost, duration, and productivity data to support estimates for future projects, at the release and enterprise level? C-level executives, development managers, and project stakeholders are all involved at some level in project planning. They want quick access to information on a regular basis and they want web-based solutions to make it happen. So how does it all work? There are web-based analytics tools that allow you to create a centralized database for all of your projects. These tools store the data, leverage it to generate project and portfolio estimates, and then provide a communication vehicle throughout the organization to ensure that everyone involved is on the same page. It all starts with having the data in one place. 

Once you have all of your project data in one place, then you can focus on analyzing the completed projects. You can compare them against industry trends and leverage a 5-star report to show how they rate on performance in the industry. The initial measures to focus on would be size, duration, effort, reliability, and productivity. A project's productivity will be calculated automatically once you have entered the size, duration and effort. We call this measure a Productivity Index. This measure can be compared to industry and used as a benchmark to measure process improvements over time.  These numbers give you a quantitative picture of your current project environment.  

I am a big sports fan and since I work for a software metrics company, I started thinking about the similarities in productivity measurement in both industries. From draft picks to game planning decisions, managers in sports measure their team’s productivity to help them make better decisions in the future. Software executives and product owners do the same thing; they measure productivity in order to make better planning decisions regarding upcoming projects.

To measure productivity in baseball, we look at measures like batting average, on-base percentage, slugging percentage, and earned run average. When measuring the productivity of agile software projects, we often look at the velocity, which takes into account the number of user stories completed in each sprint. This type of historical data helps us plan effectively at the detailed level.

As part of our work at QSM, we are often asked to provide plans at the release and portfolio level. To provide these plans reliably, we use a macro level, empirically-based productivity measure that encapsulates a number of project-related factors. This measure is called the Productivity Index, an integral part of the Putnam Model.  Also known as the SLIM Model, the Putnam Model was invented by Larry Putnam Sr. almost 40 years ago and is having a big impact on software measurement more than ever today.

Once we know the total number of user stories (or any size measure), the release level effort, and the duration, we can calculate the Productivity Index of a project. The Productivity Index also takes into account the project environment including: the experience level of the team, the complexity of the software, and the quality of the tools and methods being used on the project.

The QSM team thoroughly enjoyed our time at the Gartner Symposium/ITXPO in Orlando a couple of weeks ago. We spent most of our time at the QSM exhibit, networking sessions, and at various presentation sessions.

We met many C-Level executives with process improvement on their mind. Our main focus was to learn about their specific needs in the IT budgeting and vendor management areas. We conducted question and answer sessions and provided real world examples on how to use business and software analytics to manage the complexities of IT budgeting, taking into account in-house projects as well as the project durations and costs proposed by their vendors.

Many of the senior executives we spoke with also had a need to integrate cost, duration and resource information with their project portfolio management solutions. The big focus was leveraging demand management to improve capacity planning. We provided demonstrations showing the SLIM-PPM Integration Framework which provides the release level cost, effort, resources and schedules that PPM solutions need validated to reduce risk in the enterprise portfolio.

There are so many questions around agile planning, one of the biggest being: do we need an estimate? Project managers and scrum masters will spend months developing a system either for internal use or for their clients, yet some of them say that estimates are not needed. Some recommend starting the project without an estimate. They say they will see how the first few weeks go before they generate an estimate. Others say not to worry about an estimate at all; they are a waste of time.

The problem with those recommendations is that there are business decisions that need to be made regarding whether or not to even start the project. Reliable estimates for cost and duration are needed to make these decisions. Also, for the projects that do move forward, there is usually limited information available early in the lifecycle, not enough to provide a detailed plan. Product owners need to see the big picture before a reliable detailed plan is generated.

There is also the IT manager that needs to figure out how they will allocate their resources. There is the vendor manager that needs to evaluate multiple bids for a large software system that could cost the company millions of dollars. There is the proposal manager that needs to write a proposal that must be cost competitive to win business. And, there is an annual budget at stake and the CIO needs to know how much money their development organization is going to spend over the next 12 months. You can’t support these decisions in the best way possible without reliable release level estimates.

Before an agile project starts, many product owners will run an early release estimate. Once the activities get started, managers or scrum masters begin to track the progress. When they track, they usually include the person hours of effort and the number of user stories within each sprint.  There are a number of agile tracking tools and methods in the marketplace for these tasks. 

But wouldn’t it be great if the tracking and estimation process could be combined, using the actual tracked effort and user stories to run new and improved ongoing estimates at the release level? At QSM, we have applied this process to hundreds of software projects. This type of adaptive forecasting can help save time and effort by showing when a software release is headed down the wrong path. It can also help organizations avoid signing up to inflated resource planning numbers that cause many companies to waste millions of dollars at the release and enterprise levels.

In the SLIM-Control charts below, we see the blue plans versus the red actuals and the new forecasts in white. We are capturing the total effort spent and the actual work delivered each week, then using that information to generate mathematical models that produce new empirically based forecasts at the release level.

One of the things I hear from many c-level managers is how difficult it is and how long it takes to generate reliable resource plans at the enterprise level. Many organizations take months to generate their annual budgets and often times the negotiated budgets end up being unrealistic. To fix this problem we need to combine good capacity planning with good demand management. There are a number of project portfolio management tools to help with the capacity planning. The problem is the numbers will be off if we don’t get the demand management part right.

There are world class demand management tools available that can be used by business decision makers. These tools allow us to come up with empirically based, reliable project and enterprise level resource plans. In the SLIM-Estimate view below you can see the forecasted effort by role by month.

The view below shows the plan being sanity checked with industry data so we can better negotiate our budgets with confidence.

The even bigger news is that we can automatically feed our empirically based demand numbers into our PPM tools, making the job of capacity planning much easier and more reliable. In the view below you can see two separate plans, represented in side by side columns. The original PPM plan was updated automatically from an empirically based and sanity checked plan from SLIM-Estimate.

Many of the project managers that I speak with track their software and systems projects at a very detailed level. They use detailed spreadsheets or other tools to track hours and tasks on a daily basis. This is fine, but it's important to manage the big picture so we can avoid assigning detailed tasks to duration and budget goals that are unrealistic.

By "big picture" I mean tracking at the project release level and focusing on a few key actuals: size, duration, effort, reliability, and efficiency. It's important to track these actuals to a reliable plan. These are the measures that can give us the biggest and quickest insight into a project’s potential success or failure. You can see this analysis in the SLIM-Control graphs below, showing the blue plans versus the red actuals.

Once the project is underway and we start tracking the actuals, we can generate new project forecasts based on the actual work delivered, time elapsed, and effort spent. These new forecasts are empirically-based. This will enable us to adapt to change requests, see when the project will finish and how much it will cost. The SLIM-Control graphs below show the blue plans versus the red actuals plus the new forecasts shown in white. SLIM-Control is curve-fitting to the actuals and running empirically-based mathematical models to generate the new forecasts.

We speak to a number of scrum masters, project managers, and CIOs each month. QSM does research on software development projects - all types, including agile and waterfall. We work with a huge database of completed software projects, updated with new industry data on a regular basis. We provide predictive analytics and we study cost, schedule, risk, quality, size, resource demand management, business intelligence, and vendor management.

Within the last couple of years, we have been hearing some agile managers say that there is no need to estimate agile projects. They say that they are managing in smaller increments or sprints and that they already know how much velocity can be achieved. They also say they are constantly “grooming the backlog” so there is no need for a formal estimation model to forecast the amount of work that needs to be done. They are managing at the sprint and task level so they feel like they have everything under control.

The bottom line is that agile projects still need to be estimated to see the big picture: the release estimate. It's essential to know the release cost and effort before the project plan is handed off to the scrum master and before committing to the customer. The negative feedback we hear regarding estimation is somewhat ironic because agile processes actually lend themselves very well to formal estimation since they promote the use of size measures like user stories, story points, and epics. Having good size information is a big part of estimating successfully.