Software Reliability Modeling in the Age of Continuous Integration / Continuous Delivery

Defects Prediction Curve

Quantitative Software Management (QSM) consultant, James Heires, recently discussed the benefits of estimating and forecasting software reliability at RAMS (Reliability & Maintainability Symposium) 2023.  Theme for the conference:  "Artificial Intelligence and Machine Learning (AI/ML) application to our R&M tools, techniques, and processes (and products) promises speed and scale.... When program management instantiates advanced techniques into R&M engineering activities, such as digital design and machine learning and other advanced analytics, it enables products to evolve at a much more proactive, effective, and cost-efficient approach.  Ultimately it facilitates increased speed to market, adoption of new technology, and especially for repairable systems, products that are more reliable, maintainable, and supportable."

Software Estimation Basics: What Is a Top-Down Estimate?

With over 45 years of experience in the software estimation field, you might say we know a thing or two about the subject.  This is why we wanted to define key principles with our Software Estimation Basics series. Let's start with a common question we hear a lot: what is a top-down estimate?

What Is Top-Down Estimation?

Top-down estimation is a technique that uses high-level, summary project goals or constraints - scope, cost, schedule, resources, and productivity - to evaluate all possible scenarios to meet a project's goals and the risks associated with each scenario.  Top-down estimation is the best approach to employ early in the project and project portfolio planning process when very little is known.

A common misconception about top-down estimation is that it is used solely to produce a ballpark figure for cost. As we'll explain more in this post, good top-down estimates calculate values for key management metrics, not just cost.

A top-down estimate can be generated by anyone who needs to know a software project's budget, schedule, or staffing - project managers, anyone in the PMO, technical architects, cost analysts, and product owners. C-suite executives can leverage top-down estimates when they need to approve project budgets and schedules.

Blog Post Categories 
Estimation SLIM-Estimate

What Do Software Estimation and the NFL Have in Common?

Football season is well underway, and as a loyal football fan I enjoy watching my team go through the highs and lows of wins and losses. Many of the weekly games come down to crucial plays where coaches must decide the plays to call.  Coaches now more than ever are analyzed by what plays they called or how they managed the game.  It’s interesting that these days you often hear of data driven decision making and what the analytics say.  More and more, technology is playing a big role in our lives, including sports.  Like so many industries, data-driven approaches are using models to determine “win probability”.  The data is now being used to determine if a team will go for it on 4th down, punt, kick the extra point, go for a 2-point conversion, and even when to take time outs!  This has changed how coaches manage the game. The analytics provide coaches the ability to make informed decisions and defend their actions based on data.  Coaches can now make critical decisions with confidence.  John Harbaugh (Baltimore Ravens Head Coach) is known to have an analyst in his ear during games feeding him probabilities. 

What if you could do the same for your software delivery?  QSM’s SLIM-Estimate tool leverages industry trends on similar projects along with your own historical data to provide valuable insight on potential cost and schedule outcomes. That’s like having your own personal analyst in your ear. 

Blog Post Categories 
Estimation SLIM-Estimate Risk Management

Get Estimation Results Faster with QSM's New SLIM Pilot QuickStart

Estimation can be time-consuming. Whether it's gathering initial requirements from stakeholders, performing risk trade-off analysis, or reworking estimates when plans change, these processes can take days or, worse, weeks that you simply don't have.

We hear that. And at QSM, our goal has always been to make the estimation process as quick and painless as possible. For almost 45 years, we've refined our tools to provide accurate project predictions with very few inputs required. Leveraging the power of industry data and proven mathematical models, you can generate defensible estimates within minutes and examine alternative scenarios without tedious rework. But as with any tool, there is a learning curve and we recognize how hard it is to find time in a busy schedule for training. We want to get you set up and reaping the benefits of our tools so you can get back to working with your team to ensure that your clients and stakeholders are happy with the overall product delivery. This is why we're introducing the SLIM Pilot QuickStart, a training and coaching program with flexible delivery designed to work with individual client schedules.

Software Estimation Training

We believe the best way to learn is to learn by doing. The SLIM Pilot QuickStart program is an online, instructor-led live session that focuses on applying QSM's SLIM estimation tools and best practices to your current measurement problems. This "learn as you go" approach allows you to improve your estimation process quickly to improve the bottom line. The program centers around a 2-hour intro session filled with a wealth of information including: 

Blog Post Categories 
Training Estimation SLIM Suite SLIM-Estimate

How to Mitigate Risk when Migrating to the Cloud

Moving to the cloud is big business these days. With any type of digital transformation; managing the migration, integration, and development work can be a stressful experience with a lot of risk. Wouldn't it be great to have a crystal ball that allowed you to make these big cost, staff and schedule decisions early in the planning process? I'll propose the next best thing: a database of completed and validated projects spanning multiple industries and development methodologies that can help you predict the way your project will behave.

The QSM Industry Database can provide this type of knowledge. With over 40 years of research and development, it includes metrics from thousands of completed technology projects from various application domains and industries. By leveraging this type of database, you can access industry benchmarks that can help you navigate the uncertainty that comes with planning IT transformations, software engineering, and cloud migrations.

What should this cloud migration cost? How long should it take? How many people will you need? What level of quality and productivity needs to be achieved to have a successful delivery? What are the chances of finishing within a 6-month time frame versus a 10 month? The secret to finding these answers is in the historical data. The way to bring that data to life is to use good scope-based estimation methods. In a view from QSM's SLIM-Estimate tool below, you can see some examples of how you might measure the size of a particular cloud migration to get started with an estimate.

New Video: How to Use Project History for Early Software Decisions

Early project decisions, when not much is known, are easily the hardest. They're also often the most critical. Maybe you've found yourself in a position where you need to communicate to stakeholders what your work is going to cost and how long it will take to deliver. Feeling the pressure to deliver, you might have to make decisions based on gut feel instead of past performance. This can lead to setting unrealistic targets and often results in projects going late or over budget. 

At QSM, this is when we recommend turning to historical data. Whether it's your own data or trendlines from the 13,000 validated projects in the QSM industry database, leveraging actual completed projects can make your estimates more reliable. 

Believe it or not, collecting your own project history isn't as difficult as it sounds. We recommend capturing just a few basic metrics: Functionality Delivered, Total Effort, and Total Duration. Once you have this information, you can calculate a Productivity Index, which is the measure of productivity for the overall project or release. Then all of these metrics can be leveraged by any of the other project lifecycle tools in the SLIM-Suite for estimating, tracking, and benchmarking.

In the video above, you can see how easy it is to gather your own completed projects to use early in the planning process and determine if your estimates are reasonable or not. This helps you understand the big picture before you make any important project or portfolio decisions. 

There's No Risk in Software Project Planning

I like listening to audiobooks when I go for a morning run. This month it is a David Baldacci thriller about two CIA professional killers pitted against each other who end up working together to save us all from global catastrophe.  Apparently, there is a ton of planning involved in stealthily hunting a target, making the kill, and then getting away unseen.  That’s because there is a lot of risk.  Timing is critical, down to the split second, and the slightest mistake can end your life.  Discussing the highly complex plan to foil an assassination attempt with his partner, one agent says to the other, “There’s no risk in planning. The risk is in the execution.”

That got me thinking about software development and QSM’s SLIM-Suite estimating, tracking, and forecasting tools.  Do I agree with that statement?  Yes and no.  Let’s look at it both activities – project planning and execution.  


The activity of planning is not risky as far as your personal safety is concerned. You probably aren’t in danger of getting attacked or making a mistake that will cause bodily injury (you may experience emotional trauma or at least endure a minor headache).  It is most definitely risky for software development programs and initiatives, however, because aggressive plans based on poor estimates handicap the delivery team.  Without understanding the dynamics of software development projects or the ability to rapidly compute a range of potential outcomes to identify risky scenarios, planners may inadvertently commit to unrealistic schedule, budget, and staffing goals.  In fact, most plans are “goal based” ― task lists and staffing plans derived to give management or the customer what they want, because there is no solid framework or supporting data to defend against it. 

QSM Releases SLIM-Suite 10.3

We are pleased to anounce the release of SLIM-Suite 10.3, the latest version of our flagship software management tool suite. The pandemic has put enormous pressure on business leaders to utilize resources wisely while juggling development teams in remote locations. With that in mind, our goal with the newest release was to provide several small, yet powerful improvements designed to save time and increase consistency at every step of the project and portfolio planning process.

Improvements to SLIM-Estimate's Skills Breakout configuration make it easier to fine-tune skill categories and labor rates to support better resource demand planning at both the project and portfolio levels.  Additionally, the tool's API now exposes the Sizing and PI (Productivity Index, QSM's proprietary productivity measure) calculators for users who wish to leverage these features directly from a spreadsheet or other external applications.

SLIM-DataManager, QSM's database repository tool, has new functionality designed to make it easier to validate, understand, and analyze your portfolio of projects. Power editing and enhanced keyword management allow quick changes to multiple projects right from the master project list. Power editing makes it easy to add/update descriptive “tags” used to group projects into relevant categories for benchmarking and estimate calibration.

In addition to the new configuration capabilities, SLIM-Suite 10.3 features new agile enhancements, such as an updated SLIM-Estimate agile template. SLIM-Estimate and SLIM-Control now allow users to instantly toggle the display of agile increment lines between sprints, program increments, or both levels of detail.

Value vs. Cost in Software Sizing and Estimation

Stripped down to the bare bones, value in software estimation measures the functionality that a software product provides to its users (both human and non-human) while production cost measures not just value but the work that is required to deliver that functionality. Software estimates need to account for both. Examples of non-functional cost items include configuration, throw-away code, cloud architecture, and quality requirements. Size measures such as IFPUG and NESMA function points quantify value (delivered functionality) and are recognized as functional size measures. Both measures intentionally ignore technical requirements. They can be very useful when used for asset management, measuring scope creep on a project, or assessing software quality (defect density per delivered unit). For estimating they are an important input; but one that needs to be supplemented to reflect the non-functional cost factors: i.e. what needs to be done behind the scene to create that functionality.

Blog Post Categories 
Sizing Estimation SLIM-Estimate

The “Secret Sauce” in SLIM-Estimate


For over 20 years I’ve been an advocate of using metrics for improving IT processes. Shortly into my career as a COBOL developer, I was introduced to Function Point Analysis; and ever since it’s been the most powerful tool in my toolkit. After all: size matters! Once I learned to quantify the amount of functionality delivered by a project or an application, I could zoom in on cost, effort, duration, productivity, and quality because I now had a normalization factor to perform comparisons (Cost per Function Point, Hours per Function Point, Defects per Function Point, etc.).

Shortly after getting my Certified Function Point Specialist certification, I became obsessed with the different measures and metrics pertaining to software and IT. Soon I became a Certified Software Measurement Specialist, where I learned everything there was to know about how to measure everything there was to measure in software (or so I thought). It’s a pretty powerful feeling being able to help organizations baseline their current capabilities so they could determine if implementing the latest and greatest silver bullet was really going to give them the gains in productivity they had been striving for. 

Blog Post Categories 
SLIM-Estimate Training