Software Reliability Modeling in the Age of Continuous Integration / Continuous Delivery

Defects Prediction Curve

Quantitative Software Management (QSM) consultant, James Heires, recently discussed the benefits of estimating and forecasting software reliability at RAMS (Reliability & Maintainability Symposium) 2023.  Theme for the conference:  "Artificial Intelligence and Machine Learning (AI/ML) application to our R&M tools, techniques, and processes (and products) promises speed and scale.... When program management instantiates advanced techniques into R&M engineering activities, such as digital design and machine learning and other advanced analytics, it enables products to evolve at a much more proactive, effective, and cost-efficient approach.  Ultimately it facilitates increased speed to market, adoption of new technology, and especially for repairable systems, products that are more reliable, maintainable, and supportable."

How Can I Tell When My Software will be Reliable Enough to Deliver?

Usually when I am online making a payment or using social media, I am not thinking about software quality. But lately I feel like I have been encountering more bugs than usual.  From activities like clicking on a link where I should be able to input my payment information, to doing a search and receiving an error message, or being redirected to a completely different page which had nothing to do with the mission I had set out to accomplish.  These bugs are sometimes frustrating and I started to wonder what could have been done to prevent these from being released into production.

Since I spend a lot of time speaking with people that manage software projects, I have noticed that quality is often one of the most overlooked aspects of a software system. People I’ve spoken with have mentioned that quality is often not even discussed during the early planning stages of development projects, but it is usually a deciding factor when the software is ready to be released and should be considered from the beginning of the project.

Using a tool like SLIM early in the planning stages of a project can help us with these issues. Not only can it provide reliable cost and schedule estimates, but it can also help estimate how many defects one can expect to find between system test and actual delivery. It can also estimate the Mean Time to Defect (MTTD), which is the amount of time between errors discovered.

Software Defect Tracking

New Article - Forecasting from Defect Signals

Forecasting from Defect Signals

On large software development and acquisition programs, testing phases typically extend over many months. It is important to forecast the quality of the software at that future time when the schedule calls for testing to be complete. In this article, originally published in CrossTalk, Paul Below shows how Walter Shewhart’s Control Charts can be applied to this purpose, in order to detect a signal that indicates a significant change in the state of the software. This signal detection is then used to improve mapping of project progress to forecast curves and thereby improve estimates of project schedule.

Blog Post Categories 
Defects Articles Quality

Managing Project Risk through Early Defect Detection

Managing Software Project RiskWith the most recent spurt of inclement weather, there is really no denying that winter is here.  After awaking to about 4 inches of snow accumulation, I begrudgingly bundled myself up in my warmest winter gear and proceeded to dig out my car.  Perhaps the brisk air woke me up faster than usual because as I dug a path to the car, I began to think about software testing, specifically how effective early testing can reduce the chances of schedule slippages and cost overruns.  Allow me to explain.

Being an eternal optimist, I was grateful that the snow I was shoveling and later brushing off my car was light and powdery.  Despite the frigid temperature and large quantity of snow, I realized that it was good that I had decided to complete this task first thing in the morning.  At the time the snow was relatively easy to clear, and had I waited until the afternoon, the sun would have melted enough of the snow to make this task significantly more difficult and time consuming.

"Managing Productivity with Proper Staffing" Webinar Replay Available

Just before the holidays, we hosted our first in-house webinar, "Managing Productivity with Proper Staffing Strategies." Confronted with challenges presented by the current economy, we see more and more systems development groups trying to do more with less.  The ultimate goal is to maximize productivity and minimize defects, but many teams struggle to get there.  It is possible, but the most effective methods used to achieve maximum efficiency are counter-intuitive.  People always think more effort will produce more product.  The fact is using less effort is often more effective.  Presented by industry expert, John Bryant, this webinar explains and proves the correct way to maximize productivity while at the same time minimizing cost and defects. 


John Bryant has over forty years of IT experience.  He spent the last several years using the SLIM Suite of Tools to improve the software development process by properly estimating, tracking, and analyzing software development efforts.  His expertise includes project management, teaching and mentoring from initial project evaluation and planning through construction. 


In case you missed it, you can view the replay of this webinar here

Blog Post Categories 
Defects Team Size Webinars

Calculating Mean Time to Defect

MTTD is Mean Time to Defect.  Basically, it means the average time between defects (mean is the statistical term for average).  A related term is MTTF, or Mean Time to Failure.  It is usually meant as the average time between encountering a defect serious enough to cause the system to fail.

Is MTTD hard to compute?  Does it require difficult metrics collection? Some people I have spoken to think so.  Some texts think so, too.  For example:

Gathering data about time between failures is very expensive.  It requires recording the occurrence time of each software failure.  It is sometimes quite difficult to record the time for all the failures observed during testing or operation.  To be useful, time between failures data also requires a high degree of accuracy.  This is perhaps the reason the MTTF metric is not widely used by commercial developers.

But this is not really true.  The MTTD or MTTF can be computed from basic defect metrics.   All you need is:

  • the total number of defects or failures and
  • the total number of months, weeks, days, or hours during which the system was running or being tested and metrics were recorded.  You do not need the exact time of each defect.

Here is an example.  I will compute MTTD and MTTF two ways to demonstrate that the results are identical.  This table contains defect metrics for first three days of operation of a system that runs 24 hours a day, five days a week:

Blog Post Categories 

Finding Defects Efficiently

Several weeks ago I read an interesting study on finding bugs in giant software programs:

The efficiency of software development projects is largely determined by the way coders spot and correct errors.

But identifying bugs efficiently can be a tricky business, when the various components of a program can contain millions of lines of code. Now Michele Marchesi from the University of Calgiari and a few pals have come up with a deceptively simple way of efficiently allocating resources to error correction.

...Marchesi and pals have analysed a database of java programs called Eclipse and found that the size of these programs follows a log normal distribution. In other words, the database and by extension, any large project, is made up of lots of small programs but only a few big ones.

So how are errors distributed among these programs? It would be easy to assume that the errors are evenly distributed per 1000 lines of code, regardless of the size of the program.

Not so say Marchesi and co. Their study of the Eclipse database indicates that errors are much more likely in big programs. In fact, in their study, the top 20 per cent of the largest programs contained over 60 per cent of the bugs.

That points to a clear strategy for identifying the most errors as quickly as possible in a software project: just focus on the biggest programs.

Nicole Tedesco adds her thoughts:

Blog Post Categories 
Defects Testing Software Reliability