QSM Resources

Thirty years of research – in one convenient place.

Articles & Papers

In the interest of supporting the software development industry, the following resources are available free of charge. Click a category to jump to the corresponding section:

Articles and White Papers
Risk Management Series
Process Improvement Series
Familiar Metric Management Series 
Newsletters 

Articles and White Papers (PDF and HTML)

In Agile, What Should We Estimate? - Instead of debating #YesEstimate vs. #NoEstimates, we can ask a more useful question: “what should we estimate and why?”  To answer this, we need to distinguish between consumable value and potentially deliverable software. Both are useful concepts but for different purposes.  By choosing small enough developer-sized bites, we can time-box potentially deliverable software to get frequent feedback and review.  But a meal that provides consumable value that satisfies our users and customers must consider the tradeoff of benefits to both the business and the consumer.  In the second article of QSM's Agile Round Table series, Andy Berner explains why setting goals for consumable value and estimating what it takes to reach those goals are both needed to guide the choices every organization needs to make about what to develop and how to allocate resources. December, 2016.

Using Software Project Metrics - Software measurement by itself does not resolve budget, schedule or staffing issues for projects or portfolios, but it does provide a basis upon which informed decisions can be made. Here are examples of how to use metrics to determine present capabilities, assess whether plans are feasible, and explore trade-offs if they are not. This is the third article of a three part series by QSM's Don Beckett for Projects at Work. November, 2016.

A Lead Role in Software Success - When organizations base their decisions on desires instead of data, it usually backfires. Here are four important actions that executives, PMO directors and program leaders can take to improve the predictability and success rate of their software development and enhancement projects. This is the second article of a three part series by QSM's Don Beckett for Projects at Work. November, 2016.

Obey the (Software) Laws - Many business leaders are unacquainted with the wealth of knowledge about how software projects behave. No surprise, they are unable to explain why these projects fail repeatedly, much less do something about it. In the first of a series of articles for Projects at Work, QSM's Don Beckett outlines the five fundamental “laws” of software development that all executives (and teams) should understand and follow. October, 2016.

The Agile Round Table - For well over a decade, agile software development methods have been adopted by a wide variety of software organizations across the globe.  QSM has worked with these types of software organizations for more than 35 years to establish data-driven, defensible estimation and lifecycle management practices as the foundation of quality software projects and products. The QSM Agile Roundtable was formed to provide a platform to brainstorm the role of estimation in agile environments, and chart a path toward better understanding for all stakeholders.  A mixture of long-standing and newer customers shared their questions, challenges, and experiences to answer the big question, and effectively communicate the relevance and benefits of scope-based estimation.  This article by QSM's Laura Zuber is the first of the QSM Agile Round Table series of publications that will present specific concepts and practices that connect SLIM and agile, creating common ground for the benefit of all.  It is our hope that this series will answer some of your questions, and that you will share your thoughts. October, 2016.

Estimation Center of Excellence - Why do so many companies fail at software development projects? More often than not, they haven’t built a foundation of process, people and tools to accurately plan and estimate. An Estimation Center of Excellence is a great starting point to bring these components together and maximize their benefits. In this article for Projects at Work, Larry Putnam, Jr. describes how all of these components work together to help organizations achieve software project success. September, 2016.

How a Center of Excellence Can Help Teams Develop Excellent Software - The ways that enterprises handle software development have changed immensely over the past couple of years. But as many organizations are upending traditional business cultures as they strive for greater collaboration, some core principles remain the same. Business stakeholder requirements need to be delivered within a reasonable timeframe and budget, with a good user experience and solid return on investment. By implementing an Estimation Center of Excellence, organizations can ensure that their projects remain on track, even (or perhaps especially) in highly agile environments. In this article originally published in SD Times, Doug Putnam outlines best practices for establishing an Estimation Center of Excellence. June, 2016.

Sizing Matters by Jay Daniel. Agile is about adapting to change, not completely abandoning documentation or dismissing helpful planning and estimating inputs. In this article for Projects at Work, QSM's Jay Daniel explains how the benefits of an agile approach can shine brighter when used in conjunction with a fundamental development practice such as sizing. May, 2016.

5 Core Metrics to Reduce Outsourced Project Failure by Joe Madden. Outsourcing was supposed to make government IT executives’ lives easier. Yet in too many cases, it's had the opposite effect, leading to cost overruns, inefficiencies, and solutions that do not work. In this article for GCN, QSM's Joe Madden explains how the five core metrics of software estimation make a powerful tool that can be used at each phase of the software acquisition life cycle to help government IT program managers make more objective, quantitative decisions. April, 2016.

Avoiding a Doomed Software Project by Checking the Staff Build-Up Plan by Gene Shuman. The staff build-up plan defines how many, what kind, and when staff are needed for the entire project. Too many or too few, bringing them on too early or late, employing the wrong mix of expertise or experience - avoiding all these pitfalls with a staff build-up plan will ensure a successfully staffed project. Reviewing proposals for a complex project, such as major software development or support, is a challenging activity. Since labor is the major cost and feasibility determinant for such projects, requiring the submission of a "staff build-up plan" and verifying its realism is crucial in determining whether a proposed project can realistically succeed. In this article for Contract Management Magazine, Gene Shuman identifies the key components of an effective staff build-up plan. December, 2015.

10 Steps to Better Metrics by Carol Dekkers. An effective software measurement program is a long-term investment, not a quick fix. In this article originally published in Projects at Work, Carol Dekkers identifies 10 steps to ensure your organization's metrics deliver a positive return on that investment, from more accurate cost and schedule estimation, to streamlined processes and better insights into current and future commitments. July 2015.

Full-Circle Estimating by Doug Putnam and Taylor Putnam-Majarian. While creating estimates is a fundamental step toward improving productivity on software development projects, it is not enough. In "Full-Circle Estimating," originally published on Projects at Work, Doug Putnam and Taylor Putnam-Majarian present a full-circle model that organizations can apply to track actual performance against estimates, reforecast when significant changes occur, and then continually refine the process through post-mortem assessment. July, 2015.

Failing with the Best Intentions by Doug Putnam and Taylor Putnam-Majarian. Enterprise application capacity planning is a difficult juggling act. On one side of the equation you have business demand, looking for innovative technology to help improve business performance and increase profitability. The IT organization stands on the other side of the equation, responsible for satisfying these demands. The capacity of this team is limited by the organization’s facilities, the number of developers and their specific skills, and the infrastructure and tools they use. This leaves the business and technology executives in the unenviable position of trying to balance the demand for IT development with the current capacity levels. In this article for Software Magazine, Doug Putnam and Taylor Putnam-Majarian demonstrate how top-down parametric estimation can be leveraged by organizations to manage capacity and demand effectively. April, 2015.

Estimating Before, During, and After the Software Project by Larry Putnam, Jr. A common misperception is that an estimator’s job is done after a software project’s parameters are set. On the contrary, software estimation should be conducted throughout the project lifecycle to reflect inevitable changes and to improve estimates on other projects. In this article, originally published in Projects at Work, Larry Putnam Jr. identifies three ways to maximize estimating efforts — before, during and after your project is complete. October, 2014.

Forecasting from Defect Signals by Paul Below. On large software development and acquisition programs, testing phases typically extend over many months. It is important to forecast the quality of the software at that future time when the schedule calls for testing to be complete. In this article, originally published in CrossTalk, Paul Below shows how Walter Shewhart’s Control Charts can be applied to this purpose, in order to detect a signal that indicates a significant change in the state of the software. September, 2014.

A Case Study in Implementing Agile by Taylor Putnam. This case study for Agile Connection by Taylor Putnam serves as an example of how adopting agile can be extremely beneficial to an organization, as long as situational factors are considered. Adopting a new development method is a strategic, long-term investment rather than a quick fix. As this article shows, making deliberate, fully formed decisions will ultimately lead to better outcomes. August, 2014.

Counting Function Points for Agile / Iterative Software Development by Carol Dekkers. Function points are proven to be effective and efficient units of measure for both agile/iterative and waterfall software deliveries. However, inconsistencies come to light when comparing function points counted in agile/iterative development with those counted in waterfall or combination development . These inconsistencies can create confusion for cost, productivity, and schedule evaluations that span multiple software delivery methods. This paper, recently published on IFPUG's Beyond MetricViews by QSM Consultant Carol Dekkers, seeks to marry International Function Point Users Group (IFPUG) definitions with equivalent concepts in agile/iterative processes in order to create a basis for consistent comparison. August, 2014.

Traits of Successful Software Development Projects by Larry Putnam, Jr. Enough already with Healthcare.gov and the many (many) other high-profile IT project failures; let’s talk about government software projects that actually worked. Successful software projects are no accident. Best-in-class government IT projects share common traits that agencies can use to ensure success. In a recent article for Government Computer News, Larry Putnam, Jr. leverages data from from the QSM Database to identify best practices for successful government projects. July, 2014.

Set the Stage for Software Project Success by Don Beckett. Management decisions made before a software project is underway are a significant factor in determining whether it succeeds or fails. In a recent article for Projects at Work, QSM's Don Beckett identifies seven principles, based on comprehensive studies, that leaders must support and uphold to help create an environment in which projects can succeed. Ignoring them practically guarantees failure. May, 2014.

Software Estimation: How Misperceptions Mean We Almost Always Get It Wrong by Carol Dekkers. In this highly-discussed article for Dr. Dobb's, QSM's Carol Dekkers asks a tough question: why are we so woefully poor at estimating software projects? It's a tough pill to swallow considering software developers are among the smartest people on the planet, often boasting advanced degrees in mathematics, engineering, or computer science. Yet study upon study cites that less than one-third of projects are delivered on time or on budget. The problem of software project estimation is not straightforward. To get the heart of the issue, Carol Dekkers takes us through the five top misperceptions about software estimating, and what we can do to address them. April, 2014.

Big Agile: Enterprise Savior or Oxymoron? by Larry Putnam, Jr. We know agile works well for small teams and small projects, but monster enterprise projects often require greater capabilities than a small team can provide. So why not scale up agile teams to maintain the cost and efficiency benefits of the agile process while accessing the necessary manpower to pursue complex global projects? On the surface, it makes sense, but what if agile only works when teams and projects stay relatively small? That's the question most CIOs want answered before investing scarce time, energy, or resources chasing the big agile paradigm. In this article originally published on Agile Connection, QSM's Larry Putnam, Jr. turns to cold hard data from completed projects in the QSM database to determine whether big agile is "enterprise savior or oxymoron." March, 2014.

Project Clairvoyance by Larry Putnam, Jr. Can advances in data-driven estimation turn software project failure into a distant memory? Well, if learning from experience is the key to success, imagine what you could do with real-time access to three decades of research, thousands of projects and more than 600 industry trends. In this article, originally published on Projects at Work, QSM's Larry Putnam, Jr. identifies key benefits of employing estimation best practices for project success. January, 2014.

Constant Velocity Is a Myth by Andy Berner. Is your agile team’s velocity constant from sprint to sprint? No? That’s not a surprise. Many teams assume that their velocity will be constant. In this article, the third in a series recently published on Projectmanagement.com, QSM's Andy Berner explains why that’s not the right expectation--and how that affects how you use this metric. December, 2013.

Ready, Set, Go...and Ready Again: Planning to Groom the Backlog by Andy Berner. In an agile project, the backlog (the prioritized set of requirements) is the main input to iteration planning. For an agile project to progress smoothly, the backlog must be groomed and ready for each sprint. That work must be included in your project plan. In this article, the second in a series recently published on Projectmanagement.com, QSM's Andy Berner gives you five points to consider when planning that work. 

An Analysis of Function Point Trends by Don Beckett. Function point analysis has played an important role in software measurement and analysis for 30 years. This study looks at the QSM software project database and examines a set of validated projects counted in function points that have completed since the year 2000 to see what they tell about productivity, schedule, and staffing. We are fortunate to have several thousand projects in this sample to work with as this allows us to parse the data many different ways and still have enough projects to be statistically significant. For this study only unadjusted function points were used. August, 2013.

Is It Bigger than a Breadbox? Getting Started with Release Estimation by Andy Berner. It’s becoming clear to organizations adopting Agile methods that one still needs to estimate how long a project or a release of a product will take. It won’t suffice for businesses to simply take guesses or accept unreasonable constraints. We must be able to derive credible estimates, based on a history of similar projects. But how can we estimate a project in advance, while still maintaining the ability to manage the backlog in an Agile manner? In this article, Andy Berner answers that question, compares release-level estimation to the techniques used for iteration estimation, and gives some pointers on getting started with release estimation in an Agile environment. Originally published on Projectmanagement.com, July, 2013.

Does Agile Scale? by Larry Putnam, Jr. What happens to Agile projects when they’re forced to scale to the size of major government enterprise initiatives? In this article, originally published in the May-June 2013 issue of Modern Government, Larry Putnam, Jr. looks at 93 Agile projects completed between 2002-2012 to see how these projects fared as their sizes increased. July, 2013.

Data-Driven Estimation, Management Lead to High Quality by Kate Armel. Quality assurance comprises a growing share of software development costs. To improve reliability, projects should focus as much effort on upfront planning an estimation as they do on remedial testing and defect removal. Industry data show that simple changes like using smaller teams, capturing average deviations between estimates and actuals and using this information as an explicit input to future estimates, and tuning estimates to an organization's historical performance result in lower defect creation rates. Access to accurate historical data helps projects counter unrealistic expectations and negotiate plans that support quality instead of undermining it. Originally published in Software Quality Professional, March, 2013.

Predictable Change: Flexing the Five Core Levers of Software Development by Andy Berner. In this white paper, Andy Berner introduces the key metrics used to predict what it takes to do a new project and some of the issues you’ll encounter when moving from Agile iteration planning to planning new projects and releases. September, 2012.

The Difference Engine by Phillip Armour. Originally published in Communications of the ACM, "The Difference Engine" focuses on building teams of differently skilled people. The article is partly based on University of Michigan Professor of Complex Systems, Scott Page’s book, The Difference, which shows the power of cognitive diversity in building systems and solving problems. January, 2012.

Technology Can Only Do So Much by Kate Armel. Kate Armel examines the applicability of Fred Brooks' age-old maxim - “Adding manpower to a late software project makes it later.” - in today’s IT development environment. Using data from the QSM Database and research performed by QSM’s Don Beckett, Kate analyzes projects sized from 50K to 500K ESLOC to determine whether project size has any effect on the point at which adding more people causes a project to come unglued. Is Brooks’ statement still universally true? The results might surprise you… March, 2011.

Data Mining for Process Improvement By Paul Below. What do you do if you want to improve a process and have 100 candidate predictor variables?  How do you decide where to direct causal analysis effort?  Similarly, what if you want to create an estimating model and you have so many factors you do not know where to start? Data mining techniques can be used to filter many variables down to a vital few to attack first or to build estimating models to predict important outcomes. Originally published in CrossTalk Magazine, January 2011.

Making The Right Choices For Healthy Software - By David Rubenstein.  SD Times looks at the latest QSM research into best- and worst-of-class projects to see how successful software development managers estimate, plan, and manage projects. January 2006.

How QSM Products Support PMI® Knowledge Areas A convenient matrix showing how QSM tools support different PMI® Key Knowledge Areas.

SLIM and Christopher Columbus - By Ernst Van Waning.The commercial development of software is not a process that can be done 'by the eye': the long accepted software crisis is now seen as a chronic situation facing the IT industry. Developing software is hard enough but managing software development projects is even harder to do 'by eye'. Tools are needed to show your development process as it really is: in project execution, in plans and bids and in comparison to other projects in the market. Easy to say, but what does it actually mean? August 2005. PDF version
 

"Ensuring Delivery of Highly Reliable, Complex Software Releases" - By Jim Greene.A case study from major telecom supplier demonstrates how to estimate, plan, and manage distributed subsystems that are developed separately, then integrated to form a release. Examines the challenges of delivering a distributed, high-reliability system. October 2003.

"Telecom Software Benchmarks: 10 Years Apart" - By Jim Greene.Benchmarks taken 10 years apart measure the benefits of process improvement initiatives and the impact of increased time-to-market pressures on a major telecom supplier. September 2003.

"How Much Software Can Be Built In A Year " - By Doug Putnam.In today’s world, customers expect to see software projects delivered in increasingly shorter cycles. The maximum time most IT customers seem willing to wait is about one year. In this fast-paced environment, the question becomes: just how much software can a typical IT shop create in 12 months?March 2002.

"What I Did Last Summer: A Software Development Benchmarking Case Study " - By James T. Heires.This article describes a vendor-supported benchmarking study of an applications development department. The study established a quantitative performance baseline of the organization and compared it to industry trends. October 2001.

"QSM Software Research Bulletin" - By Doug Putnam.Results of a long-term productivity study covering the period from 1982-1997 in condensed format. November 2001.

"Software Quality Assurance (SQA) of Management Processes Using the SEI Core Measures" - By Jim Greene.Using the right metrics for the three key management processes - benchmarking, estimation and risk analysis, and project tracking and control - to deliver a quality product. September 19, 2001.

"Build in Quality" - By Lawrence H. Putnam Sr. and Ware MyersQuality is the positive side of quality-defect continuum. The use of metrics is fundamental to allow sufficient time and effort to build a quality product. April 2001.

"Build it Faster!" - By Lawrence H. Putnam Sr. and Ware Myers How can we build software faster? Specifically, how can metrics help reduce time-to-market? April 2001.

"Productivity Statistics Buck 15-Year Trend" - By Douglas Putnam.  Examines the effects of new technologies such as ERP, object-oriented development, and Internet/e-commerce on long-term productivity trends from the QSM database.  July 2001.

"Software by the Numbers: an Aerial View of the Software Metrics Landscape" - By Michael Mah and Lawrence H. Putnam, Sr.   As we muse on what 'new metrics' apply in the ever-changing world of software, we should also be asking ourselves the more basic question, 'What will we do with metrics?' . Many organizations don't have the basics down before they pile on measure after measure.  This article originally appeared in Vol. 10, no. 11 of AMERICAN PROGRAMMER. 1998.

"Linking the QSM Productivity Index to the SEI Maturity Levels" - By Lawrence H. Putnam, Sr.   This paper explores the correlation between QSM's Process Productivity Index (PI) and SEI CMM level.  July 2000.

"Purchasing Software Intensive Systems Using Quality Targets" - By Jim Greene.  DOD process for tracking reliability against quality targets.  November 2000.

"16 Critical Software Practices For Performance-Based Management" -  The Airlie Software Council was formed in late 1994 under the coordination of Dr. Norm Brown of the DoD Software Program Managers Network. Its mission is "to identify best software practices in Government and Industry, and to cause these best practices to be implemented in DoD software management, development and maintenance".

"What We Have Learned" - By Lawrence H. Putnam and Ware Myers.  Lessons learned from studying the SEI core metrics and process productivity (PI).  Published in Crosstalk, April 2000.

"Without Software, No Megatrends" - By Lawrence H. Putnam and Ware Myers.  Improving software productivity is the key to supporting the megatrends of the future.

"Key Things We Have Learned" - By Lawrence H. Putnam and Ware Myers.  A retrospective view on how Larry Putnam's seminal thoughts (and one new observation) on software development have held up over the past 25 years.  February 2000.

"The Princess and the Pea" - By Lawrence H. Putnam and Ware Myers.

"A Throughput Measurement Procedure Using SLIM" - By Lawrence H. Putnam.  How to calculate and demonstrate the value of staffing and process improvement strategies using SLIM-Estimate.

"A Process for Costing Requirements Changes" - By Mike Ross.

"Analysis of On-Board Spacecraft Software Development" - By Jim Greene.

"Avoiding the Premature Delivery of Software" - By Jim Greene.

"Estimating Software Size and Uncertainty" - By Jim Greene.

"Estimating When You Are Moving to New Technologies" - By Doug Putnam.

"First, Get the Front End Right" - By Lawrence H. Putnam, Michael Mah, and Ware Myers.

"Getting a "RUNAWAY" Software Development Under Control: A Case Study" - By Jim Greene.

"Home Runs in Management Science" - By Doug Putnam.

"Implementation of a Software Project Office at Honeywell Air Transport Systems" - By Mike Ross.

"Independent Research on Software Reuse" - By Mike Mah and Ira Grossman.

"Larry Putnam's Career in Software Estimation" - By Larry Putnam, Sr.

"Managing Major Distributed Software Development" - By Jim Greene.

"Measures For Software Acquisition" - By Jim Greene.

"Product Review -- SLIM-Metrics®: A Powerful New Repository and Analysis Tool" - By James T. Heires, Rockwell Collins, Inc. See also ADT Magazine, June 1998 issue.

"QSM Reliability Model." - By Doug Putnam.

"Size Does Matter: Continuous Size Estimating and Tracking" - By Mike Ross.

"Sizing and Controlling Incremental Development" - By Jim Greene.

"SLIM Historical Data Form"

"SLIM Sizing and Estimation Input Form"

"Software Process Improvement: Management, Commitment, Measures, and Motivation"- By Jim Greene.

"The (Almost) Perfect Software Project Using the SEI Core Measures" - By Jim Greene.

 Back to Top

Risk Management Series (HTML)
 

"Haste Makes Waste When You Over-Staff to Achieve Schedule Compression" - By Doug Putnam.
"Simple Project Tracking Approach Puts You in Control of Your Project" - By Doug Putnam.

Process Improvement Series (HTML)

"You Need the Right Map to Know Where You're Going With Process Improvement" - By Doug Putnam.
"Team Size Can Be the Key to a Successful Project" - By Doug Putnam.

Familiar Metric Management Series (PDF)

The Familiar Metric Management Series was written by Larry Putnam, Sr. and Ware Myers for Cutter Consortium's IT Metrics Strategies

(32) Effort, Development Time, and Defects Interact
(31) Get Yourself a Little Whip!
(30) Multiple Uses of Software Metrics
(29) Year 2000 Turns the Mythical Man-Month on its Head
(28) Small is Beautiful-Once Again
(27) QSM Database Shows Drop in Productivity
(26) Management Vision Precedes Metrics
(25) "Let Me Count the Ways"
(24) Experience the Wisdom That Goes With Knowing Where You've Been
(23) Know What You Have to Do
(22) Leonardo da Vinci Knew Feasibility
(21) Come to dinner, Henry!
(20) From Work to the Bit-Stream
(19) "The software cost estimation problem is solved!"
(18) What Will the Year 2000 Fix Cost Me?
(17) Productivity of the Reuse Organization
(16) Software Reuse
(15) The Future of Metrics in Software Development
(14) Produce More Systems
(13) The Effort-Time Tradeoff: It's In The Data
(12) The Power of the Tradeoff
(11) Negotiate from a Fact-Based Position
(10) Evaluating a Technology
(9) Fads
(8) Time-to-Market
(7) Executive Backing
(6) Software Adds Business Value
(5) Time Boxing
(4) Size
(3) Reliability
(2) Familiar Metric Management +
(1) Familiar Metric Management
 

 Back to Top

QSM Newsletters

QSM periodically publishes a newsletter aimed at keeping our current and prospective customers informed about new developments and issues relevant to software project management. Back issues of our newsletter are available as PDF files for download, and can be viewed with the Adobe Acrobat® Reader. Click on the issue number to download the desired newsletter.

February 2012 Newsletter
November 2011 Newsletter
April 2011 Newsletter
February 2011 Newsletter
April 2010 Newsletter
January 2010 Newsletter
October 2009 Newsletter
QSM Launches New Website and Blog
March 2008 QSM News Alert
January 2008 QSM News Alert
October 2007 QSM News Alert
May 2007 QSM News Alert
September 2006 QSM News Alert
July 2006 QSM News Alert
April 2006 QSM News Alert
February 2006 QSM News Alert
December 2005 QSM News Alert
August 2005 QSM News Alert
November 2003 QSM News Alert
July 2003 QSM News Alert
QSM Perspectives Volume 20 Number 2 (Fall 1997)
QSM Perspectives Volume 20 Number 1 (Spring 1997)
QSM Perspectives Volume 19 Number 2 (Winter 1996)

 

Back to Top

 

PMI® is a registered mark of Project Management Institute, Inc.