Predictability in software development

Ernst van Waning
QSM
De Corridor 27
3621 ZA Breukelen
the Netherlands

The commercial development of software is not a process that can be done 'by the eye': the long accepted software crisis is now seen as a chronic situation facing the IT industry. Developing software is hard enough but managing software development projects is even harder to do 'by eye'. Tools are needed to show your development process as it really is: in project execution, in plans and bids and in comparison to other projects in the market. Easy to say, but what does it actually mean?

Software development projects are like ocean crossings, you don't reach your goal on gut feelings! Successful crossings are made with instruments that tell you where you are, where you are going and what weather you can expect. Managing software development projects is very similar, without instruments that can tell you what you have achieved and what you can expect, you are like Christopher Columbus. To his credit, Columbus did reach the other side of the world but on his first trip he had no idea how long the journey would take and when he arrived, never actually knew where he was!

The value of measurement instruments

Further evidence of the importance of measurement instruments can be found by moving from ocean crossings to everyday traffic on the roads, as anyone with a speeding ticket will testify.

A speedometer is a device that tells us our speed when driving and it does this in terms of kilometers or miles per hour. Kilometers and miles are meaningful terms to us and it is essential that we talk about speed in a way that everyone can understand. Legal speed limits are the hard reality of such understanding.

The beauty of these instruments is that they translate measurements into meaningful terms. How the instrument does this translation into speed is only of interest to the designer and not to its user. In fact we have used speedometers for centuries and the way in which measurements have been translated has changed over time, but the concept of what speed is has never changed at all. It's the capability of these instruments to convey meaningful information to the user that makes them so useful.

They make us aware of the situation and enable us to communicate this information effectively to others. Measurement instruments provide us with immediate and a meaningful knowledge about the state of our environment. Knowledge about your own performance and the potential risks you can take, are expressions of the old proverb 'know thyself'. Companies that are well informed about their performance tend to be better at project planning, project bidding and project delivery than companies with less self-awareness. This knowledge alone will lead to better customer satisfaction. Possibly the greatest advantage of this approach is that the knowledge of one's own performance is the key to systematically improve it!

A dashboard for software development

During a project execution we need to see immediately if we are on course or on schedule. In the event that we are off schedule, we need to effectively change course and this requires timely and accurate information to diagnose the faults and to make the necessary corrections.

As every project leader will testify during the course of a project, the world or the perception of the world changes, sometimes even a combination of both. This normally results in change requests. Always accepting change requests will quickly damage the project leader's credibility, but always refusing will inevitably lead to the same result. A better method is to produce information that encourages a culture of rational decision-making. For example, an estimate of the consequences from a change request that is based on actual project performance will encourage more rational decision-making.

The tool that can do this will use statistical techniques to evaluate progress. The tool will provide timely warnings that a project may overrun, estimates of a project's end-date, expected costs, expected quality at delivery time, all based on actual project data. This tool would not only inform you about the current status of a project, but it will also help to determine, (based on actual observations), how the project may look in future stages of the project lifecycle. In the diagrams below the black dots represent the observed project data. The hollow dots are the expectations of how the project will develop in the future, based on the observed project data.

This tool allows you to manage a project on actual project data. The data tells you how much time has been spent on what parts of the projects and how much has been produced in that period of time. This is often the type of data collected on a simple time sheet. When this data is entered in a measurement tool, you can quickly see how your project is performing and where attention is necessary if the project is underperforming.

Effectiveness

Companies that start to manage their projects by using metrics never regret the decision. Everyone involved or even interested can quickly see if a project is performing to company expectations, or if action is needed to get it back on track. This makes project discussions more effective, productive and rewarding. Moreover, it will be become clear very quickly if a project needs corrective action and due to the quality of the data, any corrective actions are more likely to produce positive results.

In turn this means that ongoing project discussions and project negotiations take up less of the total time, leaving more time for proactive project management and more focused attention on projects in need. Apart from spending your time more effectively, you will also be better informed on the performance of your projects. Companies that manage their projects in this way report that they have become more effective, delivering more projects on budget, on time and with the expected level of quality. By logic, these are the companies that have a better record of managing their project risks.
Measuring your performance
All companies in some form collect actual project data and this data tells you exactly how your company is behaving. The tool provides additional advantages of calculating important parameters like the performance of your software development projects and the potential stress that is put on the staff in project delivery.

Better plans, happier customers

As humans we have the remarkable capacity to make rational decisions in environments that lack any form of accuracy or clarity. We can give approximate decisions based inaccurate, incomplete and sometimes rather dubious information. But if this is the case, how has man been able to progress and develop in what is an ever increasingly complex world?

Clients expect, if not demand this sort of behavior. They expect bids before projects start, knowing that correct calculations can only be made after projects have been finished. Plans and bids are therefore always uncertain.

Project information

A sudden bright idea has often been the start of a software project. Discussion follows and comparisons are made about the similarities with other development projects. These comparisons, however, inaccurate are the first approximations for a plan.

When other people get interested a somewhat more serious plan is hurriedly made. As there has been some past experience based on a similar project, a feasibility study will not be necessary. There will be Requirements and Design phases and Corrective Maintenance after the first delivery. The system will probably consist of 80% business code, 5% telecom code, and 15% system code. Experience suggests that the system will have 63000 lines of C and there will be no more than 7 people on the project.

We know very little, if anything about this project, but to see its consequences in terms of project parameters would be extremely useful at this early stage. At the very least, an estimate would give us an idea of the duration and the effort (cost) for such a project. Duration and effort are expressed in terms of each other and therefore provide useful measure by trading off duration v effort. There are other useful parameters that may not instantly appear at first sight. For example, do we know our productivity for software development or the stress we work under when we develop software? If we do not use metrics or tools as described in the section 'A dashboard for software development', then we will have no way of knowing these. In this case, we could fall back on available market data to find reasonable values to apply to our project.

With any tool the data collection, calculations and metrics are the all important inputs but equally such a tool must have the capability to present our findings in clear, unambiguous manner to deliver the maximum benefit to the decision makers.

The Staffing and Probability Analysis shows the number of people working on the project during its entire lifecycle, a table with expected values, a graph showing the probability that certain constraints will be met (this is not populated as we currently have no constraints) and pointers (that can be adjusted) indicating productivity, peak staff and lines of code.

The solution presented above is a likely project scenario. This means that at a 50% probability we may need more resourcing but also 50% may need less. In practice, it is advisable to produce plans with a higher guarantee of delivering what they are planning. The table below shows the forecast for the entire lifecycle at 50% and 80% probabilities.

During the meeting where these numbers are discussed someone states that this project should take much less time: a year would be more than sufficient. The same person adds that if we take the maintenance phase for granted, we would be the first to market.

Ok, how does this affect our project?

The results are that the financial cost and the manpower effort are three to four times higher than in the first plan. The stress on the project to deliver within one year will be very high and partly because of this stress the expected quality of the delivery could be lower. Furthermore, the analysis suggests that a time frame of 13 months is more likely than 12.

Market data

It has been said that a plan without data to calibrate it is pure speculation. In our case we can compare our plans with relevant market data.

Below are four graphs and two tables. The graphs plot the lines of code (LOC) against expected duration, expected effort, peak staff and the expected number of errors found during construction. The three lines show market performance, the central line is the average and the other two show one standard deviation above and below the average.

Looking at the duration and the expected number of errors, our plan appears reasonable. In comparison, the plans for a 12-month project (red circled dot) demand a much higher level of effort and a much higher peak of staff, compared to the market data.

Company data

Can we trust our plans if we compare them to our own records? The graphs below demonstrate how our plans compare to projects we have done previously.

The conclusion remains the same as before in that the peak of staff for the 12-month project is, when compared to our own records, extraordinarily high.

A small amount of work can provide a great insight into a project's feasibility. Estimates of duration and effort are very important for decisions, but when compared with market data or even better company data, the confidence will increase that embarking on such a project is a sound decision.

Benchmarking and performance improvement

Clients of a software development company in the Telecom market received complaints that they were expensive and their software contained a high number of errors. To address this, the company decided to benchmark itself with its peers in the market.

The positive points were the staff's high motivation and technical know-how, effective company management, a genuine desire by the staff to contribute to the company and its goals. In contrast a negative was the amount of pressure applied to deliver projects. Due to this pressure, there was insufficient time to test code and no time to document it. The Development Team was completely separate to the Quality Team. Budgets for education and training were small, but the time pressure meant that there was no time for education and training anyway.

The graphs above show trend lines of the company's market. The horizontal axes show line of code. From left to right the graphs show 'the number of errors found before delivery', 'the pressure on the project', 'the productivity per staff member', and 'the duration of the functional design phase'.

Errors found:
Four projects have a high number of errors. The company saw this as a reason for the client to complain about quality.

MBI (Pressure on projects):
Manpower Buildup Index (MBI). MBI is a measure for the change of effort per time unit, or the pressure on the project. The company's projects are all on the high side. Higher MBIs lead to higher project costs and lower quality.

Productivity per staff member:
In contrast to the market data, the company experiences diminishing productivity as the projects get larger. This company has complaints about high prices and low quality. The company spends little time on documentation and used high staff cost rates. This meant that the staff could not access the information it needed through documentation and relied by asking their colleagues. The end result was diminishing productivity throughout the team and an increasing error rate.

Functional design:
Analysis shows that the company spent an equal amount of time on functional design in four of the projects regardless of the size of those projects

The company introduced a series of improvements. The Development and Quality teams started to work together, investments were made in better tooling and plans were based on metrics. Documentation and its benefits were taken seriously even after the design phase.
Soon the staff became more familiar with the code and the work became less hectic.

Management realized that project pressure could be lowered and this resulted in the company achieving lower operating costs and improved quality for the client.

Conclusion

In today's world, to consider crossing the Atlantic Ocean with only the instruments and knowledge that Christopher Columbus had available would be considered both insane and wholly irresponsible. We would only travel the ocean with every available piece of navigation technology and communications equipment to safely guide us.

In the centuries after Columbus we have worked out what data to collect and how to build the instruments that present a journey's data in the most comprehensible fashion. We know exactly where we are on the ocean, what kind of weather we can expect and when we will arrive. Comparable instruments have not only helped us fly around the world, they have even helped us into space. Good metrics and instruments made it possible to plan a journey accurately in advance and to execute that plan with the flexibility to adapt quickly and effectively to unexpected circumstances. These developments are closely linked with the progress and development of science.

We have also learned to measure and interpret our measurements in other areas. Insurance companies translate measurements (counts) into insurance contracts. Like the development navigation instruments the development of insurance instruments is work for specialists (in many cases actuaries). The insurance specialist needs the right data and then must analyze and interpret the findings in order to translate the results into a viable product contract.
QSM (www.qsm.com) did the same with software development, by finding laws and invariants wherever they are present in data regarding software development. These laws and invariants have found their place in SLIM Estimate, Control and Metrics.

The cornerstone of the SLIM product suite is that it is based on a large set of project data and that you collect you own project data (Control) and use them (Estimate and Metrics). These products give you an insight into the performance levels in the market and an objective view of your own performance. When combined these products allow you to effectively manage the risks involved in software development projects.

The use of measurement instruments has had an enormous influence on the development of our society. The use of measurement instruments for company projects has both a major and a positive impact on the companies using them. Measurement is the knowledge and as the knowledge about the work increases there is the beneficial effect that the staff involvement increases also. The goal of all software development activity is where the plan works through to completion, the project delivers to the plan and the company is at all times focused on delivering the project to meet the 'all important' client expectations.