Summary
- Estimating and forecasting in software development requires a probabilistic vs. a deterministic approach.
- Monte Carlo simulation is a mathematical technique that predicts a range of possible outcomes for an uncertain event and can be applied effectively in forecasting
- This technique can be used to set more realistic expectations with customers and clients for things like release dates.
The Monte Carlo simulation is a mathematical technique that predicts possible outcomes of an uncertain event. ‘Uncertain events’ are ones for which there is no single deterministic answer, but a range of possible outcomes. This includes things like cycle time or throughput. The Monte Carlo Method can be used to analyze past data and then predict a range of future outcomes. For example, if you want to estimate the number stories a team can complete in a sprint, you can give a Monte Carlo simulation program your historical sprint throughput data, and it will estimate a range of values. It can produce a result like: 20 stories can be completed in 10 days or less, with a probability of 85%. Or, for Cycle Time, once a story begins development, there is a 50% probability that it will be completed in 8 days or less. The output from a Monte Carlo forecast is made up of 2 parts: a range and a probability. This reflects the uncertainty associated with each metric. Forecasting and estimating in software development requires one to take a probabilistic vs. a deterministic approach, that is, a forecast will have a range of values vs. a single deterministic number.
Definition: Monte Carlo Simulation is a statistical sampling technique used to predict the probability of different outcomes, based on input of randomly selected historical data.
Monte Carlo Simulation Steps
Let’s assume we have a Product Backlog and want to forecast how long it will take (how many sprints) to deliver it. Or, How much can we get done in the next 10 sprints?
There are 3 basic steps:
- Gather historic actual throughput data
- Run simulations based on the historical data
- Extract and visualize the results
Gather the historical throughput data (completed/done stories).
The table below shows the actual number of stories done in the last 6 sprints (If you are not using Scrum, use any period of time for each sample):
| Sprint (Period) |
Throughput (Done Stories) |
|---|---|
| 1 | 10 |
| 2 | 12 |
| 3 | 8 |
| 4 | 9 |
| 5 | 10 |
| 6 | 7 |
Run simulations based on the historical data
Example: How much can we get done in the next 10 sprints? To create a forecast we will randomly select a period from the historical data, by rolling a die, and use the corresponding throughput value as our forecast for the first sprint. We continue selecting a period from the historical dataset and use its throughput value to forecast a value for each sprint. This is illustrated as follows:
Repeat this step for 10 iterations. How much can we get done in 10 iterations: 89 stories.
This is the result after one simulation run.
Extract and Visualize the Results
Repeat this process multiple times. The results from 1500 runs can be visualized on a histogram:
How many stories can we complete with a specific confidence level? For example, how frequently (how many occurrences) are we likely to complete at least 83 stories?
How frequently (how many occurrences) are we likely to complete at least 80 stories?
The probability of any one particular outcome (for example, complete at least 80 stories), is determined by summing up all of the occurrences for any result greater or equal to 80. In this case: 75+110+140+ … divided by 1500, which gives us a probability of 90%. It should be clear from the chart that expectations for higher confidence levels around outcomes will mean lower, more conservative outcomes. The following table has more detail:
For example: what’s the likelihood of completing 80 stories in the defined period? 1500 runs of Monte Carlo result is that we can complete at least 80 stories in 90% of the runs. Every organization and business will have its own acceptable degree of uncertainty for predicting throughput or delivery schedules. Teams must strive to improve their predictability for basic metrics like Throughput and Cycle Time, and reduce uncertainty over time.
Conclusion
To answer the question: How many backlog items or user stories can be completed in the next sprint or specific time period, is better put: How much can be done within a given time period given a level of risk. To answer this question, capture historical throughput data and run these through a Monte Carlo Simulation process as described above. The results give a set of probabilistic forecasts about how much can be done. Each result comes in 2 parts: a forecast range of completed stories plus a probability. For example: we can complete 50 stories in the next 6 weeks with a confidence level of 85%. This approach can be used to set more realistic expectations with customers and stakeholders.