Every developer and leader is familiar with project deadlines. They are particularly painful if you’re still attempting a Waterfall approach, but even Agile projects are expected to deliver according to longer-term roadmaps.
This would be easy to do if we knew exactly how much time each feature was going to take. However, we all know from experience that estimating development is hard. This is backed-up by the research: 2003 paper “A review of software surveys on software effort estimation” by Molokken and Jorgensen found that 60-80% of software projects take longer than expected, by an average of over one 30-40%. Furthermore, they found that “there is no evidence that formal estimation models lead to more accurate estimates”.
Ok, so software project effort is difficult to predict, yet the world loves predictions. Are we doomed to have unhappy stakeholders? Should we circle the wagons and “protect our team” ?
Of course not!
There are multiple techniques you can use to excel at executing projects in the face of uncertainty. In my first article on this topic, I’m going to dive into a technique that’s applicable to most projects: Quantifying Confidence.
Quantify Confidence
What’s the difference between an easy one week task and a hard one week task?
Are both really one week?
Of course not. We both know that the hard task is much more likely to take longer than expected.
Sometimes this is even explicit:
“How long do you think it will take to apply that new style to widget X?”
“Well, I think there’s a setting for that, so a couple of hours. We’ve never used that feature before though, if the library doesn’t support it we’d have to rewrite widget X ourselves.”
“How long would that take?”
“Not more than a week or two. But it probably won’t happen.”
Some risks are so large that we really can’t commit to a project schedule until we understand the problem better. However we can’t do that for every little thing. Too often developers and managers instead write down the most likely number then move on.
But wait, I hear you say, I already do better than that. I already capture how long we expect tasks to take on average – so I’d add an extra day or two. I also add a comment to remind us of the risk. That way, if it does go sideways, we at least remember that we anticipated the problem.
Great job! If you’re already estimating as a weighted-average of likely outcomes, you’re way ahead of the curve. You’ve already figured out half of what I’m about to discuss.
However, you can do even better. With either approach above is that the schedule variability has disappeared.
Fortunately there are two tools invited years ago that work very well for capturing this type of risk quickly and summarizing it effective:
- Three point estimates
- Monte Carlo simulations
The way I was taught these ideas in university was pretty formal. In this article I’m going to describe how to use them with very little extra work. I’ve also included a more theoretical background for those who are interested.
Measure Risk: Three Two-point estimates
The first step in capturing risk is to write down two estimates per work item instead of one.
The first estimate should be your usual “most likely” estimate, while the second one should be “worst case”. If you use planning poker with a large team, the largest number anyone raised (excluding serious misunderstandings) can be an acceptable quick proxy for a worst case estimate on smaller tasks.
You’ll probably notice the following pattern:
- For common, every-day, tasks, the worst case estimate will probably be an extra 50% or so.
- For a few tasks, the worst case estimate will be a large multiple of extra work.
Congratulations, you just quantified the task risk.
The traditional technique here is the three-point estimate, but two points are good enough to get started.
Summing Risks: Monte Carlo Simulations
The most obvious thing to do with the numbers you just captured is to sum the two columns: expected and worst case.
For projects with only a few work items this is a reasonable approach. However, for most projects, it will be overly pessimistic. Each worst case estimate should be unlikely, so the odds of hitting every single worst case in a large project should be almost zero.
Mathematicians have some rather complex math to sum probabilities correctly. Fortunately, programmers have a simple numerical solution: the Monte Carlo simulation.
- For each work item, pick a random outcome in between the expected and pessimistic estimates and add them up
- Repeat (2) many times to find out what outcome is most likely.
How To Run a Monte Carlo Simulation
I’ve put together a Google Sheet you can use to run Monte Carlo simulations on two or three point estimates. Just make a copy of the sheet provided and fill in the Expected and Worst Case columns. The first 5 rows in the template example look like this:
| ID | Title | Expected | Worst Case |
|---|---|---|---|
| 1 | Task | 2 | 3 |
| 2 | Names | 0.5 | 1 |
| 3 | Have | 5 | 7 |
| 4 | Been | 0.5 | 5 |
| 5 | Removed | 4 | 5 |
| … | … | … | … |
The sheet has an embedded macro that will run the simulation when updated and give you results that look like this:
If you have more than 100 work items, check the range in the formula just under “Days” to ensure that it covers all your data.
In this example the expected or optimistic completion time is 56 days and the sum of the worst-case values is actually 93 days. However, we can see that both of these outcomes are very unlikely. If you need to report just one number, the 80% confidence value is a good one to use: in this case about 67 days. Alternatively, you might use the 50% and 95% values to say “I expect this project to wrap up in 62 days but it could take 70.”
Practical Tips
Worst Case Estimates
It’s worth mentioning that the best “worst case estimates” really are estimates of the absolute longest amount of time the task could take, without being silly. Do account for that 1% chance you’ll have to rewrite the whole module, but not for being hit by a meteor. You also don’t want to account for the same worst-case scenario multiple times.
Three-Point Estimates
The traditional way to do this is to use three-point estimates: optimistic, expected, pessimistic. I’ve included a second tab on the spreadsheet for this style. It gives you more control and, in theory, better results.
That said, most developers are pretty optimistic when estimating anyway. Using this to optimize my process, I’ve found I can get similar results by setting the expected and optimistic estimates to the same value.
Limits
This technique makes some assumptions about the probability distribution of risks occurring (see the theory section). This works well for adding up a large number of relatively small risks. However:
- When there are a few specific risks that would have a large impact on the overall project, those still need to be managed on a case-by-case basis (or with more sophisticated tools).
- When a single risk has a significant impact on multiple work items, you will need to combine those into a single line in the simulation.
Theory
The theory will give you additional context to understand fine-tune the process. Feel free to skip to the free D4D Monte Carlo Planning Tool if you just want quick results.
Three-point estimates and Monte Carlo simulations rely on two assumptions:
- The estimates represent parameters of a probability function that models the probability distribution of actual effort.
- Each work item has independent risks.
Let’s break that down
Probability Functions
When we use multi-point estimates, we’re saying “I don’t exactly know when it will be done, but it will probably be somewhere in this range.”
Do to math on this, we have to assume a specific probability distribution over the given range: is each outcome equally likely, or are some outcomes more likely than others?
According to the PMI Project Management Body of Knowledge (PMBOK)
* That’s an affiliate link. Feel free to search for it yourself if you prefer not to use Amazon or just don’t like affiliate programs.
The triangular distribution looks like a triangle with the most likely estimate as the peak and the optimistic/pessimistic estimates as the upper and lower bounds. This implies you think there’s almost zero chance of the task going outside of the upper and lower bounds you set. The average value of the triangular distribution is equal to the average of the three estimates: (a+b+c) / 3. This is the distribution I’ve used in the free google sheet.
The PERT Beta distribution is a smooth distribution with a much higher peak at the expected estimate. This is suitable when you have high confidence in your expected estimate or have worst-case estimates that are really out there. The average value of the PERT distribution is ( a + 4b + c) / 6.
As usual, Wikipedia has great entries for both the Triangular and PERT distributions.
Use History
If you are a developer yourself you can choose any distribution tailored to your situation. If you have historical expected and actual effort values for a similar team or situation, you can plot these real examples and select a distribution that fits well. For example, on one team I used a half-normal distribution with the pessimistic estimate set at 2 standard-deviations.
The great thing about numerical methods is that you just need to code the distribution you want and feed it in. If you have enough historical data you can even sample it directly as your random set.
Independent Work Items
The second assumption is that the estimates on the work items are independent. That is, one item going well or badly is unrelated to other times going well or badly.
This assumption doesn’t have to be perfectly true. However, imagine you have a group of work items that will all be short or long based on the same risk. In this case the estimates are tightly coupled and the independence assumption is violated. The easiest way to handle this is to group the work into a single item for the purposes of the simulation.
Monte-carlo simulations aren’t inherently limited to the independence assumption. However, buying or writing a more complex simulation is overkill for most projects.
Put it into action
Starting tomorrow, you can
- Add a “Pessimistic” or “Worst Case” column to your planning system, whether that’s JIRA or a spreadsheet.
- Make a copy of the free D4D Monte Carlo Planning Tool (Google Sheets)
Over the next month, you can
- Capture your confidence explicitly by making two or three point estimates
- Use a Monte Carlo simulation to sum those estimates next time someone asks you how long a major project will take
- Pro tip: If your boss is super-nerdy, she might love the cumulative probability chart. If not, I’d suggest keeping it to yourself and reporting just one or two points.
- Remember to focus on value – Two and Three-Point estimates with Monte Carlo simulations are a helpful tool. However, they won’t actually get the project done. Resist the temptation to spend too much time finessing numbers and trying different scenarios.
Discuss
Use the comments section below to share your successes, and let me know what you think. Do you have some great ideas I didn’t mention above?
The forum is moderated and your first post may not appear until it has been manually approved.