The trap of averages - Techcamp Hamburg
•Télécharger en tant que PPTX, PDF•
0 j'aime•183 vues
Talk given at the Techcamp Hamburg 2019
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à The trap of averages - Techcamp Hamburg
Similaire à The trap of averages - Techcamp Hamburg (20)
Plus de Daiany Palacios
Plus de Daiany Palacios (6)
Dernier
Dernier (20)
The trap of averages - Techcamp Hamburg
- 1. #tchh19 @DaianyMargarita The Trap of Averages How to avoid it in software development estimates April 15-16, 2019 By Daiany Palacios
- 2. Daiany Palacios System Analyst Online Solutions daiany-palacios.com linkedin.com/in/daiany-palacios #tchh19 @DaianyMargarita
- 4. Your head may be on a freezer and your feet in an owen... but on average you are fine #tchh19 @DaianyMargarita
- 5. It‘s just not accurate #tchh19 @DaianyMargarita
- 6. Still, many plans in the world are based on average values #tchh19 @DaianyMargarita
- 7. not the exception Software development plans are #tchh19 @DaianyMargarita
- 8. Team A Team BSoftware House #tchh19 @DaianyMargarita
- 9. Ralph, the Project Manager takes the average estimate of the two teams #tchh19 @DaianyMargarita
- 13. Whenever an average is used to represent an uncertain quantity, it disturbs the results because it ignores the impact of the inevitable variations #tchh19 @DaianyMargarita
- 14. We need to act on a distribution rather than on single values #tchh19 @DaianyMargarita
- 15. Many programs can simulate uncertainty #tchh19 @DaianyMargarita
- 16. For this we use the Monte Carlo Simulation #tchh19 @DaianyMargarita
- 17. Here is how it works 1 2 3 4 Input #tchh19 @DaianyMargarita
- 18. With 85% of probability we will be ready by Feb 25th or earlier It’s a probabilistic forecast #tchh19 @DaianyMargarita
- 19. We need to think probabilistically and not deterministically #tchh19 @DaianyMargarita
- 21. Some important facts for the whole thing to work... #tchh19 @DaianyMargarita
- 22. The Flow of the development process… …needs to be well taken care of #tchh19 @DaianyMargarita
- 23. Predictability is more important… …than the pure speed of delivery #tchh19 @DaianyMargarita
- 24. “Our ability to accurately and reliably answer customers’ question “when it will be done?” is the true measure of predictability” By @danvacanti #tchh19 @DaianyMargarita
- 25. What are the Benefits #Continuous Forecasting #tchh19 @DaianyMargarita
- 26. #ContinuousForecasting does not mean #NoEstimates for us #tchh19 @DaianyMargarita
- 27. We avoid the trap of averages We provide a more realistic forecast We reduce the waste of time We keep a better eye on the course of the team Some benefits we have from #ContinuousForecasting #tchh19 @DaianyMargarita
- 28. Time to Wrap Up #tchh19 @DaianyMargarita
- 29. The Trap of Averages How to avoid it in software development estimates 04 03 02 WHAT IS THE PROBLEM? Plans based on average fail on average, because they ignore the impact of variations. Well-considered projects are doomed to disappointing results. 01 Designed by PresentationGO.com WHAT TO DO? Act on a distribution of values and their likelihood of happening, instead of using single values (an average). HOW TO DO THAT? Use a Monte Carlo Simulation for a probabilistic forecast based on historical data. Adopt a Continuous Forecasting strategy. WHAT ARE THE BENEFITS? Chances of hitting the estimated date are increased. Waste of time is reduced. The team can use its precious time to build awesome things instead! #tchh19 @DaianyMargarita
- 30. Averages are a trap… … and now you know how to avoid it When estimating, remember… #tchh19 @DaianyMargarita
Notes de l'éditeur
- Good morning everyone! My name is Daiany Palacios. Today I am going to be talking about phenomenom that has a wide impact on different fields: the problem of using averages for planning and forecasting. But of course since I am coming from a software delevopment background, I will be narrowing it down to our developers world, and most importantly, I will be sharing with you the solution that has worked out for me and my team within our context.
- Originally I come from Venezuela, but live and work in Germany since 10 years. I belong to one of the development departments of the global IT organization of Kühne + Nagel in Hamburg, the city that has adopted me and become my home.
- Now, once you found out a little bit about me, I invite you to meet Joseph. He was a brilliant statistician who drowned while fording a river that he calculated was, on average, 1 meter deep. He went straight to the trap and fell on it.
- Just like the saying „Your head may be on a freezer and your feet in an owen, but on average you are fine“ is also a trap. It‘s obviously wrong, isn‘t it ? Yes, it might be obvious in this context. But still there are lots of people out there influenced by „average“ prices, „average“ rents and „average“ growth To a strategic investor for example, basing decisions on „average“ numbers is just like developing a medical diagnosis based on the above saying.
- It‘s just not accurate.
- However, it‘s very usual to find all different kinds of plans out there, based on average values. We live in a world of headlines that pump national average house price, average rent prices and the average household debt.
- Software development plans are not too different. Just consider the following situation, and tell me if it rings a bell:
- The next piece of software is to be developed as quickly as possible by two teams and they now have to answer how long it’s going to take. Team Alpha says 5-8 months and Team Bravo 6-9 months.
- But Ralph, the project manager is under pressure to give a date. He decides to simply take 7 months as the average of the two teams.
- Unfortunately, 10 months later the project wasn’t ready yet. And the disappointment was very high. At this point I would like to ask: How many of you have had a similar experience and can relate to this story? How many of you have experienced that a project planned like this was ready on the initially estimated date?
- It‘s just not accurate.
- Sam L. Savage in his book “The Flaw of Averages” makes a simple but powerful statement: plans based on average assumptions are wrong on average. This basic but almost always unseen “flaw”, as the author calls it, shows up everywhere in business, distorting accounts, undermining forecasts, and dooming apparently well-considered projects to disappointing results. Just like Ralph’s project.
- The problem is that whenever an average is used to represent an uncertain quantity, it ends up disturbing the results because it ignores the impact of the inevitable variations. According to Mr. Savage, averages routinely spoil accounting investments, production planning, even weather forecasting. Just like they also routinely ruin delivery forecasts of software development projects.
- We need to act on a distribution rather than on single values, in order to avoid falling into the “Averages” trap. Some executives are already aware of the importance of this, and employ statisticians who estimate the distributions of everything from stock prices to electricity demand. Increasingly, companies are also turning to software-based cures for the flaw.
- Many programs now simulate uncertainty, generating thousands of possible values for a given scenario. The result comprises a whole range of possible values and their likelihood of happening, the frequency distribution. We should better use something like this to get a more accurate forecast.
- In order to do so, what we have to do is to get the help of a Monte Carlo Simulation Hands up who is familiar with the MCS and what it does ? For those who didn’t raise the hand, in short: The MCS is a mathematical tool that allows to account for risk and helps making data-driven decisions. It is based on historical data that is ran through a large number of random simulations to generate the probable outcome of future projects under similar circumstances. We have been using it for the past 3 years and helped us a lot to estimate the probability of future outcomes without just shooting in the dark.
- Here is how it works: When we receive a new requirement, we break it down till we know how many steps we would need to finish it. Then we take the data of how long it took to finish all different working items in a past time frame and give it as input to a Monte Carlo Simulation. The simulation will use a statistical equation that takes the cycle time of a random item in our predefined past time frame and simulates several options of how long the team is likely going to need to finish a random item in the future. Basically it rolls the dice thousands of times in order to get a statistically believable prognosis.
- The result is a distribution on the likely delivery dates for our specified amount of working items. When our product owner comes and asks us how long the new requirement is going to take, our answer goes: with 85% probability we will be ready by <Date> or earlier. This answer is a probabilistic forecast: We provide a range of values, and a probability as an indicator of our level of confidence on this forecast. These kind of forecasts are rather hard to accept, as my experience shows. To understand and internalize the meaning of a probability in the provided answer, it‘s not easy.
- This means a shift in mind-set: we need to think probabilistically and not deterministically. Thinking probabilistically means, acknowledging that there is more than one possible future outcome Which is exactly right when it comes to forecasting, because it is an uncertain thing.
- So, how do we deal with that uncertainty?, that is for me the most important question. (PAUSE) Well, we take a probabilistic forecast as the initial answer, but we update it as we know more, as we get more information. (CLICK 1) We do continuous forecasting. It means that we get used to the fact that the target date changes with the time (probably not so extreme as in that example), but it is a living forecast. So we adopt the habit of sitting together regularly with the product owner and update the initial forecast as we have more information and less uncertainty. That’s the only way we can maximize the chances of hitting the estimated date, and avoid falling into the trap of averages.
- Now, before I talk about the benefits we have got from this strategy, there are some important points that I want to mention, that are key for the whole thing to work:
- Since we started doing continuous forecasting, we noticed that the more disciplined we are with the flow of our development process in our team, the more we can trust the forecast of the monte carlo simulation. For example, it‘s important to make sure that we dont start more work than we can finish, to keep an eye on the throughput and to focus more on finishing work before starting with new items. Kanban principles help us out a lot with this.
- In our organization, predictability is more important than pure speed of delivery. And it‘s probably the same in many other organizations out there. Therefore, the development process has to be „stable“ in order to be „predictable“. That‘s why it‘s important to take good care of the flow.
- The „Actionable Agile Metric for Predictability“ book by Dan Vacanti has been a very useful guide for us to improve the predictability of our development process. I highly recommend it. In his own words: „Our ability to accurately and reliably answer customers‘ question „when it will be done?“ is the true measure of predictability. So the ultimate goal is according to him, to be able to make as accurate forecasts as possible with as little effort as possible. And this can be achieved by using a Monte Carlo Simulation for a probabilistic forecast, following a continuous forecasting strategy.
- Now let‘s talk about the main benefits we have got from following a Continuous Forecasting strategy
- At this point I want to make clear that I am not trying to sell you the idea of #NoEstimates. There is still an estimation we do: the number of work items to be done to implement a new requirement. However, coming up with this value is something we have to do anyway before starting the implementation, since we have to think what steps need to be done anyway. And it‘s more likely to be accurate in this number, than on a time estimate, because there are usually less uncertainty factors. On the other hand, our reality is that we provide a date-range and a probability as a forecast to our product owner, but the further communication with the managers of the Business Units (our customers in our context) is still a single date ! The business managers can‘t get along with a probabilistic forecast, they work with a date and there is no easy way out of it. Our organisation works with projects that need lots of estimates, including delivery dates. So what‘s the point? You might be wondering now…. Right ?
- Well, despite of that reality, we do have a lot of advantages: We avoid the trap of averages. We provide a more realistic forecast with an important piece of information: the level of confidence on the prediction. Which was always missing before. And with this, we contribute to a more accurate project delivery plan. We reduce the waste of time since we don‘t do detailed work-item estimations anymore. We use our time for real development instead. And we feel great about it Forecasting continuously helps us to identify the course we have as a team and more importantly, correct it on time if necessary.
- Plans based on average fail on average. Whenever an average is used to represent an uncertain quantity, it ends up disturbing the results because it ignores the impact of the inevitable variations. (CLICK 1) The problem is that thanks to the trap of averages forecasts are disturbed, and apparently well-considered projects are doomed to disappointing results. (CLICK 2) In order to avoid falling into the trap, we should better act on a distribution of values and their likelihood of happening. (CLICK 3) The Monte Carlo Simulation is a powerful tool that helps us to produce realistic forecasts based on our historical data, and not on averages. The idea is to adopt a #ContinuousForecasting strategy, in which the simulation is run regularly in order to cope with the uncertainty of the initial forecast. It should be a living forecast, that is updated as we know more and have less uncertainty. (CLICK 4) This is the only way we can increase the chances of hitting the predicted dates. At the end, we don‘t waste time with detailed item estimations and use our precious time for building awesome things instead. - If you have any other questions or want to know more about my experience, feel free to contact me, I will be happy to help.
- Just one last thing before we finish: Remember that story from the beginning of Ralph the project manager and his two development teams? Well… keep this in mind: If you ever find yourself in his shoes, please resist the impulse of taking an average value for forecasting. Because it‘s a trap. And now you know how to avoid it.
- Thank you