Session slides for LSSC12 (Lean Software and Systems Conference), Boston. MA. 2012. Sessions discusses how to model and simulate software projects.

1. Using Simulation to Manage
Software Delivery Risk
Effective Modeling and Simulating
Kanban and Scrum Projects using
Monte Carlo Techniques
Troy Magennis
Troy.magennis@focusedobjective.com
@AgileSimulation

2. Schedule Risk
Are we meeting our
commitments?

3. Staff Risk
What team skillset additions
(or losses) have the biggest
impact?

4. Risks
What are the top three risks
jeopardizing delivery?

6. My Mission
Arm my teams (and yours)
with the tools and techniques
to answer these questions
And manage risk more effectively

7. • Currently: Founder and CTO Focused Objective
• Previously: Vice President of Technology (Arch)
• Travelocity and Lastminute.com
• Director Architecture, Corbis
• Various: Automotive, Banking
Contact:
@AgileSimulation and @t_magennis
FocusedObjective.com
Troy.Magennis@focusedobjective.com

9. What, when, who, why
DEFINITIONS, HISTORY AND USE

10. Definition: Model
A model is a tool used to
mimic a real world process
A tool for low-cost
experimentation

11. Definition: Simulation
A technique of using a model
to determine a result given a
set of input conditions

12. Monte Carlo Simulation
Performing a simulation of a
model multiple times using
random input conditions and
recording the frequency of
each result occurrence

13. Simple to more complex model and simulation of a software project
DEMO: VISUAL MODEL SIMULATION
DEMO: MONTE-CARLO SIMULATION
In case of demo disaster, press here…

14. History
Stan Ulam Holding
the FERMIAC
Credits: Wikipedia

15. When to use Monte Carlo Simulation
When there is no correct
single answer (knowable in
advance) or when the
time/effort taken to compute
an answer is beyond realistic

16. When to use Monte Carlo Simulation
When a range of input
conditions can MASSIVELY
alter the final outcome

18. Who Uses Monte Carlo Simulation
High risk industries
Natural resource exploration, insurance,
finance, banking, pharmaceutical…
Software Development == High Risk!
Just look at our reputation, and on-time, on-budget success rate…

19. APPLYING MONTE CARLO
METHODS TO SOFTWARE DEV

20. Why? To Answer Tough Questions…
Date and cost forecasts
Impact of staff hire/loss
Cost of defects
Cost of blocking events
…
And my three 1:1 questions each week!

21. But doesn’t it require estimates?
Yes, but very few…
MUST: Estimate major risks
SHOULD: Column cycle-times
and story counts

22. We need to estimate risk events
**Major risk events have the predominate role
in deciding where deliver actually occurs **
We spend all our
time estimating here
1 2 3

23. Is it Accurate?
1. Gin still equals Gout
2. Doesn’t suffer from the
“Flaw of Averages”

24. Flaw of Averages
50% 50%
Possible Possible
Outcomes Outcomes

25. The average Major issue: Race
release condition, third party
date!!! component failure…
25
Frequency of Result
20
15
10
5
1
Major Risk Event Shifts
Developer Estimates
Delivery Shape Right

26. We need to estimate risk events
**Major risk events have the predominate role
in deciding where deliver actually occurs **
We spend all our
time estimating here
1 2 3
See model example…

27. Risk likelihood changes constantly
95th
Confidence
Interval
1 2 3

28. Risk likelihood changes constantly
95th
Confidence
Interval
1 2 3

29. Risk likelihood changes constantly
95th
Confidence
Interval
1 2 3

30. Risk likelihood changes constantly
95th
Confidence
Interval
1 2 3

31. DEMO: FORECASTING (DATES & COST)
DEMO: SENSITIVITY (COST OF DEFECTS)
DEMO: STAFF IMPACT (STAFF RISK)
In case of demo disaster or no internet, press here…

32. BEST PRACTICES AND TIPS

33. Sensitivity Model
Test (a little)
The Model
Creation
Cycle
Monte- Visually
Carlo Test Test

34. Make
Informed Baseline
Decision(s)
The
Experiment
Cycle
Make
Compare
Single
Results
Change

35. Best Practice 1
Start simple and add ONE
input condition at a time.
Visually / Monte-carlo test
each input to verify it works

36. Best Practice 2
Find the likelihood of major
events and estimate delay
E.g. vendor dependencies,
performance/memory issues,
third party component
failures.

37. Best Practice 3
Only obtain and add detailed
estimates and opinion to a
model if Sensitivity Analysis
says that input is material

38. Best Practice 4
Use a uniform random input
distribution UNTIL sensitivity
analysis says that input is
influencing the output

39. Best Practice 5
Educate your managers’
about risk. They will still want
a “single” date for planning,
but let them decide 75 th or
95 th confidence level
(average is NEVER an option)

40. Q1. Are we meeting our commitments?
Is the likelihood of the models forecast date
increasing or decreasing?
Q2. What are the top three risks
jeopardizing on-time delivery?
Top three items in the Sensitivity report
Q3. What skillsets do your next three
hires need to have?
Skills applicable to the top three WIP limit increases
that cause the biggest reduction in forecast

41. Call to action
• Read these books
• Download the software FocusedObjective.com
• Follow @AgileSimulation
• Learn: http://strategicdecisions.stanford.edu/

42. Questions?
My Contact Details and to get these slides, the
software or the book used in this session -
FocusedObjective.com
Me: Troy.magennis@FocusedObjective.com
Follow: @AgileSimulation and @t_magennis

44. BASICS OF MODELING AND
SIMULATION
Return to main presentation…

45. Manual Kanban Model & Simulation
2 3 4
Design Develop Test
Backlog 1 – 2 days 1 – 2 days 1 – 2 days
Deployed
1
2
5
PLUS: For this manual example, at least 1 defect,
blocking event and scope-creep item.

46. Day 1
Design Develop Test
Backlog 1 – 2 days 1 – 5 days 1 – 2 days
Deployed
1 Day picked at random
2 for this columns cycle-
time range

47. Day 2
Design Develop Test
Backlog 1 – 2 days 1 – 5 days 1 – 2 days
Deployed
2
1 day

48. Day 3
Design Develop Test
Backlog 1 – 2 days 1 – 5 days 1 – 2 days
Deployed
2
1 day

49. Day 4
Design Develop Test
Backlog 1 – 2 days 1 – 5 days 1 – 2 days
Deployed
Added
Scope
2
1 day

50. Day 5
Design Develop Test
Backlog 1 – 2 days 1 – 5 days 1 – 2 days
Deployed
Added
Scope
2
1 day

51. Day 6
Design Develop Test
Backlog 1 – 2 days 1 – 5 days 1 – 2 days
Deployed
Added
Scope
2
1 day

52. Day 7
Design Develop Test
Backlog 1 – 2 days 1 – 5 days 1 – 2 days
Deployed
Added
Scope
2
1 day

53. Day 8
Design Develop Test
Backlog 1 – 2 days 1 – 5 days 1 – 2 days
Deployed
2
1 day
Added
Scope

54. Day 9
Design Develop Test
Backlog 1 – 2 days 1 – 5 days 1 – 2 days
Deployed
Added
Scope
2

55. Day 10
Design Develop Test
Backlog 1 – 2 days 1 – 5 days 1 – 2 days
Deployed
Added
Scope
2

56. Day 11
Design Develop Test
Backlog 1 – 2 days 1 – 5 days 1 – 2 days
Deployed
Added
Scope
2

57. Result versus Frequency (50 runs)
More Often
25
Frequency of Result
20
15
10
5
1
10 15 20
Less Often
Result Values – For example, Days

58. Result versus Frequency (250 runs)
More Often
25
Frequency of Result
20
15
10
5
1
10 15 20
Less Often
Result Values – For example, Days

59. Result versus Frequency (1000+ runs)
More Often
25
Frequency of Result
20
15
10
5
1
10 15 60
Less Often
Result Values – For example, Days

60. Central Limit Theorum
Return to main presentation…

61. Flaw of Averages
50% 50%
Possible Possible
Outcomes Outcomes
Return to main presentation…

62. Software Development Model
4
3 Blocking 5
Events Added
Defects
Work
2 6
Staff
Work
Vacations
1 7
Columns
& WIP Model …
Return to main presentation…

63. Return to main
presentation…

64. SIMULATION EXAMPLES
Return to main presentation…

65. unlikely Forecasts Return to main presentation…
certain

66. unlikely Forecasts Return to main presentation…
50% 50%
Possible Possible
Outcomes Outcomes
certain

67. Return to main presentation…
Sensitivity Report
Actively Ignore for the
Manage moment

68. Return to main presentation…
Staff Skill Impact Report
Explore what staff
changes have the
greatest impact

69. Return to main
presentation…

70. MULTI-MODAL RESULT MODEL

71. Return to main presentation…

72. Return to main presentation…
<setup>
<backlog type="custom" >
<deliverable name=“work">
<custom count="10">Build website</custom>
</deliverable>
<deliverable name="performance issues, add caching" skipPercentage="50">
<custom count="10" >Rework: Performance Issues</custom>
</deliverable>
</backlog>
<columns>
<column id="1" estimateLowBound="1" estimateHighBound="5"
wipLimit="1">Develop</column>
<column id="2" estimateLowBound="1" estimateHighBound="5"
wipLimit="1">Test</column>
</columns>
</setup>

73. Return to main
presentation…