Primera jornada en España especializada en estimación software

1. Overruns or Underestimates?
"... their judgment was based more on blind
wishing than upon any sound prediction; for it is
the habit of mankind to entrust to careless hope
what they long for, and to use sovereign reason
to thrust aside what they do not desire."
Thucydides (c. 460 BC – c. 395 BC)
A Political Perspective on
Software Cost Estimation
Eduardo Miranda
Carnegie Mellon University
October 15th, 2012
© Eduardo Miranda, 2012

2. Estimation politics
 The purposeful shrinking or padding of an estimate to
achieve an objective or favor a course of action not
warranted by the unaltered estimate
– Shrinking consists in reducing the estimate below what the estimator
honestly believes will be required
– Padding refers to the practice of increasing the estimate beyond what
the estimator honestly believes will be required to develop a system
 An adjustment to compensate for what you think others
are or will be doing
© Eduardo Miranda, 2012

3. Loss leader strategy ≠ Target costing ≠ Political
behavior
Loss leader strategy Target costing
© Eduardo Miranda, 2012

4. Price ≠ Budget ≠ Cost ≠ Estimate
 Estimate
 – A range of values within which an
organization believes it is possible for it to
achieve the objectives of the project with a
defined probability
 Price
– What the seller or developer will get for doing
the work. Depending on the strategic intent
of the seller it could be higher than the cost
estimate, with the difference being the
expected profit or mark-up or they could be
lower, a loss, with the hope of entering a new
market, acquire a new technology or recoup
it in follow-up work
 Budget
– A definite amount in terms of effort,
schedule, and resources, hopefully chosen
from the range of possible values defined by
the estimate, to which the budgetee or
executor of the project commits to
 Margin
– The desired profit
 Cost
– The amount of money or resources that has
been used to produce or acquire something,
and hence is not available for use anymore
 Profit (or loss)
– The difference between the price and the
cost
© Eduardo Miranda, 2012

5. Agenda
 The cone of uncertainty revisited
 What is the evidence?
 Successful projects, price and cost, the consequences of
overruns
 What can be done?
 Conclusion
© Eduardo Miranda, 2012

6. The cone of uncertainty
Relative
Size
Barry Boehm,
Range Software
Engineering
Economics,
Prentice Hall,
1981
Todd Little,
Schedule
Estimation and
Uncertainty
Surrounding
the Cone of
Uncertainty,
IEEE Software,
May/June 2006
Plans and Product Detailed Development
Feasibility
Requirements Design Design and Test
Phases and Milestones
© Eduardo Miranda, 2012

7. Some plausible explanations
 Cognitive bias
– Overconfidence
– Anchoring
– …….
 Individual & administrative behaviors
– Student syndrome
– Parkinson’s law
 Political behavior
– Shrinking
– Padding
 All of the above
© Eduardo Miranda, 2012

8. Cognitive bias
Overconfidence Anchoring
Variability and Calibration of Expert Judgment in Software Anchoring and Adjustment in Software Estimation, J. Aranda, 2005
Estimation during the Bid Phase of a Project – An Exploratory
Survey, Pedro Faria, University of Coimbra, 2011
© Eduardo Miranda, 2012

9. Individual & administrative behaviors
Parkinson’s Law Student syndrome
0.1 1.8
0.09 1.6
0.08
1.4
0.07
1.2
0.06
1
0.05
0.8 Normal
0.04
0.6
0.03 Parkinson's
Law
0.4
0.02
0.01 0.2
0 0
10 15 20 25 30
Task duration
© Eduardo Miranda, 2012

10. Cognitive bias vs. political behavior as cause of
chronic underestimations
(Politics)
(Cognitive bias)
B. Flyvbjerg, From Nobel Prize To Project Management: Getting Risks Right, 2006
© Eduardo Miranda, 2012

11. So, what is the evidence?
© Eduardo Miranda, 2012

12. "The Montreal Olympics can no more have a
deficit, than a man can have a baby"
 Montreal won the bid for the 1976 Olympic
Games in May 1970, the estimated budget
was $120 M
 On Jan. 29, 1973 J. Drapeau, Montreal’s
mayor announced that the games would
cost $310 million
 The 1972 Munich Olympics had cost
around $600 million
© Eduardo Miranda, 2012

13. Montreal’s Olympic Stadium and Toronto’s CN
Tower
•Completed in 1976
•Budget 63 million
•40,500 cubic meters of concrete
•Headcount peak 1,500
•Cost per cubic meter of concrete
•Original budget 130 million. $1,555
•Final construction 770 millions
•Total cost including repairs and
interest: 1.5 billion
•Planned completion 1976
•Completed 1987
•400 000 cubic meters of concrete
•Headcount at peak of construction
10,000
•Cost per cubic meter of concrete
$1,925
© Eduardo Miranda, 2012

14. The C-130 Avionic Modernization Program (AMP)
according to GAO
GAO-08-467SP Assessments of
Major Weapon Programs
© Eduardo Miranda, 2012

15. C-130 AMP
According to the C-130
AMP PO the amount of
wiring and the number of
harnesses and brackets
needed for the installation
had been underestimated
by 400 percent
GAO-08-467SP
Assessments of Major
Weapon Programs
© Eduardo Miranda, 2012

16. The EMS Commercialization Project
© Eduardo Miranda, 2012

17. EMS Project
 Migrate 2,000,000 SLOC of application code from
mainframe shared memory to client-server architecture
– Fortran
– Assembly
– Hard real-time
– Custom interfaces to SCADA system
– Complicated GUI
– Safety critical application
– Two organizations that have never worked together and with very
different cultures (public service, business) located 600 km away
– Productizing the system
– Deployment in a foreign country
– Generating documentation
 Constraints
– 8 months from bid to promised date + 2 months ready on-site
– 1.5 M budget (hardware excluded)
© Eduardo Miranda, 2012

18. Sensitivity chart for EMS
Unfeasible
Need to rewrite
CPF MPT
Feasible
© Eduardo Miranda, 2012

19. © Eduardo Miranda, 2012

20. ERP Implementations
Medium Organizations $m
Hardware
–Application, Web, and database servers including 0.8+
storage
Software ERP application Suite License
–HR, Financials, Distribution $3.2+
–1,000 seats
Implementation 9 months to complete pilot
site including process engineering, apps
configuration, and testing $9.3+
–30 external consultants @ $1,200 a day
–30 internal staffers @ $100,000
Deployment
–3 external consultants at 9 sites for 3 months
–9 internal staffers at each site for 6 months
–5 days of user training at an average burdened user $7.5+
salary of $50,000
–3 full-time training staff at an average burdened
salary of $100,000
Total 20.8+
© Eduardo Miranda, 2012

21. The Validation Of A Political Model Of Information
Systems Development Cost Estimating, A. Lederer and
J. Prasad, 1991
© Eduardo Miranda, 2012

22. Software Management And Cost Estimating Error, A.
Lederer and J. Prasad, 1999
Uses Practices Methods Result
Justify using cost- Multiple
benefit analysis
(+) phased
Staff projects (+)
estimates
Monitor project (+) User (+)
management
Schedule Structured
signs-off estimate
projects
Estimate (+) (+)
Select projects prepared Accuracy of
proposal stage (-)
(+) estimates is part Project
of IS performance overruns
Evaluate review
estimators (+)
Evaluate Estimates and
developers actuals are
Charge users (+) audited
Accuracy of Informal
estimates is part Software
of users management and
cost estimating
performance error, A. Lederer
& J. Prasad,
review 1999
© Eduardo Miranda, 2012

23. Other studies
 Any other cost estimation inhibitors? A. Magazinović & J. Pernstål,
Chalmers University, 2008
– The results of this study were used for validation of the results of the Lederer and Prasad study
that was conducted in the form of a questionnaire, answered by 112 software professionals with
a response rate of 28%
– Pressures from manager, users or others to increase or reduce the estimate was fully validated
by the issues found in both cases.
– Removal of padding was partly validated by an issue found in the VCC case.
 Better sure than safe? Over-confidence in judgment based software
development effort prediction intervals, M. Jørgensen et al, University of
Oslo, 2002
– Hidden agendas: Software professionals have goals other than just a high correspondence
between confidence level and hit rate. In particular, the desire to be evaluated as a skilled
software developer may be an important agenda that leads to overly narrow effort Pis.
– Project managers favor narrow intervals and high confidence: Study D indicates that most
software developers preferred effort PIs that were much too narrow or based on a much too high
confidence level, even when they knew that they were much too narrow
 When planners lie with numbers, M. Wachs, 1989
– Planning, however, is not just analytical. We work in the fishbowl of politics and public-policy
making. We serve as staff to politicians, consultants to government bodies, and representatives
of private landowners and real estate developers. These roles are usually associated with
clearly articulated interests. Our agencies, employers, and clients favor particular policies or
programs for reasons that may be derived more directly from ideology, political commitments, or
economic self-interest than from the results of analytical studies.
– I have experienced this conflict between planning as science and planning as advocacy in my
own consulting, and have accumulated dozens of case studies from alumni who return to the
university to talk about their anxieties and conflicts as professionals.
© Eduardo Miranda, 2012

24. Successful projects & the consequences
of underestimations
© Eduardo Miranda, 2012

25. Budget overruns are not the same as project
failures
 Iridium
– 66 LEO satellites designed,
built, launched, and operated
successfully
– Completed on schedule
– Completed below budget -$5B
– Bankruptcy
– Entire system was sold for $25M
in 2000
 Titanic
– Six months late
– $100M budget, final cost $200M
– Financial and creative
blockbuster
– Revenue of $1.8B
© Eduardo Miranda, 2012

26. Even if overruns do not equate to project
failures, underestimations are not free:
 They affect which projects get done
 How much we end up paying for them; and
 Might throw the organization into a vicious firefighting
cycle
© Eduardo Miranda, 2012

27. Which projects get done?
“We have found it isn’t necessarily the best
ones, but those projects for which proponents
best succeed in conjuring a fantasy world of
underestimated costs, overestimated revenues,
undervalued environmental impacts and
overvalued regional development effects”
Machiavellian Megaprojects, Bent Flyvbjerg, 2005
© Eduardo Miranda, 2012

28. Olympic gold
Atlanta Sydney Athens Beijing London Rio
Year 1996 2000 2004 2012 2012 2016
AUS$ Euros
Original US$ 800 US$ 15 US$ 2.4 US$14
1.7 1.5
budget millions billions billions billions
billions billions
Euros US$
Actual US$1.8 AUS$ 6 US $43*
12 15+
cost billions billions billions
billions billions
* Estimated. The actual cost has not been revealed by the Chinese government
© Eduardo Miranda, 2012

29. The Freiman Curve
Budget
29
© Eduardo Miranda, 2012

30. Underestimated projects cost more
Fatigue
Overtime (Eoa)
Learning Effort
contributed by bringing in
additional personnel (Ea)
FTE
Fatigue
Effort contributed thru overtime (Eob)
Communication Overhead (Eco)
Budgeted Effort (Eb)
Ramp-up Coaching
preparation
Td Ta Tl
Tb
© Eduardo Miranda, 2012

31. Recovery cost
Project Parameters
Tb = 12 months
Ta = 1 month
Tl = 1 month
FTEb = 20 people
Teams = 4
Lag = 2 months
Ci = 0.025
Cr = 0.1
Cc = 0.1
Cd = 0.1
Co = 0.0
Co = 0.0
2
b t b t 4a t c u, t
FTEa t, u
2a t
© Eduardo Miranda, 2012

32. Underestimations lead to system inefficiencies
Number of + -
+
worked Fatigue Productivity
hours -
PM +
intervention
+ + Number +
- QA
of Rework
+ activities
defects
+ -
Rate of - Project
progress workload +
+ Other
Resources + projects’
allocated to delays
the project +/-
- -
+
-
Add - Organization Multitasking
+ resources al resource
- + availability
+
Underestimation Management
intervention + Scope + “Band + Organization
Aid”
cuts workload
projects
© Eduardo Miranda, 2012

33. What can be done?
© Eduardo Miranda, 2012

34. Before You Start
 Attachment to issues have career consequences. How do
you expect your advocacy to affect your goals?
 The introduction of an estimation or screening process
will impose limits in otherwise discretionary decisions,
and although people vary to the extent that they seek
power, they rarely relinquish it voluntarily.
 There is not unequivocal substantive solution to the
problem. Other parties have views and solutions of their
own and they are not likely to support you until they
recognize their own views in it.
 Not matter how good the process, it cannot succeed on
its own. It depends on others who can amend, delay,
obstruct and even block its deployment and operation.
© Eduardo Miranda, 2012

35. Estimation Perspectives
Political: There are multiple and distinct interests in
organizations, each pursuing its own, occasionally
parochial, objectives
Cognitive: Our predisposition to judge future events
in a more positive light than is warranted by actual
experience
Epistemological: What do we know? How?
When?
Models: Measurement errors,
assumption errors, and scope
errors
© Eduardo Miranda, 2012

36. Your options
 Reduce political influences
 Compensate for political influences
© Eduardo Miranda, 2012

37. Reducing political influences (The inside view)
 Visibility
– What is the work to be done?
– What risks were considered and which ones were excluded?
– Worst, most likely and best case estimates for effort, cost and schedule
– Safe estimates (x) for effort, cost and schedule
– Staffing curves
– Correlations between work elements
 Sanity checks
– A final check that you should do whenever you solve a problem. It’s an
informal test to see whether your answer makes sense in the context of the
problem. It’s not detailed enough to verify that an answer is correct, but it is
good at detecting logic or assumption errors which produce answers that
cannot possibly be right
 Reliability
– Given the same outputs two persons should arrive at more or less the same
result for the same reason
 Traceability
– It should be possible to understand how the outputs are related to its inputs.
Not black magic.
 Advocates / Non-advocates estimates
© Eduardo Miranda, 2012

38. Benchmarking & Compensating (The outside view)
 Rolls Royce’s Lead Time Model
 Class reference forecasting
© Eduardo Miranda, 2012

39. Lead Time Model – Rolls Royce Controls
A. Powell, Right on Time: Measuring, Modelling and Managing Time-Constrained Software Development , PhD. Thesis, University of York, 2001
© Eduardo Miranda, 2012

40. Class Reference Forecasting
© Eduardo Miranda, 2012

41. Uplift curves
Project sponsor must
present justification for
the uplift not to be
applied
The British Department for Transport
Procedures for Dealing with
Optimism Bias in Transport
Planning
Guidance Document
June 2004
© Eduardo Miranda, 2012

42. Reference Class Forecasting for DoD Projects – Cost Correction
Cost Overruns & Underruns For 45 DoD Projects According to
 C130 – AMP
GAO 2008
14 100.00%
90.00% proposed budget =
12
80.00% 3.9B
10 70.00%
 Acceptable chance
Frequency
60.00%
8
6
50.00% of cost overrun =
40.00%
30.00%
10%
4
2
20.00%  Required uplift =
0
10.00%
0.00%
190%
-25 0 25 50 75 100 125 150 175 200 225 250 275 300 More
 Uplifted budget =
Percentage of Original Budget
Frequency Cumulative % 11.3B
Required Cost Uplift  Latest estimate with
350 half of original units
300 to be delivered =
Original Budget Adjustment
250 5.3B
200  Latest estimate with
150 original number of
100 units to be delivered
50 = 12.5B
0
0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% 90.00% 100.00%
Acceptable Chance of Cost Overrun
Distribution Trendline
© Eduardo Miranda, 2012

43. Reference Class Forecasting for DoD Projects – Schedule
Correction
Schedule Overruns & Underruns for 45 DoD Projects According
to GAO 2008  C130 – AMP
14 100.00%
proposed schedule
12
90.00% = 48 months
80.00%
10 70.00%  Acceptable chance
Frequency
8 60.00%
50.00%
of cost overrun =
6
40.00% 10%
4 30.00%
2
20.00%  Required uplift =
10.00%
0 0.00%
90%
-10 0 10 20 30 40 50 60 70 80 90 100 More
Percentage of Original Schedule  Uplifted schedule =
Frequency Cumulative % 91 months
Required Schedule Uplift
 Latest estimate = 84
160
months
140
Original Schedule Adjustment
120
100
80
60
40
20
0
0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% 90.00% 100.00%
Acceptable Chance of Schedule Overrun Distribution Trendline
© Eduardo Miranda, 2012

44. Applying Reference Class Forecasting to
software development and information systems
acquisitions
Collect Past
completed project data Perfromance project data base
Data
A1
Select reference projects
Reference
query Projects
A2
Generate schedule reference distribution
Class
Reference
Distributions cost reference distribution
A3
cost uplift
Calculate schedule uplift
Uplift
acceptable cost overrun chance
Values
acceptable delay chance
A4
adjusted cost
adjusted schedule
Adjust
Estimates
estimated cost
estimated schedule
A5
© Eduardo Miranda, 2012

45. Summary
 Prices and targets are not estimates. They should be
recognized as such and the cost of following a given
strategy acknowledged
 Underestimations cause inefficiencies
 Political and cognitive biases distort the inputs used by
models and determine the fate of its outputs
 When an estimate can only be explained by delusion or
deception, it is safer to assume the latter
© Eduardo Miranda, 2012

46. The estimation
problem isn’t
unsolvable, we
just need to
understand what
is involved and
perhaps use it as
a competitive
advantage
© Eduardo Miranda, 2012