1. Psychological Impacts on Judgment
in Cost Estimation
Jordan Garner
UC Davis (JPL Summer Hire)
Art B. Chmielewski
Jet Propulsion Laboratory
California Institute of Technology
September 12, 2011
1

2. Special Thanks to
• Dr. David Ullman of Robust Decisions and Oregon State
University for his assistance with the web experiment and
continued support of this novel research.
• Prof. Don Forsyth of the University of Richmond for his expert
consultation on socio-psychological effects in decision making.
2

3. 258% 275% 1100%
100%
1500%
Bad Cost Estimates
Happen
1400%
600%
2200%
3
220% 1400%

4. Overruns Start with
Bad Initial Cost Estimates
• Bad cost estimates are in every sector of business
world: construction projects, movie
business, transportation projects, military
programs, aerospace, etc.
• Bad cost estimates know no borders, race, sex or
century.
4

5. Causes of Overruns
• Overruns start with flawed initial cost estimates and
inadequate reserves.
• However, the post mortem analyses give less blame to the
estimating than to failures in execution such as:
– Changes in scope and requirements
– Inadequate communication
– Government and contractor intervention
– Unforeseen technical issues
– New technology
– Acts of god
• Specific technical reasons for overruns seem to be more palatable
than poor cost estimates.
5

6. Are Estimates Getting Better?
“For the past 70 years, for which data on cost estimation is
observable, no significant improvements in
forecasting, estimating or prediction a project’s cost have ever
been made. This is despite the increase in awareness of past
estimation inaccuracy, new strategies of estimation, the hiring of
more experts to help the estimation process, inventions solving
past technical and communication issues.”
– Prof. Bent Flyvbjerg, at Oxford University's Saïd Business School
6

7. Unaccounted Psychological Effects?
• Thesis: Could humans be prone to psychological
factors that make them truly and honestly believe in
poor estimates?
• We conducted a simple experiment to test and
quantitatively measure the power of psychological
fallacies on people’s ability to make estimates.
7

8. Overheard During Cost Estimating:
• “I have a bogey of $400k. Please give me your own estimate.”
• “We will hold 30% reserve for you.”
• “I sent you a WBS cost table. Can you fill it in?”
• “We need your best estimate by Friday.”
• You have an allocation of $1.3M, can you give me an
estimate?
Our simple experiment proved that the above common costing
phrases guarantee overruns!
8

9. Dishwashing Experiment
• Participants in the on-line experiment were asked in
different ways to estimate the time needed to
perform a simple task – washing the dishes shown on
the next chart.
9

10. 10

11. Psychological Effects Tested
• 5 psychological effects were tested :
1. Anchoring
2. Q&A Mismatch
3. Decomposition
4. Reserve Comfort
5. Planning Fallacy
• Every respondent to the survey was randomly asked one
of several questions testing different psychological
heuristics or fallacies.
• 507 volunteers participated: 142 JPLers, 305 college
students and 60 other adults. ~2300 data points were
collected.
11

12. All answers were graphed and
analyzed to establish conclusions
90
80
70
60
50
40 upper Standard Deviation
estimate
30
lower Standard Deviation
20
10
0
Psychological category
12

13. Effect #1: Anchoring
The objective was to test how easily influenced
people may be by a wrong answer – “the anchor.”
The anchor set asked:
Estimate how many minutes it will take you to clean the
kitchen. One respondent estimated that it will take
about 10 minutes to finish cleaning up. He may be
wrong of course.
13

14. 1. Anchoring Results
• The nominal value was 30 min, the anchored case 25 min.
• The “best case scenario” estimate (described later) was 27
min which was 2 min LONGER than the anchored result.
• The result points out that it is very easy to dramatically skew
the estimates by asking anchored questions, such as: “We
would like you to come in around $6M”, “I have a bogey of
$400k for you”, “the last robot arm we built cost $7M”…
14

15. Effect #2: Q&A Mismatch
The purpose was to test if there is a mismatch between the type
of estimate expected and provided.
Different participants were asked:
• Estimate how many minutes it will take you to clean the whole
kitchen.
• There is a 50% chance that you will finish this task within __ min
• There is a 75% chance that you will finish this task within __ min
• There is a 99% chance that you will finish this task within __ min
15

16. 2. Q&A Mismatch Results
• The 50% confidence estimate was 31 min. The nominal
estimate was 30 min. People unconsciously interpret the
nominal as the 50% case, meaning that you will exceed your
estimate in half the cases!
• However, when a manager asks for an estimate he/she
expects a much more reliable result, possibly in the 80%-90%
confidence range. This points out that there is mismatch
between the expectation and the answer.
16

17. Effect #3: Decomposition
The objective was to test if decomposing the project into
smaller pieces and deeper levels of a WBS improved accuracy
of the estimate.
Estimate decomposition was simulated by asking:
1. How many minutes will it take to clean all the plates and the
sets of silver?
2. How long will it take to clean the sets of coffee cups and
saucers?
3. How long will it take to clean the bowls?
4. Etc.
17

18. 3. Decomposition Results
• Decomposition average was 31 minutes, just one
minute longer than the nominal average (30 min).
The attempt at becoming more accurate by cutting
up the project was not accomplished.
• Decomposition, at least in this case, was more time
consuming than helpful.
• Deep decompositions provide more detail but
compound psychological effects.
18

19. Effect #4: Reserve Comfort
This question tested the realism of “a comfortable” reserve.
Respondents were asked:
1. I am 90% sure that the time it will actually take to clean the
kitchen is within plus or minus __ minutes from my estimate.
19

20. 4. Reserve Comfort Results
• The reserve for 90% confidence was 8 min or 28%. The 25-30% seems to be
the magical intuitional comfort level that is used by many industries.
• When a manager asks for a reserve he/she means “I want to be very sure
that I will not exceed this reserve. I want my reserve to cover almost the
worst case.”
• However, that is not how it is interpreted by the employee.
– The worst case estimate was 51 min and required 70% reserve.
– The 99% confidence case averaged 45 min. and needed 50% reserve.
Both of these cases are significantly higher than the popular 30%.
20

21. Large Projects Reserve Comparison
100%
90%
The realistic amount of
Needed Reserve
80% budget reserve required
70% for 18 large projects
60% studied is 52%.
50%
40%
30%
20%
10%
Recent aerospace projects
10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Planned Reserve 21

22. Effect #5: Planning Fallacy
The planning fallacy, as defined by Daniel Kahneman and Amos
Tversk is a tendency to be overly optimistic in planning.
To asses the extent of optimism we asked:
1. In the best case scenario (if everything went as well as
possible), how many minutes would it take you to clean the
whole kitchen?
2. In the worst case scenario (if everything went as poorly as
possible), how many minutes it would take you to clean the
whole kitchen?
22

23. 5. Planning Fallacy Results
• The following results were obtained:
– 51 min worst case
– 45 min 99% confidence
– 30 min nominal
– 27 min best case
• These results show how skewed people are toward optimism. The
nominal estimate was 10% longer than the best case but 70%
shorter than the worst case.
• People are so optimistic that it was easy to anchor them down but
anchoring up failed.
23

24. Conclusions
• To improve the quality of cost estimates it is recommended to
diminish the effects of psychological impact on judgment:
 Train the managers not to anchor.
 Establish proper Estimation Language which makes the
questions compatible with common interpretation.
 Deep decompositions do not improve accuracy.
 Calculate the reserve based on risk.
 Account for optimism by including in the baseline
likely, historical and common risks.
24