1. Evidence-Based Management
Helping Managers to Make Better Decisions
Denise M. Rousseau
H.J. Heinz II University Professor of Organizational Behavior
Carnegie Mellon University
denise@cmu.edu
What It Means to be an Evidence-Based Manager
2. EBMgt is the practice of making organizational
decisions based upon conscientious use of
1. Science-based principles & knowledge
2. Valid & relevant organizational and business facts
3. Critical thinking aided by decision supports
4. Ethical considerations (i.e., effects on stakeholders)
What is Evidence-Based Management?
3. Better Decisions by Using Practices that Work (and
avoiding those that don’t!)
Defensible Decisions that Stand Up to Scrutiny (using
best evidence and best process)
Developing Expertise throughout a Career
(experience can be a poor teacher--bad habits!)
20 years of valid experience is different than 1 year of experience
repeated 20 times!
Why Should We Care about EBMgt?
6. Chesley Sullenberger, USAIR pilot, has been a
visiting scholar at UC Berkeley’s Collaborative
for Catastrophic Risk Management since 2007
Does research on how to make decisions to
maintain safety despite technological
complexity and crisis conditions
1. Use of Scientific Findings
7. Has written and analyzed aviation accident
reports for over 20 years
2. Reliance on Reliable and Valid
Organizational Facts
8. Used Decision Aids to Support Good Decision: As Sully
considered what decision to make that day, he had his
copilot review and follow all checklists on board
relevant to crash landings
Formal Education to Prime His Skills: Sully is a graduate
of the U.S. Air Force Academy and holds masters
degrees from both Purdue University in Industrial
Psychology and the University of Northern Colorado in
Public Administration
3. Mindful Decision Making:
Becoming Decision Aware
9. The last person to leave the plane, Chesley
Sullenberger twice walked the plane’s aisle to
check all passengers were off
Sully’s last act onboard was to grab the
passenger list. Used on-shore to verify rescue
of all passengers and crew
4. Ethics and Responsibility to
Stakeholders
10. In Sullenberger’s Own Words…
“One way of looking at this might be that
for 42 years, I've been making small,
regular deposits in this bank of experience,
education and training.
And on January 15, the balance was sufficient
so that I could make a very large withdrawal.”
12. EBMgt is a means to
improve decision quality.
It’s a career, not a course.
13. Evidence-based practice movements abound in
medicine, education, and public policy
Management research from
psychology, engineering, operations research (ETC.)
yields 1000s of studies annually
Internet (scholar.google.com) gives ready access
Innovative companies now hiring “chief evidence
officers”
Public demands accountability (quality decisions that
are defensible)
The Zeitgeist
14. EBMgt Overcomes Limits of Unaided Decisions
Bounded Rationality
The Small Numbers
Problem of Individual
Experience
Prone to See Patterns
Even in Random Data
Critical Thinking
Decision Supports
Research
• Large Ns > individual
experience
• Controls reduce bias
The “Human” Problem Evidence-Based Practice
15. 1. Get Evidence into the Conversation
2. Use Relevant Scientific Evidence
3. Use Reliable and Valid Business Facts
4. Become “Decision Aware” and Use
Appropriate Processes
5. Reflect on Decision’s Ethical and Stakeholder
Implications
Five Good EBMgt Habits
16. #1 Get evidence into the conversation
Regularly ask “what’s the evidence…?”
Illustration- Discuss with your seatmates…
What’s a practice in your organization that you suspect might
not be NOT evidence-based?
Five Good EBMgt Habits
17. Evidence is not the same as „proof‟ or „hard facts‟
... can be
- so strong that no one doubts its correctness, or
- so weak that it is hardly convincing at all
What is evidence?
18. Evidence of effect (do!)
Evidence of no effect (don‟t!)
No evidence of effect (research!)
Don’t confuse
19. #2 Use Relevant Scientific Evidence
Focus on Action Principles Where Science is Clear
Rely on Science-based Sources
Example: Locke’s Handbook of Organizational Behavior
(access electronic copy in blackboard)
Peer-reviewed research, especially meta-analyses
Reduce dysfunctional variations in practice
Build effective routines, procedures, checklists
Five Good EBMgt Habits
20. Scientific Evidence
Best Scientific Evidence is
based on large N (sample size of
people/organizations)
well-controlled studies with comparison groups
&/or longitudinal data
peer-reviewed
Five Good EBMgt Habits
22. #3 Use Reliable and Valid Business Facts
Best Business Facts are
large numbers sampled relative to population
(not single or isolated cases, e.g. sales/# sales calls)
linked to context (season, location, #users, etc.)
provide key indicators for business decisions
Five Good EBMgt Habits
23. Illustration--Discuss with your seatmates…
What indicators does your organization most
commonly use to make important decisions?
Are these the “best business facts” you need to
make these decisions?
What indicators would be more useful, if you
could get them??
Five Good EBMgt Habits
24. # Medication errors in Unit 1 were 200% greater in 2011 than Unit 2’s. Is patient
safety worse in Unit 1? Depends on number of unsafe incidents divided by #
patients or # procedures—needs a control.
Mike has w/10 subordinates & 20% turnover while Kim has 55 employees & 10%
turnover. Is retention better in one? Hard to determine. Small N’s have greater
bias and are more variable.
McDonald’s stores average 300+% turnover/year. Does Mickey D. have a problem?
Depends on industry comparison and business strategy.
Company A managers focus decisions on monthly cost, downtime and revenues.
Company B managers focus on service quality, employee retention and
profitability by customer category. So what? B’s more diverse performance
criteria can promote attention to longer-term and growth-oriented outcomes.
A’s narrower economic focus can promote shorter-term thinking.
Help Learner How to Interpret Business Facts
25. #4 Become “Decision Aware”
Identify different kinds of decisions learners
face? What kinds of different approaches are
used to them? Why?
How can you determine whether you made a
“good decision” when you cannot know the
outcome? (The answer to this question is what is
known as “decision quality”)
Five Good EBMgt Habits
26. “Decision Awareness” Promotes Decision Quality
To manage decisions, know what decisions must be
made.
Map out decisions that affect key outcomes.
Who is responsible? (Are they prepared?)
What information is required? (Will it be available when
needed?)
Five Good EBMgt Habits
27. Five Good EBMgt Habits
Awareness Calls Attention to Decision Process.
Proper Processes Improve Decision Quality
What is the process for making the decision?
Different processes work better…
- for routine decisions (create validated checklists and action plans)
- for decisions with known unknowns (systematic sequence of
considerations)
- for decisions with unknown unknowns (pilot-tests and trial/
experiment)
Decisions have an “aftermath” and a “pre-math” that a
good manager actively manages. Is the decision well-
managed? Help make it so.
29. #5 Reflect on Decision’s Ethical
Implications
Who are stakeholders for this decision?
Possible effects?
How might the decision be altered to optimize positive
stakeholder effects and reduce negative?
Five Good EBMgt Habits
30. Scientific Principles for Effective Teaching
Set learning goals (2-5)
Pre-test: where does learner stand on learning goal before course
Build opportunities for practicing those learnings throughout
course (curriculum)
Post-test: Measure progress on each learning goal and provide
feedback
Feedback & Redesign: Use feedback to make course more
effective over time
Five Good EBMgt Habits
32. J. Ehrlinger, K. Johnson, M. Banner, D. Dunning, J. Kruger. (2008) Why the unskilled are unaware: Further
explorations of (absent) self-insight among the incompetent. Organizational Behavior and Human Decision
Processes, 105,(1) pg. 98
E.A. Locke (ed.), Handbook of Principles of Organizational Behavior, 2nd edition, 2009. Malden, MA: Blackwell.
D. M. Rousseau (2012) Oxford Handbook of Evidence-Based Management, New York.
D.M. Rousseau, D.M. & E. Barends (2011) Becoming an evidence-based manager. Human Resource
Management Journal, 21, 221-235.
D.M. Rousseau, J. Manning & D. Denyer (2008) Evidence in Management and Organizational Science:
Assembling the field’s full weight of scientific knowledge through reflective reviews. Annals of the
Academy of Management, 2, 475-515.
R.C. Schank, D. Llyras & E. Soloway (2010) The future of decision making. New York: Palmgrave Macmillan.
J.F. Yates. (2003). Decision management. San Francisco: Jossey-Bass.
J.F. Yates & M.D. Tschirhart (2006). Decision making expertise. In K. A. Ericsson, N. Charness, P. J. Feltovich, &
R. R. Hoffman. (Eds.). Cambridge handbook of expertise and expert performance (pp. 421-438). New York:
Cambridge University Press.
J.F. Yates, E.S. Veinott & A.L. Patalano (2003). Hard decisions, bad decisions: On decision quality and decision
aiding. In S. L. Schneider & J. C. Shanteau (Eds.), Emerging perspectives on judgment and decision research
(pp. 13-63). New York: Cambridge University Press.
Got Evidence? References