This presentation, on cost effectiveness and cost benefit analysis for impact evaluations, was delivered at the World Bank DIME Field Coordinator workshop on June 8, 2016. A range of resources for carrying out cost analysis are included in the final slides.

1. $
Manage Successful
Impact Evaluations
FIELD COORDINATOR
WORKSHOP
6-10 JUNE 2016
WASHINGTON, DC

2. How Much Does It Cost To Get That Impact?
Measuring Cost Effectiveness
8 June 2016
David K. Evans, Senior Economist, Office of the Chief
Economist for Africa

3. The Challenge
Increasing Evidence of Impact
Legovinietal.2012

4. We have all this amazing
evidence of interventions that
will work in your country!
Yes, but how much is this going
to cost me?
Not just you, man! How much is
it going to cost me?
Government Minister
Entrepreneur
IE Expert #1

5. 1. Summarize a complex
program in terms of a
return on investment
2. Permit comparison
across contexts &
times
Combining cost & impact data has two goals
Dhaliwaletal.2011

6. Intuitive appeal of cost-
effectiveness estimates
JPAL2014

7. Cost Effectiveness Can Reverse
Ranking of Effective Interventions
What are the most
effective education
programs?
Source: J-PAL, “Increasing test score performance”

8. Cost Effectiveness Can Reverse
Ranking of Effective Interventions
What are the most
cost-effective
education programs?
No overlap between
those two groups.
Source: J-PAL, “Increasing test score performance”

9. Cost Effectiveness Can Reverse
Ranking of Effective Interventions
If we want to responsibly use impact evaluation to inform policy,
cost analysis is crucial.
Source: J-PAL, “Increasing test score performance”

10. Definitions and Debates
Cost benefit
analysis
Cost
effectiveness
analysis

11. Cost Benefit Analysis
𝑁𝑒𝑡 𝑝𝑟𝑒𝑠𝑒𝑛𝑡 𝑣𝑎𝑙𝑢𝑒 =
𝑡=0
𝑛
𝐵𝑡
(1 + 𝑟) 𝑡
−
𝑡=0
𝑛
𝐶𝑡
(1 + 𝑟) 𝑡
1. Monetize the
benefits of the
program
2. Sum those up
over time (for n
years) 3. Discount
future gains
4. Do the same
thing for costs
6. Voilà! A summary
measure of the lifetime
net value of the
intervention!
5. Subtract costs
from benefits
McEwan2012

12. Cost Effectiveness Analysis
McEwan2012
1. Calculate the
cost
2. Divide it by
incremental
effect
𝐶
𝐸
= 𝑐𝑜𝑠𝑡 𝑝𝑒𝑟 𝑎𝑑𝑑𝑖𝑡𝑖𝑜𝑛𝑎𝑙 𝑢𝑛𝑖𝑡 𝑜𝑓 𝑏𝑒𝑛𝑒𝑓𝑖𝑡
What the h#@! is an additional unit of benefit?
10% increase in business profits 1 standard deviation increase in crop yield
10% increase in farmer knowledge Additional woman receiving prenatal care
Additional student in school 1 s.d. increase in student learning

13. CBA
 Absolute desirability of
intervention
 Allows comparison across
totally different
interventions (health
versus agriculture)
 Requires very strong
assumptions (lifetime
benefits of intervention…)
Pros and Cons
CEA
 Much more transparent:
No need to monetize range
of benefits
 Simpler to explain
 Useful for comparing
interventions with
common outcome
(agricultural extension
versus targeted savings)
 Implicit assumption:
Common post-intervention
trajectory
McEwan2012

14. Question: The Minister of Finance is deciding whether to scale
up an Agricultural Extension program or a Business Training
program. What kind of analysis will be most useful?
1. Cost-Benefit
Analysis
2. Cost-Effectiveness
Analysis
Answer: (1)

15. Question: The Minister of Education wants to increase student
test scores. What kind of analysis will be most useful?
1. Cost-Benefit
Analysis
2. Cost-Effectiveness
Analysis
Answer: (2)

16. Whose costs?
Take a social perspective
Seed distribution program
Ministry of Finance
DeliveryCommunity leadersHouseholds
Ministry of Agriculture NGO Partner
McEwan2012

17. Question: The government is giving grants to entrepreneurs.
Where do these enter the costing equation?
1. As cost to the
government.
2. As benefit to the
entrepreneur.
3. Both.
Answer: It depends. In CBA, the answer is
(3), so it cancels out.
In CEA, we don’t include other benefits
(e.g., increased revenues, better health),
so why this? So (1).
McEwan 2012 and Dhaliwal et al 2011

18. Ingredients method
• Personnel
• Facilities
• Equipment
• Materials
• Other program inputs
• Client inputs
Recipe for Cost Analysis
Metaphor for
successful cost
analysis
Looks delicious!
McEwan2012

19. Where do we get the info?
Budget
• What it was supposed to cost
• Not actual cost
• Often misses client costs
Interviews & direct observations
• Did it go how it was supposed to go? (No)
• How many seeds were actually distributed? How much time did it
actually require?
Data collected in impact evaluation
• Remuneration
• Time use
McEwan2012
Need early: Guides
rest of cost analysis
Very difficult to
gather ex post
Must be
designed ex
ante

20. Where do we get the info?
Budget
• What it was supposed to cost
• Not actual cost
• Often misses client costs
Interviews & direct observations
• Did it go how it was supposed to go? (No)
• How many seeds were actually distributed? How much time did it
actually require?
Data collected in impact evaluation
• Remuneration
• Time use
McEwan2012
Need early: Guides
rest of cost analysis
Very difficult to
gather ex post
Must be
designed ex
ante

21. Value each ingredient
Adjust for *
– Inflation ---
– Time-value *
– Currency ---
+ Add it all together
Cost of the program
What do we do with the info?
McEwan2012

22. Question: My agricultural extension program has
community volunteers who spend 10 hours a week and
receive a bag of rice. How do I cost that?
1. Market value of a bag of
rice
2. Market value of 10 hours
of time for people with
similar background
3. Both
4. I don’t know; I was
checking my email.
Answer: (3). Hopefully not (4). Come
on team, heads in the game. Really, it
depends.
If “government only” perspective,
could leave out, but volunteers in
pilot may not be available at scale.
McEwan 2012 and Dhaliwal et al 2011

23. JPAL2014
What Cost Effectiveness
Looks Like in Practice

24. • Cost analysis is extra work
• Most studies don’t do it
– Of 77 RCTs, 56% reported
zero data on incremental
costs (McEwan 2014)
• Not doing it significantly
limits policy relevance
Well, that was easy. Not!
That was exhausting!

25. Now that you know how to do it
A few caveats to keep in mind
Caveats

26. Cost-Effectiveness Sensitivity
to Errors in Estimates
Additional s.d. of student learning per $100
Evans&Popova2014
Gray: Measured
cost
effectiveness
Black: 90%
confidence
interval on
impact estimate
Lesson: Be humble about cross program ordering.

27. Cost-Effectiveness Sensitivity to Place
Evans&Popova2014
Actual
effectiveness-
cost ratio for
India program
Gray: Other
programs
Additional s.d. of student learning per $100
Red: Use other
country costs of
community
teachers
Lesson: Adapting cost-effectiveness across countries takes work.

28. Pilot vs Scaled up
Evans&Popova2014

29. Cost Sensitivity to Exchange Rates
Dhaliwaletal.2011
Standard exchange rate Purchasing power parity
Assumes relative prices in country Adjusts for price levels across countries but
messes up relative prices

30. Ignoring:
1. Categories of ingredients: facilities, equipment, client
inputs
2. Sensitivity (to place, to scale, etc.)
Common Pitfalls
AdaptedfromDeLaBriere
3. Exchange rates
4. Reality (costing ideal vs.
actual program).
Not providing enough
details to verify 1 to 4

31. 1. No policymaker adopts a program without
knowing the cost (we hope!)
2. Costing and cost effectiveness take us from
scholarship to practical policy implementation
3. It is just as important to be prospective with costs
as with impact evaluation
4. Field coordinators are particularly well placed to
gather cost data
Remember!

32. • Papers!
– Cost-Effectiveness Analysis of Education & Health Interventions in
Developing Countries, by McEwan (2012)
– Comparative Cost-Effectiveness Analysis to Inform Policy in
Developing Countries, by Dhaliwal et al (2011)
– Cost-Effectiveness Measurement in Development: Accounting for
Local Costs & Noisy Impacts, by Evans & Popova (2014)
• A book!
– Cost-Effectiveness Analysis: Methods & Applications, by McEwan &
Levin (2000)
Additional Resources - 1

33. • IDB Evaluation Hub: Cost Benefit and Cost-
Effectiveness
– Background materials
– Terms of reference
– Examples
– Templates
• Poverty Action Lab
– Examples for student participation and student
learning
– What does this look like in Excel?
Additional Resources - 2

34. Credits
• All clip art from openclipart.org
• “Just Do It” from Adweek.com
• NGO partners from OneAcreFund.org
• Some other photos from World Bank Flickr
stream
Thank you!