How do impact evaluations improve operations and policy? 9 ways and 15 examples

How do impact evaluations improve
development operations and policy?
David K. Evans, World Bank
Africa Chief Economist’s Office
June 1, 2016 | 3ie/IFAD
devans2@worldbank.org | @tukopamoja

How do impact evaluations affect operations and policy?
Own project
• Clarify thinking about the theory of
change
• Improve quality of data systems
• Improve efficiency of projects
• Shifts in project design
• Influence scale-up of projects
Other projects
• Design of new projects
• Choosing between designs
• Culture of evidence
• Contribute to a body of knowledge
that changes how we do business
1
Evans|WorldBank|impactevaluations,operations&policy

I’m going to tell you some
stories (and show some
analysis).
But how do we measure impact?
2
Isn’t impact evaluation about careful
causal inference?
Where’s the counterfactual?
Image source: http://imperialcentrepaintingstudio.blogspot.com/2012/02/reading-comprehension-teach-your-child.html

Measuring the impact of impact evaluation
Second cousin to the event study
• Before-after is a poor evaluation
method
– Because usually many other things
are changing
• But useful when no other credible
cause
Embrace the ambiguity
• The best impacts from impact
evaluation can come when the
resulting decisions are completely
internalized by government (Goldstein
2016)
• Short of an RCT of RCTs, embrace
the ambiguity.
3
Jensen (2007)

Indicators for IE impact
3ie
• Closing knowledge or evidence
gaps?
• Changes in policy or programs?
• Key stakeholders and key
influencers?
• Plan to engage with them at each
stage of the IE?
• Cleary theory of change for policy
influence?
DIME (World Bank)
• Capacity building
– IE results discussed with client
– Training provided for data analysis
• Quality of data
– Availability of data
– Improved administrative data
• Quality of policy decisions
– Used to motivate scale-up/scale-down of
policy
– Adopted delivery mechanism based on IE
results
– Baseline informed design or implementation
4
Source: Legovini 2014Source: 3ie 2012

• Impact evaluation helps clarify thinking on exactly
how one gets from inputs to impacts
• By focusing on questions of
– Causality
– Measurement
• Impact evaluation improves theories of change
Own project
Clarify thinking about the theory of change
5
Figure sources: Imas & Rist 2009

Own project
Improve quality of data systems
• Kenya Judicial Performance IE
– Impact of incentives for greater
efficiency
– Co-developed tool for measuring
efficiency of courts: How quickly do
they get their cases out
– Tool was adopted for all courts as
official monitoring tool
• Kenya Patient Safety IE
– Impact of inspections of health
facilities
– Reform to regulatory framework of
inspection system for minimum patient
safety.
– Design of quality of care measures
6
Sources: World Bank 2013; i2i n.d. (1); i2i n.d. (2)

• Sample of 100 impact
evaluations, 1000+ projects
between 2005 and 2011
• Include time, sector, country,
and project leader effects
• Significant increase in projects
carrying out their activities
Own project
Improve efficiency of projects
7
Source: Legovini et al. 2015

• Notably, this increase in
disbursements happens after
impact evaluation begins
• Can’t completely control for
selection into project
• BUT suggestive that IEs help
projects roll out and that they
don’t slow projects down (as
commonly claimed)
Own project
Improve efficiency of projects
8
Source: Legovini et al. 2015

Own project
Shifts in project design
Ethiopian Nile Irrigation and Drainage Project
• Baseline survey in Ethiopia
• Women have
– Lower access to extension
– Greater time poverty
– Much lower agricultural value-added
for women than men
• Result: Pilot strategies in Farmer’s
Initiative Fund, designed for female
farmers
9
Source: World Bank 2014

Own project
Influence scale-up of projects
Financial Literacy in Brazil
• Pilot in high schools
• Modest but positive, significant
impacts
• Scale-up
– High schools around the country
– Elementary schools
Other example: Land titling in Rwanda
Conditional Cash Transfers
• Mexico
– Initial program showed positive results
– Expansion to urban
– Survival across administrations
• PROGRESA  Oportunidades 
Prospera
• Tanzania
– Pilot showed positive results
– Key evidence informing scale-up
10Source: Dhaliwal & Tulloch 2012; Mkelemi & Evans 2016Source: Bruhn et al. 2013; Ali et al. 2011

Other projects
Design of new projects
Projects reports that IEs influence design
• Education Modernization Project in
Kazakhstan
– Influenced by synthesis of education
impact evaluations
• Adolescent Girls Empowerment
Project in India
– Influenced by IEs of programs to
empower adolescent girls in Liberia
and Uganda
Citations of IEs in World Bank documents ↑ 6x
11
3,691
21,271
0
5,000
10,000
15,000
20,000
25,000
05 06 07 08 09 10 11 12
TotalIECitations
Fiscal Year
Source: Legovini 2014Source: Gender Innovation Lab 2015

• Zambia Essential Drugs Public Pilot Program
• Consistent problem with stockouts of essential malaria drugs
• Randomized impact evaluation to test two different models
• A: District planner
• B: Direct orders from clinic to
central medical stores
Other example:
Early child development in
the Gambia
Other projects
Choosing between designs
12
Source for Zambia: Vledder et al. 2015; World Bank 2010
Source for the Gambia: Blimpo & Carneiro 2014

Other projects
Culture of evidence
Mexico
• Progresa/Oportunidades 1997-2000
• 2000 Congress’ Degree: Annual
external evaluations for projects
• 2004 Social Development Law:
National Council for Evaluation of
Social Policy (CONEVAL)
Tanzania
• Two donor-initiated impact evaluations
under TASAF-2
– Community-based conditional cash transfers
– Vulnerable groups
• Three client-initiated impact evaluations
under TASAF-3
– Scaled-up cash transfers
– Cash transfers on women’s empowerment
– Cash transfers on transition to adulthood
• False dichotomy: Donor-initiated vs
client-initiated. But a shift along the
spectrum.
13Source: Hernández Licona 2008

Other projects
Contribute to a body of knowledge that changes how we do
development
Small questions or big questions?
• Should we give away insecticide
treated nets or charge for them?
• Higher prices
– lower use
– don’t improve targeting
– don‘t increase use
• Between 2000 and 2015: “450
million cases of malaria and 4
million deaths averted from ITN
distribution”
Many small studies add up to big answers
Pricing experiments around the world
14Source: Dupas & Miguel 2016Source: Glennerster 2016

3ie. “Policy Influence Plan.” 2012. http://evaluationvigie.fr/wp-content/uploads/2014/05/2012_PIP_template.docx
Ali, Daniel, Klaus Deininger, and Markus Goldstein. “Environmental and gender impacts of land tenure regularization in Africa:
pilot evidence from Rwanda.” World Bank Policy Research Working Paper 5765. 2011.
http://documents.worldbank.org/curated/en/2011/08/14833489/environmental-gender-impacts-land-tenure-regularization-africa-
pilot-evidence-rwanda
Blimpo, Moussa, and Pedro Carneiro. “Early childhood development: Some preliminary findings to guide future actions.”
PowerPoint Presentation. 2014.
Bruhn, Miriam, Luciana de Souza Leao, Arianna Legovini, Rogelio Marchetti, and Bilal Zia. “The impact of high school financial
education: experimental evidence from Brazil.” World Bank Policy Research Working Paper 6723. 2013.
http://documents.worldbank.org/curated/en/2013/12/18640673/impact-high-school-financial-education-experimental-evidence-
brazil
Dhaliwal, Iqbal, and Caitlin Tulloch. “From research to policy: using evidence from impact evaluations to inform development
policy.” 2012.
https://www.povertyactionlab.org/sites/default/files/publications/With%20JDE%20Revisions%20and%20Footnotes%202012.09.
03.pdf
Dupas & Miguel. “Impacts and Determinants of Health Levels in Low-Income Countries.“ Prepared for the Handbook of Field
Experiments. 2016. http://emiguel.econ.berkeley.edu/assets/miguel_research/73/Health_DupasMiguel_Handbook.pdf
Gender Innovation Lab. Africa Gender Innovation Lab (GIL) Influence. November 2015.
Glennerster, Rachel. “Not so small.” Running Randomized Evaluations blog. May 27, 2016.
http://runningres.com/blog/2016/5/27/not-so-small
Goldstein, Markus. Conversation with Dave Evans on a bus. Rome, Italy. May 31, 2016.
Hernandez Licona, Gonzalo. “Impact evaluation in Mexico.” January 2008.
http://siteresources.worldbank.org/INTISPMA/Resources/Training-Events-and-Materials/449365-
1199828589096/GonzaloHernandezLicona.pdf
References - 1
15

i2i. Changing Policy through Research: DIME IE Portfolio in Kenya. n.d.
i2i. Selected Examples of DFID’s Contribution to Evidence-Based Policy through i2i. n.d.
Imas, Lnda G. Morris, and Ray C. Rist. The Road to Results: Designing and Conducting Effective Development Evaluations. World
Bank. 2009. https://openknowledge.worldbank.org/bitstream/handle/10986/2699/52678.pdf
Jensen, Robert. “The Digital Provide: Information (Technology), Market Performance, and Welfare in the South Indian Fisheries Sector.”
Quarterly Journal of Economics 122(3), August 2007. http://www.jstor.org/stable/pdf/25098864.pdf
Legovini, Arianna. “The development impact of impact evaluation.” World Bank. 2014.
http://www.worldbank.org/content/dam/Worldbank/Feature%20Story/japan/pdf/event/2014/120514_Arianna_Legovini.pdf
Legovini, Arianna, Vincenzo di Maro, Caio Piza. “Impact evaluation helps deliver development projects.” World Bank Policy Res earch
Working Paper 7157. 2015. http://documents.worldbank.org/curated/en/2015/01/23173058/impact-evaluation-helps-deliver-
development-projects
Mkelemi, Godfrey, and David K. Evans. “The Challenge and Success of Impact Evaluation for Tanzania Community-Based Conditional
Cash Transfer Pilot.” Conference presentation, IFAD/3ie. May 31, 2016. http://www.slideshare.net/DavidEvans198/the-challenge-and-
success-of-impact-evaluation-for-tanzania-communitybased-conditional-cash-transfer-pilot
Prottas, Chris (@ChrisProttas). “I understand why, but real mental disconnect w anecdotes used to justify impact of impact evals.” May
27, 2016, 3:10am. Tweet. https://twitter.com/ChrisProttas/status/736137568954294272?lang=en
Vledder, Monique, Jed Friedman, Mirja Sjoblom, Thomas Brown, and Prashant Yadiv. “Optimal Supply Chain Structure for Distributing
Essential Drugs in Low Income Countries: Results from a Randomized Experiment.” Ross School of Business working paper 1269.
March 2015. https://deepblue.lib.umich.edu/bitstream/handle/2027.42/110800/1269_Yadav.pdf
World Bank. From Evidence to Policy: The Challenge of Ensuring Adequate Stocks of Essential Drugs in Rural Health Clinics.
November 2010.
World Bank. “Improving patient safety in Kenya.” November 6, 2013. PowerPoint Presentation.
https://www.wbginvestmentclimate.org/results/upload/Kenya-PS-BBL_November-6-2013.pdf
World Bank. Africa Impact Evaluation Update: Agriculture and Land. March 2014.
References - 2
16

How do impact evaluations improve operations and policy? 9 ways and 15 examples

Recommandé

Recommandé

Contenu connexe

Plus de David Evans

Plus de David Evans (12)

Dernier

Dernier (20)

How do impact evaluations improve operations and policy? 9 ways and 15 examples

Notes de l'éditeur