1. Impact Evaluation, Replication
and Ethics
Richard Palmer-Jones
School of International Development
University of East Anglia
Presented at
Impact, Learning and Innovation: Towards a Research and Practice Agenda for
the Future
Institute of Development Studies, Brighton (UK), Convening Space
March 26-27, 2013

2. Modern impact evaluation
• Addresses the attribution issue by:
– Sophisticated econometric analyses of observational data
– Randomised control trials
– (and mixed methods, theory based, process tracing, and
evaluation, agent based modelling)
– phronesis
• Methods which assert their status as science by
– Mathematisation, quantification, neutrality and objectivity,
and so on
• But lack a crucial component of science
– Replication
• Repeatability, checking, internal, external, construct .. validity

3. Identification
• Problem of unobserved confounding variables
– Are benefits of microfinance due to loans from the MFI?
• Placement and selection biases
– More favourable areas
– People who are more likely to benefit
• Identification by:
– Randomisation
• Internal and external validity
– In the absence of randomisation, or with compromised
• multiple regression
• Natural or quasi experiments
– Single and double difference estimation
• Instrumental variables estimation
• Propensity Score Matching
• Regression discontinuity
• Panel data estimation

4. How robust are these methods?
• Methods require assumptions
– Many assumptions cannot be tested
• Randomisation
– Common threats to validity
• Imperfect selection of subjects
• Imperfect randomisation – subject agency
• Imperfect adherence to treatment
• Lack of blinding
– Hawthorn & John Henry effects
• Lack of external validity
• Econometric results
– Data mining, result polishing, researcher, sponsor, and
editor allegiance & reluctance, and HARKing

5. Replication in randomised studies
• Replication is the sine qua non of science
• Many RCTs do not yield the same results
– Many of the problems listed above
• Gave rise to systematic review and meta-analysis
– Systematic review seldom resolves issues for all concerned
even when large number of good quality trials
– Meta-analysis can make it appear that lots of weak results
combine to produce a convincing (statistically and
substantively meaningful one)
• But missing studies
– Publication bias – “Bad Pharma”
– Researcher and institutional allegiance and reluctance
» to publish results that are not from the right hymn sheet
• Register of all studies in advance ….

6. Replication in observational studies
• Pure replication
– Checking the (original) data and code produce the
reported results
• Data and coding errors
• Statistical replication
– Is the study robust to plausible changes in data
cleaning, variable construction, alternative equivalent
data
• Scientific replication
– Is the study robust to alternative accounts

7. Experiences from Replication World
…In practice ….
• Little replication in practice
– More replication than realised?
• Show me! [some key cases – but more generally?]
– Low incentives for replication
• Difficulties
– Difficult access to data & code
– Repeating the analysis is very taxing – detective work in the face of
incomplete documentation
– [Publication bias & file drawer problem]
– Deterrence
• Imputation of Adverse motives
– adversarial intentions
» Political
» Career advancement
» Lack of originality
• Belligerent refutation by original authors

8. Examples
• Feldstein / Leimer & Lesnoy
– Admit and contest with new results
• Levitt / Lott
– Law suite - dismissed
• Hoxby and Rothstein
– Beligerent contestation
• Accusation of political motives and misreporting
– Delayed publication (2004 –> 2007)
• Acemoglu et al. and Albouy
– Beligerent contestation
– Severely delayed publication (2006 -> 2012)

9. Experiences from Replication World
…in development practice ….
• Randomisation studies (briefly)
– Karlan and Zinman – randomly relax credit constraints
• Observational studies
– Boyce and Ravallion, 1991 -Declining real wages of agricultural
labourers in Bangladesh
– Basu, Narayan, and Ravallion, 2002 - Benefits to illiterates of
being proximate to female literates
– Pitt and Khandker, 1998 - The benefits of Microfinance
especially when loaned to women …
– Jensen and Oster, 2009 -The power of TV on the status of
women in India
– Banerjee and Iyer, 2005 - The lasting adverse effects of colonial
land revenue polices in India
– [Macro-economics] Aid and Growth
• Dollar & Burnside -> Mekasah and Tarp vs Doucouliagos & Paldam,
JDS, forthcoming, 2013
• Trade and aid -> . Perraton, 2011, Journal of Economic Methodology

10. Access to
microfinance
reduces credit
constraints
enables or
increases fixed
& working
capital, self-
employment
Increases
business profits
Increases
borrowing, or
reduces costs
of borrowing
Wage
employment,
production,
turnover, sales
Increases
income and, or
consumption education and or health
expenditure, child health and
nutritional status , subjective
well-being
Changes
expenditure
patterns
Women
empowerment
Business
losses
+
Failure to keep
up repayments
Borrowing from
other MFIs or
informal
sources
+
Reduces
income and, or
consumption
-
Women dis-
empowerment
-
inputs
effects
impacts
failures

11. 11
• Pitt & Khandker’s quasi-experimental design
Source: Armendáriz de Aghion and Morduch, 2005.
“Treatment” Village “Control” Village
Eligible non
participants
Not-eligible non
participants
Would be
eligible
Would not
be eligible
Eligible
participants
0.5
acrescultivablelandowned
0
0.5
Compare
eligible
participants
with eligible
controls in
treatment
villages
Compare
eligible
participants
with eligible
controls in
treatment and
control villages
non-eligible
participants
But 20% of
participants
have more
than 0.5
acres
Control
placement
bias with
village
fixed
effects

12. Results
• WES-LIML-FE (Roodman & Morduch, 2011)
– Modest impacts when borrowing by women
– Disappear when outliers removed
• Propensity Score Matching
– Chemin, 2008
• More modest effects, some negative
• Does not distinguish by gender
– (Duvendack and Palmer-Jones, 2012)
• Modest, zero and negative impacts when borrowing by
women
• Impacts highly vulnerable to “hidden bias”
– Hidden bias highly likely

13. Conclusions
• Leamer, 1983
– Let’s take the con out of econometrics
• Ioannidis, 2005
– Most published research findings are false
• Manski, 2011
– Use of econometric results in policy requires
“incredible belief”
• The devil is in the detail

14. Conclusions (cont)
• Power and interests speaks to truth?
• The development industry ….
– Status and power of applied econometrics
• Cognitive biases in (evaluation) research (belief)
– see patterns where there are none;
– see causal relations when there are none;
– overvalue confirmation;
– evaluate more favourably evidence that conforms with our prior beliefs
– seek out confirmation;
• Professional interests and the avoidance of cognitive
dissonance
– The disciplinary doxa
– States of denial (with apologies to Stanley Cohen)
– Economists’ ethics?

15. Country donors,
e.g. ODA, USAID
The ‘aid industry’ version 1
Multilaterals,
e.g. World bank, EU
Government
agencies
International
NGOs, e.g Oxfam
National NGOs
Local NGOs
Projects/activities
PEOPLE
(from Gardner and Lewis, 1996)

16. The development industry (version 2)
Multilateral aid agencies
UN …, WHO, FAO, …
Bretton Woods (IMF, WB)
Bilateral aid agencies
DFID, SIDA, NORAD,
USAID, JICA, CIDA, ..
National
governments
International NGOs
(INGOs)
CARE, OXFAM,
Christian Aid,
Save the Children
WWF,
Media -TV,
Newspapers,
Freelance
journalists
International
contractors,
consultants,
suppliers
Local
governments
National NGOs
managers
fieldworkers
Project &
programme
staff
Activists &
lobbyists
Development
academics
People (beneficiaries)
brokers, leaders, followers, patrons, clients
diverse, differentiated, gendered, included/excluded,
Politicians

17. Conclusions (cont)
• Ethical analysis and publication
– Respect
• the interests of research subjects and researchers
• interests of employers and funders
• Peer groups and professions
• Research as a social practise
– Publish negative or null results
– Enable replication