Presentation given by Alessandra Garbero, Econometrician at IFAD, on the 12 February 2013.

1. Challenges to impact evaluation:
Solutions for IFAD
Alessandra Garbero, PhD
Econometrician
SSD, IFAD

2. IFAD Commitments
• Lift 80 m people out of poverty (2010-2015)
• Reach 90 m beneficiaries (direct/indirect)
• Challenge: how to measure quantitative impact?
• Solution: Impact evaluation
- Attribution 1. 30 rigorous evaluations
- Feedback 2. Retrospective evaluations using non-
experimental methods and IFAD/RIMS data
- Accountability
- Learning

3. IFAD’s projects portfolio: characteristics
• Relatively small projects
• Multiple interventions/components
• Eligibility criteria not systematically related to IFAD 9R
indicators
• Multiple treatments (same beneficiaries for different projects)
• Suboptimal baseline compliance
• RIMS policy does not foresee control group and panel
structure
• RIMS does not collect income/expenditure data
Non-experimental methods

4. Methodological challenges for impact
evaluation
1. Enhance internal validity: the lack of comparison group
prevents from causal inference between variables
- can be eliminated ex-ante or dealt within the analysis ex-post
2. Enhance external validity: extrapolate study’s results to
other project areas
- randomly selecting sites & within these sites randomly selecting
treatment and comparison groups
3. Purposive targeting of project beneficiaries: differences
between participating and non-participating households
(endogeneity)
4. No income/expenditure data: use appropriate techniques
with poverty proxies (small-area estimation methods or
alternative poverty proxy methods)

5. Two empirical applications from IFAD’s
projects
• Focus on changes in expenditure-based poverty status
• Can we measure these changes with the current data?
• Shall we think about using poverty prediction methods? (regression
models)
Vietnam (DPPR): Decentralised Nicaragua (Prodesec): Programa
Programme for Poverty de desarrollo económico de la
Reduction región seca de Nicaragua
Components: Capacity Building Components: Promote & Finance
for Decentralization process; Business and Employment; Rural
Production supports; Development
Financial services; Strengthen Rural
of rural villages’ small infrastructure
Development Policy
• Baseline 2006 • Baseline 2005
• Completion 2011 • Completion 2011
• LSMS 2002 – only available • LSMS 2005 – only available
dataset at the time of the analysis dataset at the time of the analysis

6. From assets to expenditure-based poverty
status
• Regression-based method (OLS): A prediction model that estimates expenditure
based on household characteristics (i.e. poverty explanatory factors i.e.
“predictors”) using the LSMS
• Poverty “predictors”:
- Vietnam: HH size; education of HH head; sex head; age head; assets (vehicle,
refrig., bike, moto radio, tv; toilet type and source of drinking water)
- Nicaragua: HH size; gender of HH; electricity; toilet type; source of drinking
water; farm HH; type of fuel; type of floor material.
• Model selection: conditional on sig., R squared (0.60 for Vietnam vs. 0.40 for
Nicaragua), presence of variables in both surveys
• Limitations:
- Vietnam: inferring poverty predictors based on 2002 LSMS relationships
between expenditure and key predictors
- Nicaragua: 2005 LSMS
• Definition of poverty line: set at the 30th percentile of rural households (as in
Minot 1998).

7. Results from poverty mapping: how
accurate is our model?
• Comparison between predicted and actual poverty
status using the OLS and the chosen poverty line
- Vietnam: the model identifies 72% of the observed
poor
- Nicaragua: the model identifies 55% of the observed
poor
• The model performs better for Vietnam

8. Impact evaluation
• IE  Before after: compare changes in impact indicator
before and after the project
- the counterfactual is represented by the same group
before they got the program
• What are the potential problems with this?
- Other factors contribute to change over time!
• Other secondary datasets (2 points in time) needed to
assess trends in the area (reconstruct
counterfactuals)

9. Poverty reduction? A Naïve comparison
Vietnam: RIMS Nicaragua RIMS
• Model results: Poverty declined from • Model results: Poverty declined from
35% to 9%  implies 25% of the 52% to 47%  implies 5% of the
sample lifted out of poverty sample lifted out of poverty
Problems
• Model based estimates of poverty based on limited available data
• Naïve comparison: no impact attribution, no control group, no panel
• Macroeconomic factors:
• Poverty declined from 45% to 18% in the whole Vietnamese province.
• Minor impact of project on relative wealth also based on assets
(Nicaragua). Selection bias possible or longer term impacts

10. Way forward
Rigorous ex-post evaluations need adequate secondary data
sources for both poverty prediction and matching exercises to
reconstruct the counterfactual
Theory-based impact evaluation designs need to be mainstreamed
within IFAD’s projects designs
Future data collection efforts: ensure adequate targeting
- Piggy back on national surveys (LSMS/NSOs) if overlap.
- Sample beneficiaries and non-beneficiaries within existing efforts.
Oversampling
If no national surveys underway
- Use an up to date sampling frame
- Randomize! Treatment and control (control group larger than
treatment)
- Core questionnaire with poverty predictors

11. Way forward
Rigorous impact evaluation require:
• Commitment
• Technical & analytical capacity
• Resources
Worth it?
• Strategic relevance
- Contribute to global understanding of agricultural pathways out of
poverty
• Increase evidence of well-functioning programs
• Interventions that work for scaling up