Presented by Colas Chervier (CIRAD) at "Workshop on impact evaluation methods and research collaboration kick-off", Samarinda, Indonesia, on 10 October 2022

1. Counterfactual impact
evaluation
Introduction
Colas Chervier
CIRAD
colas.chervier@cirad.fr
1

2. Content
• Session 1.
• What questions can these methods help address?
• What are the main features of these methods and why use them?
• Session 2.
• What is a counterfactual?
• What are the specific features of these methods?
• How to implement them in practice (steps)?
• Presentation of an example of an impact assessment study
2

3. The policy-relevant questions that these
methods can help address.
3

4. Examples of previous research work (1/2)
[Peru] In the 3 years between surveys, we observed a severe decline in forest revenue.
However, by using a BACI study design and matching, we show that this decrease was
not caused by the REDD+ interventions. Thus, REDD+ “did no harm” to local people, at
least in terms of forest revenues (Solis et al, 2021)
[Brazil] We find significant but small additional conservation effects from the
implementation of the PES program. Notwithstanding, treatment effects are relatively
larger in areas with higher deforestation pressure and higher potential agricultural
income (Cisneros et al, 2022)
[Brazil] We find that it is more effective to locate parks and payments away from each
other, rather than in the same location or near each other. (Robalino et al. 2015)
4

5. Examples of previous research work (2/2)
[Brazil] Results suggest that the accepted methodologies for quantifying carbon
credits overstate impacts on avoided deforestation and climate change
mitigation.(West et al. 2020)
[Indonesia] Contrary to the objective of the program, community titles aimed at
conservation did not decrease deforestation; if anything, they tended to increase forest
loss. In contrast, community titles in zones aimed at timber production decreased
deforestation, albeit from higher baseline forest loss rates. (Kraus et al. 2021)
[Nepal] Our results indicate that CFM has, on average, contributed to significant net
reductions in both poverty and deforestation across Nepal, and that CFM increases the
likelihood of win–win outcomes. We also find that the estimated reduced deforestation
impacts of community forests are lower where baseline poverty levels are high, and
greater where community forests are larger and have existed longer (Oldekop 2021).
5

6. Example of questions applied to community forests (1/2)
Directly related to the measure of impact…
• Do community forests have an additional effect on deforestation and
community incomes?
• Does CF effectiveness depends on the context (e.g. in a REDD+ area vs.
outside it), characteristics of the beneficiaries or the type of CF scheme?
• Are community forestry schemes more effective when combined with other
interventions?
• Is it possible to achieve win-win outcomes through CF?
• Etc.
6

7. Example of questions applied to community forests (2/2)
… and beyond
• Is CF worth replicating and if so, where?
• By putting the measured impact into perspective with the
costs incurred: is the CF model an efficient approach (for
example compared to other management methods)?
7

8. Definition and justification of
counterfactual impact evaluation
8

9. Overall definition
• Impact evaluation is a quantitative assessment of
the extent to which an intervention affects
outcomes
• The proper analysis of impact requires a
counterfactual, i.e. a control group representing
what those outcomes would have been in the
absence of the intervention
→ By comparing counterfactual and intervention
group, we can say how many hectares of forest have
been saved or by how many times the income of a
group of individuals has been increased by a program,
and if this result if significant
9

10. Differences with more widespread methods
• Monitoring tools
• Following a number of indicators over time in
program area
• E.g. LTKL monitoring tool →
• Different goal :does not aim to provide
information about if an intervention/policy
affects indicators measured
• Carbon sequestration scenarios
• Based on projections of historical trends in
target areas
• E.g. EK or any other forest carbon projects →
• Biased: changes cannot be attributed to
REDD+ interventions as lots of changes in the
context can influence deforestation
ERPD - FCPF
10

11. Biases associated with widespread methods
Before/after comparison
• Ex. carbon credits based on historical trends.
• Time-varying conditions that influence the target
outcome (e.g. change in the price of agricultural
products)
« Simple » with/without comparison
• Selection bias, there are initial differences
between control group and intervention (non-
random placement) that influence the results
Before/after PES
Control
Intervention (PES)
11

12. Introduction to quasi-experimental
methods and data needs
12

13. Definition of quasi-experimental methods
• Statistical methods that help identifies a control
group (e.g. households or forest plots that did not
receive the program) that represent a good
counterfactual
• Allows controlling for the main biases
• Applied in cases where the target area of the
intervention has already been selected (this
decision cannot be influenced)
• Specific data needs : baseline data (before the
program starts) & in "non-project" areas
13

14. Statistical methods to get rid of these biases
• Matching
• Selects two "similar" groups based on observable characteristics
(i.e. for which we have data)
• Double difference
• compares differences in outcomes over time between a
population participating in a program and one not participating.
• Often associated with matching
• Synthetic control method
• creates a control obtained as a weighted combination of
control units.
• Weights are assigned to control units so that their
combination is as close as possible to the treated unit's and
minimize differences in pre-treatment outcomes.
14

15. Conclusion day 1
• Provide answers to a wide range of questions relating to the
effectiveness of policies with a view to their
improvement/replication.
• Because these methods identify robust control groups, allow
to have a precise quantified measure free from bias.
• Specific data needs, especially pre-intervention data.
15