Challenges and Opportunities in Evaluating the Impact of Innovation Platforms

1. Challenges and Opportunities in
Evaluating the Impact of Innovation
Platforms
IDIAR Course 2016, Southern Sun Mayfair Hotel, Nairobi, Kenya
Karl Hughes, Head of Monitoring, Evaluation and Impact Assessment
World Agroforestry Centre (ICRAF)
IMPACT

2. IMPACT
• What do we mean by impact in
the IP context?
• Most simple definition: The
difference the IP(s) made—
whether expected of
unexpected, positive or
negative.

3. IMPACT(continued)
• If IP facilitation is the
intervention, then some (e.g.
economists) would say that
impact refers to the causal
effects of this intervention—
whether shorter, medium, or
longer term.

4. IMPACT(continued)
• Others distinguish
between outcomes
and impacts, e.g. IDRC
model
Sphere of
interest
Sphere of
influence
Sphere of
control
Outcomes = changes in
behavior/practice
Outputs
Impact = changes
in conditions, e.g.
health, poverty,
food security

5. In IP facilitation, what type of impacts do we care
most about?
• A stronger, more viable value
chain or better managed
watershed?
• Better food security and
incomes for producers and/or
other actors along the
agricultural value chain?

6. Approaches for evaluating IP IMPACT
1. Before after analysis
2. With and without (counter
factual) analysis
3. Theory-based (mechanism-
based) approaches
4. Towards an integrated approach
In science, causal inference is strongest with both
a rigorous estimation of the counterfactual and
evidence of the mechanism(s) responsible.

7. Time
Development
Outcome
(e.g., producer
income)
Pre-IP (T0) Post-IP (T1)
Change
(T1 – T0)
Before and after analysis
• Used alone not so
useful, particularly
in relation to
Sphere of Interest
level indicators

8. With and without (counterfactual) analysis
TimePre-IP (T0) Post-IP (T1)
Net
impact
Change with
intervention
Change that would
have happened
without
intervention
Development
Outcome
(e.g., producer
income)
• Use similar control or
comparison population
to estimate what would
have happened if there
were no intervention,
e.g. no IP

9. How using control/comparison group works to help
estimate average impact (for ‘large n’ interventions):
Variable (Baseline) Intervention Group Control/Comparison Group
Female Headed 10% 9%
HH Size 4.3 4.4
Adult with primary 67% 65%
Adult with secondary 14% 15%
Land Holding 0.89 hectare 0.91 hectare
Below Poverty Line 41% 39%
Asset Index 0.49 0.51
Cattle 3.2 3.5
Shoats 12.9 13.3
NGO exposure 85% 83%
Relevant unobservables, e.g. motivation Similar, esp. in RCTs Similar, esp. in RCTs

10. An ambitious example in the IP context
• Done in the context of FARA’s Sub-Saharan
Africa Challenge Programme (SSACP)
• Data from 24 wards evenly split across
Uganda, DRC & Rwanda (Lake Kivu Region)
• Random assignment at ward, rather than
village-level, due to potential of spill overs
• Sub-sample of ‘clean villages’ in IP wards
targeted, with control wards having both
‘clean’ and ‘unclean’ villages
• After two years, 17% average reduction in
poverty in IP villages, but how this
happened is unclear—IP villages seem more
likely to have customized innovations

11. Some key challenges with the
counterfactual approach in the IP context
1. Participation uncertainty
2. Fluidity of the IP intervention
3. Isolating the IP effect
4. External validity, i.e. will the result be
similar elsewhere?
5. Practical issues with controls/
comparison sites in the single IP context

12. 1. Participation uncertainty
• Value chain targeted & core IP
facilitators/leaders identified
• How sure can we be about who will
actually participate as the initiative
evolves?
• Initiative likely to evolve in unforeseen
ways, e.g. dropping unorganized
producers or moving into new
promising areas
• Unlikely to make commercial sense to
turn actors away

13. 2. Fluidity of the IP Intervention
• Interest in IPs in AR4D due to limitations of the “pipeline” model
Improve crop
variety or
practice
Extension or
seed system
Farmers
Adoption &
impact

14. Evaluation, Learning & Feedback Loops
• The translation of
research into impact
is, unfortunately,
rarely so simple and
linear…
• The purpose of IPs
is—almost by
definition—to
experiment (figure
out) how to overcome
common challenges
for the benefit of all
actors

15. • Developmental Evaluation becomes highly
relevant in the IP context
“Developmental evaluation is an
approach where evaluative thinking,
logic, and approaches – as well as
whatever data happen to be available –
are used for the purposes of continuously
developing a programme or specific
intervention.”
• So even if we happen to evidence overall
impact using the counterfactual approach,
uncertainty about actually led to it

16. 3. Isolating the IP effect
• Not only may it be difficult to determine cause
of observed impact—it may have nothing to do
with the nature of the IP concept
• e.g. an NGO could be involved that delivers
inputs to the participating producers
• e.g. the IP does something that any other
organization could have done
• Need to get at the mechanisms to really
understand the IP effect

17. 4. External validity (generalizability)
• We typically invest in impact
evaluations to generate learning
relevant for policy and/or practice
• However, IPs typically deal with such
unique and context specific issues, so
conclusions are likely not directly
transferable
• Would the cost and effort in rigorous
quantitative impact evaluation for
mainly accountability purposes be
worth it?

18. 5. Practical issues with control/comparison
sites in the single IP context
Option Issues
1. Within IP catchment area
(e.g. district) randomize or
purposively select areas
(e.g. villages) for IP to
target, while leaving others
as controls
• Likely create some commercial or programmatic inefficiencies, e.g.
transportation—need to bypass village right next to one you are
working with
• Likely to be significant non-compliance and spill-overs, e.g. producers
will got to intervention villages to sell their produce or buyers will
ignore
2. Use producers and/or
other value chain actors in
one or more other settings
(e.g. districts) for
comparison purposes
• Many of the practical issues associated with Option 1 may be
overcome
• Collecting baseline/endline data on both intervention & comparison
actors and comparing relative changes over time not a bad strategy
• But high chance that the groups will be subjected to different external
trends overtime, e.g. weather or NGO programmes, that affect
outcomes of interest

19. Approaches for evaluating IP IMPACT
1. Before after analysis (alone)
2. With and without (counter-
factual) analysis
3. Theory-based (mechanism-
based) approaches
4. Towards an integrated approach

20. Theory or mechanism based approaches
• Develop solid ToC, including
specifying desired behavior
key VC actors and other
stakeholders
• Great stuff—helps to support
adaptive mgt., etc. to better
facilitate the IP and/or deliver
better programme
• Combine with baseline and
endline snapshot of the value
chain (or other entity of
interest)

21. • For larger projects, consider
commissioning an evaluator with
expertise in contribution analysis
and/or process tracing, to drill down on
o the extent the project was responsible
for changes in the evaluation of the
agricultural value chain, etc.
• All the monitoring data collected will
prove very useful

22. But challenges again if focus is on specific actors in VC:
• Increase in VC commodity net
income not the same as overall
household income
• Even if we switch to looking at HH
income, other factors may have
been responsible for the changes
• Possible shifts in production/
income streams that may
exacerbate risk

23. Approaches for evaluating IP IMPACT
1. Before after analysis (alone)
2. With and without (counter
factual) analysis
3. Theory-based (mechanism-
based) approaches
4. Towards an integrated
approach
Almost
there

24. Towards an integrated approach
1. Baseline/Enline “snapshots” of status of VC and perhaps income
earned by actors for targeted VC products
2. Develop and continuously review and adapt ToC complemented
with Outcome Mapping and Developmental Evaluation
approaches, perhaps together with an external evaluation
3. Bring in researcher(s), if necessary, to support the testing of
innovations (Planned Comparisons) to overcome key
challenges—e.g. farmer field trials, efficacy studies, etc.
4. Focus efforts on assessing the extent to which these evidenced
innovations have been taken up and scaled, including numbers
reached

25. Group work
• Get into small groups
• Identify existing IP (or VC/NRM context in which an IP will be facilitated
• What key challenges is the existing or future IP working to overcome?
• Identify at least one challenge for which there is significant uncertainty on which
innovation would be most appropriate (cost-effective) to help overcome it
• Identify at least one innovative—in addition to the status quo—that appears
promising to address the challenge
• Use template provided to outline a planned comparison to test the effectiveness of
this innovation(s)
• If time, discuss how you would promote the scaling up and out of the innovation(s)
proven to be most effective and evidence the extent to which such scaling up and
out has taken place