ISPC14

1. SIAC Phase 1Evaluation
**Some** **preliminary** findings and recommendations for discussion
ISPC, 15 Sept 2016
Julia Compton (independent), Tim Dalton (Kansas State University) and Sophie Zimm (IEA)

2. SIAC: Strengthening Impact Assessment in the CGIAR
• Initially a three year project (2013-16) extended to 2017
• Funders: BMGF, and Window 1 (mostly DFID top-up funding)
Two “top-level indicators of success”:
a) “An expansion of the available set of impact studies, providing useful
and credible information to guide future investments in the CGIAR.”
b) “CRPs and Centers of the CGIAR have institutionalized impact
assessment such that ex post impact assessment is regarded as an
essential part of prudent research management for accountability
purposes and as an input to ex ante strategic planning.”

3. SIAC has dramatically increased central resources for IA
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
2007 2008 2009 2010 2011 2012 2013 2014 2015 2016
AnnualSPIAbudget(US$million)
Year
SIAC W1
SIAC BMGF
DIIVA study
ISPC
SIAC Phase 1
Source: SPIA data, with thanks to Ira Vater

4. Context: CRP Phase 2 IA budgets
Source: CRP proposals, compiled by SPIA
0.5%
1.7%
1.5%
1.1%
2.3%
3.9%
3.0%
3.2%
3.1%
2.5%
4.4%
5.3%
0 1 2 3 4 5 6
CCAFS Climate Change
Livestock
Water Land Ecosystems
Legumes/Dryland
Wheat
Fish
Maize
FTA Forests Agroforestry
GRiSP Rice
SIAC
Roots Tubers Bananas
A4NH Nutrition Health
PIM Policies Inst Markets
Average annual budget (US$ M) for Impact Assessment
(labels = % of CRP total budget)

5. Four main SIAC activity areas
1 (Methods): Pilot and verify innovative methods for diffusion data
2 (Outcomes): Institutionalize the collection of diffusion data
3 (Impacts): Assess the full range of impacts from CGIAR research
4 (Building a community of practice): for epIA
within the CGIAR and between the CGIAR and the development
community more broadly

6. ISPC
SPIA
SIAC
Funded projects
• CGIAR centers
• External incl.
MSU)
1
Independent experts
• For external
reviews,
consultancies
2
IA focal points
(IAFPs)
• one per CGIAR
center
3
PSC
SPIA chair (chair)
SPIA Secretariat (1 staff)
BMGF, DFID
External expert (UC Davis)
CO and IEA member (observer)
CGIAR Fund
Council EIAC
FAO
BMGF
Administrative
reporting
Fund Use
agreement
Legend
intellectual exchange
Funding
Project and financial
reporting
W1 money
SIAC Organigram (2013- June 2016, before latest reforms)
ISPC Chair and
Exec DirectorCGIAR
CO

7. Introduction to this SIAC evaluation
• Financed by SIAC and managed for (independence) by IEA
Objectives:
• Accountability to SIAC funders (BMGF, DFID, System Council)
• Learning wider lessons for SPIA, and improving the design of SIAC Phase II
(SIAC Phase II proposal going to System Council in 2017)
Scope of evaluation: SIAC, not SPIA. IEA evaluation of ISPC/SPIA planned 2017
(However, overlap since SIAC covers whole SPIA budget and workplan.)

8. Methods
• Limited time/ timing (most data collection in August).
• Analysis of project and other documentation
• Semi-structured interviews /group discussions with >60 people
• SPIA, ISPC, Fund Office, System Office
• 11 CGIAR research leaders (CRP leaders/DGs/DDG-Rs)
• 21 CGIAR Impact Assessment Focal Points
• 5 donors - BMGF, DFID, USAID, ACIAR and Germany-GIZ
• External IA experts who know SPIA/SIAC
• CLEAR and 3iE

9. Questions addressed
(in the order of this presentation)
Effectiveness (1) Outputs: Has SIAC produced its planned outputs?
Quality of Science: Is SIAC promoting high quality work and cutting
edge methods?
Effectiveness (2) Outcomes: Is SIAC on track for planned outcomes?
Relevance : Does SIAC respond to need, demand and SPIA’s mandate?
SIAC management and governance: Does this promote efficiency,
accountability and transparency?

10. Effectiveness (1): SIAC Outputs
- A very ambitious work programme, with many subactivities
- Most outputs (under project control) achieved.
- Evidence of reflection, learning and adjustments.
KEY
Completed
On track
Some setbacks
Discontinued
New or modified
Too early to tell
1: Develop, pilot and verify innovative methods for diffusion data
1.1 Methods for crop varieties
1.2 Protocols for diffusion of NRM technologies
1.3 New institutional approaches for diffusion data
1.4 Disseminate best practices learned
2. Institutionalize the collection of the diffusion data needed for IA
2.1 Institutionalise collection of adoption data
2.2 Collect and validate NRM claims
Organise (and institutionalise) POR results
Institutionalise collection of adoption data

11. Outputs continued
KEY
Completed
On track
Some setbacks
Discontinued
New or modified
Too early to tell
3. Assess the full range of impacts from CGIAR research
3.0 IA on nutrition and health
3.1 Long term/large scale epIAs
3.2 Experimental/quasi exp studies
3.3. Under-evaluated areas
3.4 Meta analyses at system level
4. Support Community of Practice for epIA with CGIAR/partners
4.1 Small grants
4.2 Capdev of IA in the CGIAR
4.3 Biennial conference on epIA
4.4 Quality star rating for CGIAR IA studies
ex-4.5 Support RROs and NARs
4.5 SPIA website (but not yet one stop shop for CGIAR IA)
4.7 Support and capdev to Consortium
At least three studies by post docs

12. Some key outputs
Methods: DNA fingerprinting for varieties;
policy workshop with PIM; NRM pilots
Adoption data: central database of 12 crops
and 17 countries with 130 crop x sub-
region combinations
CGIAR policy outcome database
Capacity development: Center/Uni
partnerships, training, IAFP meetings
Evidence of CGIAR impact: Reviews of
water management, livestock. Many other
studies under way.
Survey enumerators
practice taking leaf
samples for genotyping
(Photo: SPIA)

13. Percentage of cost of SIAC studies by area of work
Varietal adoption
43%
Crop/Natural
Resource
Management
34%
Policy
6%
Extension/
Information
4%
Nutrition/health
9%
Livestock
4%
Source: evaluation team analysis of approved
research proposals

14. Quality of science
• SIAC process (for most): competitive calls, workshops, peer review of
proposals, approval by Project Steering Committee
• We analysed processes and (accepted) proposals
Findings:
• Most proposals are good-to-high quality
• Cutting edge methods in some areas (not always the objective)
• Some proposals do not clearly specify testable hypotheses
• One or two cases of poor research design: e.g. sample too small to test hypothesis
• Few spell out plans to assess distribution of adoption/benefits by gender or by social
groups (e.g. wealth/ethnic/caste)
• Few contain human subjects statements (ethics)
• Most calls now specify open data, but not all proposals are clear on plans
• Calls managed by different people – consistency in stated requirements is increasing

15. Effectiveness (2): SIAC Outcomes
Some progress, but could be stronger
• Methods have promise for broad uptake, eg DNA fingerprinting
• Slowly building a picture of CGIAR impacts, but could be more focused
• Institutionalisation is more challenging (e.g. of adoption studies, or of
links to prioritisation).
Two main areas for improvement identified:
a) Organisation of SIAC research in Phase 1
b) Theory of change of SIAC

16. Organisation of Phase 1 has favoured productivity and high
academic quality with limited resources
• Outsourcing management of key areas to trusted academics
• Competitive calls. Most calls broad in topic with proposals chosen for high
academic quality; opportunities to leverage other initiatives
- Experimental studies
- NRM impacts
- Nutritional impacts
- Testing large scale outcome claims by CGIAR
• Lack of systematic gap analysis and linkages to key indicators (IDOs)
Result: Many high-quality studies on the way, but also fragmentation and
potential loss of opportunities for institutionalisation and learning

17. Outputs after 5 years Outcomes Impacts
Systematic collection
of adoption data
institutionalized
Greater range of
types of research
and types of impact
covered by impact
assessments
Greater rigor and
scale of CGIAR
adoption estimates
and impact
assessments
Greater
transparency about
CGIAR impacts
Greater confidence
of donors in CGIAR
system
More evidence-
based decision-
making within the
CGIAR
CGIAR system
becomes more
effective at
reaching its
System-Level
Outcomes
Increased number of
impact assessments
of CGIAR researchSIAC LOGIC
MODEL

18. Outputs after 5 years Outcomes Impacts
Systematic collection
of adoption data
institutionalized
Greater range of
types of research
and types of impact
covered by impact
assessments
Greater rigor and
scale of CGIAR
adoption estimates
and impact
assessments
Greater
transparency about
CGIAR impacts
Greater confidence
of donors in CGIAR
system
More evidence-
based decision-
making within the
CGIAR
CGIAR system
becomes more
effective at
reaching its
System-Level
Outcomes
Increased number of
impact assessments
of CGIAR research
Selection,
design and
management
of SIAC
activities
Does not specify activities,
outputs, links, assumptions
and risks explicitly and in a
testable way

19. Relevance
SPIA’s mandate (reflects demand) :
To (1) provide CGIAR members with timely, objective and credible
information on the impacts at the system level...
To (2) provide support to and (3) complement the Centers in their ex
post impact assessment activities...
To (4) provide feedback to CGIAR priority setting and ... links to ex
ante assessment...
Source: SPIA website http://impact.cgiar.org/about 7/9/16

20. SPIA Mandate SIAC Relevance
1. Timely and credible
information on the
impacts at the system
level...
Broadly relevant, but could be better focused.
• More focus on SLOs/IDOs
• More gap analysis
• More consultation on priorities
2. Complement the Centers
in their epIA activities
SIAC has moved beyond epIA - potential overlap with Center research
– needs to be clearer on its role
• Increase consultation on priorities with Centers/CRPs
• IPSC/SPIA evaluation should address wider issue of who does what
3. Support to the Centers
in epIA
Broadly relevant, but could be better focused
- needs further analysis of comparative advantage
• Demand for cross-cutting methods (working with expert CRPs), collection
and centralisation of data, (guidance, quality assurance)
• Concerns about narrow SPIA disciplinary expertise
4. Feedback to CGIAR
priority setting and ... links
to ex ante assessment...
Potentially well placed in ISPC, but
• no structured feedback loops to ex-ante and priority setting (?)

21. Management – efficient, within constraints
• Understaffed for ambitious workplan - leading to outsourcing major
parts of the work, and employing consultants
• Efficient management, learning and improving
• Routing part of funding through FAO - a major efficiency constraint

22. Governance - reconsider for Phase 2
PSC composition: SPIA, core donors, IA expert, Consortium Office
(observer) and IEA (observer)
• Approves forward work plans and research proposals
• No adequate consultation mechanism for System Council and
Centers/CRPs
• Fund Council did not give clear guidance
Potential opportunities: a) annual meeting of CGIAR research leaders
b) new Committee in newly constituted System Council

23. Preliminary recommendations (partial)
1. Revisit the SIAC Theory of Change...
2. More systematic use of reviews and “evidence gap maps” (of key indicators e.g.
IDOs) to identify key IA topics and research questions.
3. Systematic consultation of CGIAR research leaders on IA needs, priorities and
responsibilities
4. Improve two-way feedback loops from IA to ex-ante modelling (would
especially welcome discussion and comments from the ISPC meeting on this
point)
5. Invest more strategically in supporting CGIAR to institutionalise and learn from
IA. Suggestions include:
• Regular reviews of IA in the CGIAR
• Revisit design of Community of Practice for IA
• Develop a (two-way) communications strategy
6. (for IEA) Upcoming ISPC / SPIA evaluation should analyse IA and related
activities across the CGIAR and consider roles and responsibilities.

24. Thank you