David Fleming held a seminar on monitoring and evaluation in conflict-affected environments at the Post-war Reconstruction and Development Unit (PRDU), University of York.

1. Monitoring and Evaluation in
Fragile and Conflict-Affected Contexts:
The challenges of measurement
David Fleming, Senior Consultant, Itad
Date: 28th January 2015

2. Seminar Outline
1. Introducing Itad: Life as an M&E consultant
2. Introducing/recapping M&E: Why monitor
and evaluate and why important in FCAS?
3. Theories of change: what they are, why they
are useful and challenges in FCAS
4. M&E approaches and methods: how to
monitor and evaluate in FCAS; examples from
peacebuilding and humanitarian work

3. Learning objectives
1. Come away with a better understanding of why we
do M&E and why it’s particularly important in FCAS
2. Learn about and put into practice some of the most
important M&E methods and tools for FCAS
3. Be able to better identify the challenges of doing
M&E in FCAS and how to overcome these
4. Everyone to leave the room with a burning desire to
get involved in M&E at some point in the future!

5. Life as an M&E consultant…

6. 2. Introducing M&E:
Why Monitor and Evaluate?
“After decades in which development agencies have
disbursed billions of dollars for social programs, and
developing country governments and nongovernmental
organizations (NGOs) have spent hundreds of billions
more, it is deeply disappointing to recognize that we
know relatively little about the net impact of most of
these social programs”
‘When will we ever learn?’ Evaluation Gap Working
Group, Center for Global Development 2006

7. • Monitoring: “Collection of data with which managers can
assess extent to which objectives are being achieved” (World
Bank)
– Purpose: Collect information on programme outputs and
outcomes to track and improve performance and results
• Evaluation: “Determination of the value of a project,
programme or policy” (World Bank)
– Purpose: evidence-based decisions, accountability,
transparency, lesson learning
– Types: project, programme, policy, organisation, sector, theme,
formative, summative, impact…

8. Why is M&E important in FCAS?
1. Development trends in FCAS
• By 2015, 50% of world’s poor will live in fragile states
(OECD); by 2030 it might be two thirds (Brookings)
• Support to conflict, violence and fragility becoming a key
priority for most major donors
• ODA to fragile states is falling in quantity but number of
actors multiplying (OECD)
• DFID has been scaling up support to FCAS (commitment
to increase to 30% of ODA by 2015)
• DFID strategies include BSOS, cross-Whitehall CSSF, and
the ‘Beyond Aid’ agenda

9. Why is M&E important in FCAS?
2. Increasing emphasis on transparency,
accountability and fiduciary risk
• Higher risk to investments in terms of results,
security and fiduciary risk
3. More limited evidence base – need for lesson
learning and evidence of what works
• Support evidence-informed decisions and better
programming by knowing what works and
doesn’t and why and in which contexts

10. What are the biggest challenges?
Risk of
exacerbating
conflict
Hawthorne
effect
Insecurity
Political
objectives
Longer-term
nature of results
Measurement
challenges
Vulnerability to
biases
Lack of existing
data
Poor data
reliability
Poor data
accessibility
Unpredictable
chains of
causation
Complex and
dynamic
contexts

11. M&E within the programme cycle
Identification
Problem analysis
Appraisal
Evidence of what
works
Design
Most cost-
effective
intervention/s
Implementation
With M&E built in
from outset
Completion
Measure results
– did it work?
Post
Completion
Feed lessons
into future
decisions
Lesson Learning
and
Feedback

12. Challenges of programming in FCAS
Identification
Problem analysis
contested
Appraisal
Little robust
data and
research. No
time
Design
Little evidence
to assess cost
effectiveness.
Political
imperatives
Implementation
Great hurry. M&E lags
behind. No baselines/
measurement
strategies
Completion
Not enough data
to say. No
inclination to
admit failure
Post
Completion
Not enough
results
published/
stored/
synthesisd.
Disagreement No knowledge
management/sharing and
lots of uncoordinated
actors

13. 3. Theories of Change (ToC)

14. Why are ToCs useful for M&E?
A ToC is an iterative and collaborative process for thinking through how a
programme is expect to work within the context of the broader system. It
should create the space for critical reflection and learning and be adjusted
and iterated over time.
• Links to assumptions box in LF, but goes beyond this in focusing on iterating
through learning shared mental models of how change happens
• Important for developing M&E strategy – test key links and assumptions
(intellectual leaps) in the causal chain over the life of the programme
• Important for evaluability – provides foundation for a theory-based evaluation
• Important to talk of ‘theories’ not ‘theory’ – i.e. to recognise and manage a
range of theories and multiple drivers of change
• Not a tick-box exercise or management tool like the LF but a way of working
and thinking – it’s primarily a process rather than a product

15. What are the pitfalls in FCAS?
• Time and resource-consuming – so they can often be poorly conceived/
too vague
• Poorly understood/used – as linear tick-box exercise rather than iterative
approach
• Oversimplification of complex contextual (e.g. conflict) factors – reflexivity
and feedback loops in complex conflict systems – black swan idea
• Absence of/poor conflict analysis – must underpin project design
• Difficulties in evidence gathering/data collection – conflict environments
are often data rich but information poor – insecurity, staff turnover
• Difficulties of working with and aiming to influence a range of actors
• Unpacking chains of cause and effect in FCAS can be very difficult
• Death by diagram
• Funnel of attrition

16. The funnel of attritionOnly these people
may experience
improved outcomes

17. 4. M&E approaches and methods
Recent explosion of new and innovative
approaches to monitoring and evaluation:
1. Use of mobile technology and ICTs for data
collection and analysis – e.g. Ushahidi
2. Influence of complexity science – PDIA, DDD –
enabling environment for experimentation
3. Remote monitoring and verification
4. Rigorous evaluation/impact evaluation designs

19. Why evaluate?
• White and Waddington (2012):
‘The use of the systematic reviews methodology is
comparatively new among social scientists in the
international development field, but has grown
rapidly in the last 3 years...To date, there has not
been a strong tradition of using rigorous evidence in
international development. The evidence bar has
been rather low, with many policies based on
anecdote and ‘cherry picking’ of favourable cases’.

20. Why evaluate?
• Accountability and lesson-learning
– Accountability to taxpayers and beneficiaries
– Understanding what works, why, where and for whom
to underpin evidence-based programming
– Priority to evaluate interventions with a weak
evidence base
• Inform scale up of an intervention or transfer to
another context
• Make mid-course corrections
• To support spending decisions

21. What is impact evaluation?
“Impact evaluation is a with versus without analysis: what happened with the
programme (a factual record) compared to what would have happened in the
absence of the programme (which requires a counterfactual)” (White, 2013)
“Impact evaluation aims to demonstrate that development programmes lead
to development results, that the intervention has a cause and effect” (Stern
et al. 2012)
• Attribution analysis to understand what difference a programme made
• Counterfactual construction through experimental/quasi-experimental
methods for large n (comparison groups); causal chain analysis for small n
• Theory-based impact evaluation – in ideal world, an RCT should be
embedded in a broader theory-based design that addresses questions
across the causal chain (White, 2013)
• Causal chain analysis – rigorous empirical assessment of causal
mechanisms and the assumptions that underlie the causal chain

22. How do we estimate impact?

23. How do we estimate impact?

24. Pros and cons of RCTs
• Pros: RCTs are the “gold standard” for addressing
attribution when an ex ante design is possible with a
large number of units of assignment
• BUT MAJOR DRAWBACKS, ESPECIALLY IN FCAS
– Not suited to complex development pathways with
multiple non-linear causal factors
– Less appropriate where hard to identify comparison
groups – threat to validity
– When extrapolated from their context, RCT findings
lose claims to rigour (Pritchett and Sandefur, 2013)

25. How best to evaluate in FCAS?
In increasing order of robustness:
• Use of evaluation framework and robust
approach to evidence assessment – e.g.
humanitarian evaluations
• Use of theories of change and contribution
analysis to test causation and assumptions
• Realist evaluation design looking at how
different mechanisms operate in contexts

26. Using an evaluation framework
Questions
Theory/
Approach
Methods
Tools
Establishing a framework for the evaluation provides a
consistent and systematic means to designing the
evaluation, collating and analysing the existing evidence
and the new data created, and generating and
interpreting the results. (Magenta Book para 6.1)

27. Theory or approach
•Results-oriented
•Theory-based
•Participatory/
empowerment
•Utilization-focused
Methods
•Qualitative
•Case study
•Experimental
•Value for Money
•Contribution analysis
Tools
•Document review
•Key informant interview
•FGD
•Direct observation
•Questionnaire survey
•Participatory data collection
•SWOT
•Forcefield
•Stakeholder analysis
•Ranking and scoring
•[Types of analysis; CBA;
VfM; QCA; etc]]
Methods define
which tools and
how to use them

28. Evaluating peacebuilding
• Most useful definition of impact – understand
effects of intervention on conflict drivers
• Conflict analysis is critical – understand/test
relevance of intervention to conflict drivers
• Use of ToC to understand/test assumptions
about how intervention contributes to change
• Experimental approaches usually not useful –
better to look at contribution

29. M&E Group Exercise
• Split into 4 groups
• 2 groups will be responsible for designing an
outline M&E system for a peacebuilding
programme
• 2 groups will be responsible for designing an
outline proposal to do an external evaluation
of the same programme

30. Further Reading
Literature on M&E approaches and methods
• L. Morra Imas, Rist, R., The Road to Results (World Bank, 2009)
• S. Funnell, Rogers, P., Purposeful Program Theory (Wiley, 2011)
• E. Stern et al., ‘Broadening the range of designs and methods for impact evaluation’, DFID
working paper 38, April 2012
• H. White, Phillips, D., ‘Addressing Attribution of cause and effect in small n impact
evaluations’, 3ie Working Paper 15, June 2012
• G.Westhorp, ‘Realist impact evaluation: an introduction’, September 2014
Literature on M&E with specific reference to FCAS
• DFID, ‘Results in Fragile and Conflict-affected States and Situations’, 2012
• DFID, ‘Back to Basics, A compilation of best practices in design, monitoring and evaluation in
fragile and conflict-affected environments,’ March 2013
• L. Schreter, Harmer, A., Delivering Aid in Highly Insecure Environments, 2013
• S. Herbert, ‘Perceptions surveys in fragile and conflict-affected states’, GSDRC Helpdesk
Research Report, March 2013
• DFID, ‘Evaluating impacts of peacebuilding interventions’, May 2014
• J. Puri et al. ‘What methods may be used in impact evaluations of humanitarian assistance’,
3ie working paper 22, December 2014

31. Thank you for listening - any questions?
david.fleming@itad.com