Using the Outcome Mapping concept of outcomes in evaluations to enrich theories of change and to foster an evaluative culture
•Download as PPTX, PDF•
2 likes•1,255 views
Through three international development project and programme evaluations in Mongolia, India and the western Indian Ocean islands and planning, monitoring and implementation in India, we used OM concepts to refine explicit or implied theories of change and derive emergent ToC. Among the OM principles and concepts we applied were outcomes as behavioural change, boundary partners, and contribution of multiple actors and factors. Subsequently OM concepts have been used to strengthen project and programme management within the change agents.
Recommended
Recommended
More Related Content
More from John Mauremootoo
More from John Mauremootoo (19)
Recently uploaded
Recently uploaded (18)
Using the Outcome Mapping concept of outcomes in evaluations to enrich theories of change and to foster an evaluative culture
- 1. Using the Outcome Mapping concept of outcomes in evaluations To enrich theories of change and to foster an evaluative culture John Mauremootoo Richard Smith Presentation given at the American Evaluation Association on 14th November 2015 in the session entitled: Applying Outcome Mapping to the evaluation of socially transformative projects, programs, and policies Photo: Public Domain
- 2. Project locations Mongolia World Bank & SDC Funded DEFRA & DFID (UK) funded India Indian Ocean EU funded
- 3. Mongolia Strengthen the capacity of Civil Society Organisations to undertake social accountability monitoring
- 4. People and Pollinators initiative: Enhance pollination services for sustainable agriculture and biodiversity conservation India All photos of India from the Centre for Pollination Studies (CC-BY-NC)
- 5. Manage coastal zones for environmental sustainability and livelihood generation Indian Ocean All Indian Ocean photos by John Mauremootoo (CC-BY)
- 6. Theories of Change Mongolia: Results chain from proposal India: Logframe from proposal Indian Ocean: Logframe from proposal
- 7. The Use of Outcome Mapping Concepts Development Implementation Closure Mongolia Indian Ocean India Summative evaluation 3 years after project closure Summative evaluation 3 years after project closure Introduced 8 months into implementation for internal planning, monitoring & evaluation Allprojectsplannedandimplemented usingdonor-mandatedlogframes
- 8. Value of using OM Concepts to enrich Theories of Change Theory of change – the preconceived map of the journey Theory in use – what actually happens
- 9. Value of using OM Concepts to enrich Theories of Change Outcome mapping makes step two explicit by using concepts such as outcomes as behavioural change, boundary partners and contribution not attribution
- 10. Outcomes as Behavioural Change Mongolia Activities Outcomes As indicators of effectiveness & sustainability An effectiveness and sustainability evaluation framework
- 11. Outputs Outcome Challenge Boundary Partner Progress Markers Project Purpose ControlInfluenceInterest Outcomes as Behavioural Change India Activities Inputs
- 12. Outcomes as Behavioural Change
- 13. Boundary Partners From Tableatny on Flickr Individuals, groups or organisations with which the program interacts directly and which the program hopes to influence
- 14. Boundary Partners Mongolia Limited scope of intervention meant the selection of participants / boundary partners by the implementing body was central to any achievements. Evaluation data collection was directly from those who participated as trainees / mentees in the intervention.
- 15. Boundary Partners India The Boundary Partner concept highlighted the importance of site-specific political and administrative structures e.g. very different intervention models in the two project sites (not explicitly accounted for in the project design)
- 16. Boundary Partners Indian Ocean Given the short duration of the projects (<18 months) most of the successful projects were those which built on existing partnerships.
- 17. Contribution not Attribution The intervention may be only one among many factors that contributes to an outcome
- 18. Contribution not Attribution Mongolia The intervention introduced tools that were previously unused in Mongolia to CSOs with little or no experience of social accountability work therefore the contribution of the intervention could be easily identified. Several direct outcomes led to further outcomes and even impact. In such cases other factors and actors contributed.
- 19. Contribution not Attribution India Appreciation of the value of partnerships to produce sustainable outcomes Valuable for team spirit as everybody understood that they were contributing to results
- 20. Contribution not Attribution. Some projects failed to sustain outcomes nor contribute to impact because of failure to address actors and factors in the enabling environment (outside the control of the project) e.g. policy makers, legislation and the provision of infrastructure. Indian Ocean
- 21. Outcome Mapping Concepts Strengthen the Design or Management of Interventions
- 22. OM Concepts Inform Further Support Mongolia Evaluation informed what to focus on for a follow up project and made a case for further support Recommendations for discussion contributed to new project design
- 23. India OM Concepts Adopted Planning, monitoring and evaluation is now seen as integral to project implementation by project leaders who are looking to incorporate OM concepts into new project development.
- 24. Donor Interest Recommended The project was rated A++ (outcome substantially exceeded) in its final external review and it was recommended that the PME manual (based on an OM-LFA fusion) should be publicised by defra as ‘best practice’ India
- 25. Evaluation results inform programme development. Indian Ocean Incorporation of lessons learned from evaluation into guidance documents for follow up EU programme – emphasis on partnerships, participation and building upon existing actions.
- 26. OM concepts helping us to become better Cartographers OM helps us to gain a better understanding of the mental maps of those in the project’s ecosystem Gervase R. Bushe ~ Professor of Leadership and Organization Development. A quote on leadership which could equally be applied to project implementation. Leadership is a lot about influencing people and you cannot do that if you don't understand their maps.
- 27. Acknowledgements Mongolia: Jeremy Gross (Co-evaluator) Amarbayasgalan Dorj (Co-evaluator) India: Parthib Basu (Project Leader India & evaluation support) Barbara Smith (Project Leader UK & evaluation support) Soumik Chatterjee (Project Manager & evaluation support) Indian Ocean: Dunstan Kishekya (Co-evaluator)
Editor's Notes
- The Interventions: Mongolia (World Bank Governance Partnership Facility) and Swiss Development Cooperation (SDC) Eastern India (DEFRA & DFID Darwin Initiative) Western Indian Ocean (EU through the Indian Ocean Commission)
- Funded through the World Bank’s Governance Partnership Facility and Swiss Agency for Development and Cooperation’s CSO/NGO Capacity Building interventions in Mongolia
- Mongolia: Results chain loosely mapped to the funder’s (Governance Partnership Facility) log frame indicators.
- Outcomes as behavioural change To make the process of change explicit
- Idea of 2 types of effectiveness and sustainability. OH tool provided a clear standard for quality: consistent, verifiable outcome descriptions Provided clear criteria for assessing the effectiveness and sustainability of the intervention. Effectiveness - 1: Had knowledge gained through a small ($300K), short-term (13 month) intervention been put into practice by CSOs outside of the intervention 2 years after it ended? If so, could the contribution of the intervention be identified with confidence and was it significant (social accountability monitoring as a concept not widely known and new tools introduced and used)? Effectiveness – 2: Did change in actors directly influenced by intervention catalyse change in their BPs – any ‘pathways of influence’ (Yes: the findings from a social accountability exercise being used to influence policy decisions on e.g. wheelchair access to buildings, patient access to toilets in hospitals, companies fulfilling social responsibility commitments…) Sustainability: CSOs created new positions / areas of work / got new funding / created new learning networks.
- India: Filled in a gap in the logframe through the introduction of a level (PM, OC, BP) between the output and purpose to address the “who question” – who does what with the project outputs to contribute to outcomes and impacts?
- Indian Ocean: Provided a common framework for review of very disparate grassroots interventions from sea wall construction to pearl farming in terms of sustained outcomes and impacts and the contribution of the intervention.
- This concept provides a focus on who the project worked with directly in order to contribute to change
- Mongolia: Limited scope of intervention meant the selection of participants / boundary partners by the implementing body was central to any achievements. Evaluation data collection was directly from those who participated as trainees / mentees in the intervention.
- Indian Ocean: Near exclusive focus of some projects in the sphere of control but not sphere of influence Some projects failed to sustain outcomes nor contribute to impact because of failure to address actors and factors in the enabling environment (outside the control of the project) e.g. policy makers, legislation and the provision of infrastructure.
- Feedback from the World Bank Institute and the M&E advisor for follow project that is starting now, 2 years after the evaluation, indicate: The evaluation was ‘very useful’ for WBI because it clearly documented results (intended and unintended) and therefore a) informed the design of further support – what to focus on – and b) helped make the case for new support (new $3 million project, WB and SDC funding) The recommendations for discussion directly contributed to the inclusion in the new project design of a) an participatory inception design workshop; b) periodic learning events; c) ongoing monitoring; d) mid-term and final outcome evaluations.