Dehradun Call Girls Service {8854095900} ❤️VVIP ROCKY Call Girl in Dehradun U...
Lessons Learned In Using the Most Significant Change Technique in Evaluation
1. Lessons Learned In Using the Most
Significant Change Technique in Evaluation
Mary Freyder, MPH, LMSW
Bridgit Adamou, MPH
January 8, 2020
2. • Data for Impact (D4I) overview
• Sharing lessons from the field – Most Significant
Change (MSC) in five countries under MEASURE
Evaluation Project
• Deep dive - evaluating the Local Capacity Initiative
(LCI) in Uganda
• Questions and answers
Presentation Overview
3. • Generate evidence
• Strengthen capacity
• Ensure data quality
• Integrate gender
• Promote data use
• Learn
D4I works to:
4. • Participatory approach
• Participants give personal stories of significant
change, directly or indirectly related to intervention
• Stories typically analyzed and filtered through
various organization levels
Introduction to MSC
5. MSC process in three steps
1. Decide the types of stories to be collected
2. Gather stories from participants
3. Share stories with all stakeholders to learn
6. • A program adapts to different or changing
contexts, leading to differences in implementation
and outcomes
• Cause and effect are poorly
understood
MSC: Useful when…
• Multiple intervention
components exist
• Evaluation is focused on
learning, not just
accountability
7. • Flexible, captures broad range of results
• Addresses limitations of other, more
traditional methods
• Simple process
MSC: Strengths
• Goal-free, no
predetermined outcome
8. • Bias toward success stories
• Subjectivity in story selection
• Bias toward popular views
• Bias toward views of those who
are good storytellers
MSC: Limitations
9. MEASURE Evaluation’s applications of MSC
• Feed the Future’s Integrating Nutrition in Value
Chains (INVC) program in Malawi
• M&E systems strengthening in Côte d’Ivoire and
Nigeria
• USAID’s Gender, Policy, and Measurement
Program, in partnership with the Suaahara Project
and the Government of Nepal
• USAID’s Sustainable Comprehensive Responses
(SCORE) Project in Uganda
• PEPFAR’s Local Capacity Initiative (LCI) in Uganda
and Ghana
10. • Part of the evaluation of Feed the Future’s INVC
program
• Nutrition promoters trained on MSC in two districts
• 26 volunteer nutrition promoters collected 277 stories
from project beneficiary households
RESULTS:
• Stories were short, lacked details, and were repetitive
• Many of the stories covered multiple domains
• MSC was useful for understanding participants’
perceptions of the key benefits of INVC
Application of MSC in Malawi
11. • In both countries, researchers sought to determine
the most significant changes experienced by
stakeholders in the improvement of each country’s
national HIV M&E system.
• Using a facilitator well-trained in MSC, a
stakeholder workshop was held in each of the
countries to identify stories.
• Groups used an adapted self-assessment tool to
guide discussion, agree on a response, and then
provide sources of evidence for the responses.
• Researchers validated the stories and conducted
key informant interviews to obtain additional details.
Application of MSC in Côte d’Ivoire and
Nigeria
12. RESULTS:
• Engaging stakeholders in the MSC method helped
them identify important changes resulting from
interventions to strengthen M&E systems.
Application of MSC in Côte d’Ivoire and
Nigeria
• Researchers clearly heard
stakeholders’ voices and could
comprehensively and systematically
quantify improvements in each of
the 12 components of M&E
systems strengthening.
13. • Impact evaluation was conducted
to understand the contributions
of two approaches to strengthening
the capacity of Health Facility
Operation & Management
Committees (HFOMCs).
• MSC method was used for focus group discussions
with HFOMC members and community members to
see if significant changes had occurred in their lives.
• A local research organization collected stories from
community members and presented them to the
HFOMC members for group discussion.
Application of MSC in Nepal
14. Application of MSC in Nepal
RESULTS:
• Using the MSC approach with community members
was challenging because the intervention was
implemented by the same organization that
implemented another program and community
members confused the two.
• HFOMC members shared useful stories about how
the project helped them obtain information.
15. • The SCORE project aimed to benefit orphans and
vulnerable children and their caregivers.
• The program’s aims were to build
economic resilience, enhance food
security, improve child protection,
and increase access to education and
critical services.
• Forty randomly selected beneficiaries
and project staff were asked about
positive and negative changes resulting
from the program.
Application of MSC in Uganda
16. RESULTS:
• Because of time and resource constraints, the MSC
framework was used to develop open-ended
questions for the participants.
• The MSC method generated
rich and interesting responses.
Application of MSC in Uganda
• Analyzing the answers revealed
that the layered, multifaceted
components of the intervention
led to positive change,
particularly at the family level.
17. • The MSC approach is time-consuming and requires
careful facilitation.
• People collecting stories should
be well trained and supervised.
• The MSC approach is easy for
participants to understand.
• It is worthwhile to allow ample
time for the interviews—even if
this means gathering fewer stories.
Lessons learned across MSC studies
18. • Follow-up interviews with
beneficiaries, program
staff, or donors can further
strengthen the learning
aspect of this approach
and complement MSC
data.
Lessons learned across MSC studies
• The MSC technique is useful when evaluators
must narrow down the components of participant
observations of what changed.
• This method tends to have a bias for positive
responses.
19. • Research brief: Experiences and Lessons Learned:
Implementing the Most Significant Change Method
• Report: Integrating Nutrition in Value Chains in Malawi:
Using Most Significant Change stories to understand
community experiences
• Case study: A case study to measure national HIV M&E
systems strengthening: Nigeria
• Case study: A case study to measure national HIV M&E
systems strengthening: Côte d’Ivoire
• Report: Uganda’s SCORE program for vulnerable children
and their families: Mixed-methods performance evaluation
D4I resources on the MSC technique
20. This presentation was produced with the support of the United States
Agency for International Development (USAID) under the terms of the
Data for Impact (D4I) associate award 7200AA18LA00008, which is
implemented by the Carolina Population Center at the University of
North Carolina at Chapel Hill, in partnership with Palladium
International, LLC; ICF Macro, Inc.; John Snow, Inc.; and Tulane
University. The views expressed in this publication do not necessarily
reflect the views of USAID or the United States government.
www.data4impactproject.org
Notes de l'éditeur
D4I, which is a MEASURE Evaluation associate award and is funded by United States Agency for International Development, works with countries to:
Generate strong evidence for program and policy decision making
Build individual and organizational capacity
Enhance the use of data for global health programs and policies
MEASURE Evaluation has implemented the MSC technique in evaluations of health programs. This presentation summarizes our experience and shares lessons learned in the application of MSC.
But first, what is MSC? It is a participatory qualitative approach to monitoring and evaluation that involves assessing the changes and impacts as the result of a program from the perspective of program participants or beneficiaries.
It is participatory because program participants themselves and stakeholders are usually involved in deciding types of change to explore, and in analyzing the data, or stories, collected.
Project staff collect stories from beneficiaries, partners, or participants. Story collectors ask questions such as the following: “During the last period, in your opinion, what was the most significant change that took place for participants in the project?” Respondents describe both the change and the reasons they consider it significant.
The stories are then analyzed and filtered through different levels. “Typically” is in italics here because MSC has been implemented in many different ways, ranging from the original version of it, with multiple levels of story review/analysis and voting, to just asking individuals for their stories and researchers analysing those. You will see this illustrated through the MEASURE Evaluation examples coming up.
The process entails three basic steps:
1. Deciding the types of stories that should be collected from beneficiaries, partners, or participants
2. Gathering stories by asking respondents to describe both the change they experienced and the reasons they consider it significant, then collectively determining which stories are the most significant
3. Sharing the stories with stakeholders and contributors to learn about what is valued
MSC is useful when you are evaluating a program that adapts to different or evolving contexts, leading to differences in implementation and outcomes.
When cause and effect is not understood well.
When there are multiple program components
And when your evaluation is learning focused rather than solely focused on accountability.
MSC is flexible: open to broader range of results, including those that may not have originally been intended by the program.
Complementary to more traditional methods: can help address limitations of other methods. For example, if you are using a structured survey only, you may miss unanticipated changes, negative or positive.
Simple, straightforward process that anybody can understand and use. Compared to other monitoring approaches, it is easy to communicate across cultures. There aren’t any indicators, so there is no need to explain what an indicator is. Everyone can tell stories about events they think were important.
Goal-free: no assumption of what success should look like. Because it is not a linear model, there are no predetermined outcomes.
This method is biased towards successes – particularly to make the storyteller look good. Evaluators can overcome this by having a domain of change for negative stories.
There is subjectivity in story selection. Recording the reasons for selecting stories helps strengthen the process
There are biases toward popular views.
And there are also biases toward views of those who are good storytellers. This is why don’t use as stand-alone tool.
MEASURE Evaluation applied most significant change in five different settings to monitor and evaluate various community based or complex programs.
One was a nutrition program in Malawi.
Another looked at how national government HIV M&E systems in Cote d’Ivoire and Nigeria were strengthened.
In Nepal, MSC was used as part of an impact evaluation involving health facility operation and management committees.
The Sustainable Comprehensive Responses, or SCORE, program in Uganda aimed to benefit orphans and vulnerable children and their caregivers.
And then lastly there was the PEPFAR-supported global initiative to support local partners in HIV policy advocacy for highly vulnerable populations such as men who have sex with men, transgender women and sex workers. My colleague, Mary, will discuss this application more in-depth.
MSC was intended to be one aspect of qualitative data collection for a large impact evaluation of Feed the Future’s Integrating Nutrition in Value Chains program.
Volunteer nutrition promoters were trained on MSC in Malawi’s Mchinji and Lilongwe districts to obtain information from participants about their experiences with the INVC program.
Over the course of a year, 26 nutrition promoters, who were also program implementers, collected 277 MSC stories from project beneficiary households.
Outcomes:
Because the nutrition promoters wrote the stories in their notebooks, summarizing them rather than writing them verbatim, the stories tended to be very short, had few details, and were repetitive. The similarities among stories made it difficult for the promoters to discuss, rank, and select the most significant stories.
Many of the stories covered more than one domain of change, which also made it challenging for promoters to assign the stories to the domains.
The MSC method was useful for understanding participants’ perceptions of the key benefits of and challenges to the INVC program.
(Due to the cancellation of the impact evaluation, no other qualitative data was collected to supplement the MSC stories.)
In two case studies, researchers sought to produce evidence of how Côte d’Ivoire and Nigeria’s national HIV M&E systems were strengthened. The MSC technique was applied in both countries to determine the MSCs experienced by stakeholders in the improvement of each country’s national HIV M&E system.
With the support of a facilitator well-trained in MSC, a stakeholder workshop was held in each of the countries to identify stories.
Participants started by self-selecting into groups.
Next, each group used an adapted self-assessment tool to guide discussion, agree on a response, and then provide sources of evidence for the responses. Groups reported to the plenary to verify each other’s findings.
Afterwards the researchers validated the stories and conducted key informant interviews to obtain additional details and perspectives on the MSCs that were identified in the workshop.
Researchers found that engaging stakeholders in the MSC method helped them identify the important changes resulting from interventions to strengthen M&E systems, which also helped to raise interest in these interventions and the defined domains of change.
Using the MSC method also meant that researchers clearly heard stakeholders’ voices and could comprehensively and systematically quantify improvements or successes in each of the 12 components of M&E systems strengthening.
MEASURE Evaluation conducted an impact evaluation to understand the contributions of two approaches to strengthening the capacity of the Health Facility Operation and Management Committees (HFOMCs) in Nepal.
At the end line of the evaluation, we used the MSC method to conduct eight focus group discussions with HFOMC members and community members to see if significant changes had occurred in their lives as a direct result of their participation in the program or as a result of the program’s interventions overall.
The plan was to apply the MSC method in two ways: (1) HFOMC members were to engage in focus group discussions about changes they experienced through program participation and (2) HFOMC members were to collect stories of change from the community for the HFOMC to discuss and vote on. The latter story collection did not happen as planned, however, because of HFOMC members’ limited time and limited skills in facilitating discussions, among other reasons. Instead, a local research organization collected the stories from community members and presented them to the HFOMC members for group discussion.
Using the MSC approach with community members proved challenging because the intervention was implemented by the same organization that implemented another program and community members confused the two.
However, HFOMC members shared useful stories about how the project helped them obtain information.
USAID’s Sustainable, Comprehensive Responses (SCORE) project operated in Uganda to benefit orphans and vulnerable children and their caregivers.
The program’s aims were to build economic resilience, enhance food security, improve child protection, and increase access to education and critical services.
MEASURE Evaluation used the MSC method to assess the project’s effects on beneficiaries and the strengths and challenges associated with implementation.
Using the MSC techniquie, 40 randomly selected beneficiaries and project staff about positive and negative changes resulting from the program.
Time and resource constraints prevented respondents from gathering to discuss and vote on the changes that had the most impact. Instead, we used the MSC framework to develop open-ended questions for the participants, which generated rich and interesting responses.
Analysis of these answers revealed that the layered, multifaceted components of the intervention led to positive change, particularly at the family level.
Before we dive deep into the LCI Uganda evaluation, I’ll share a few lessons learned across all four studies. Mary will highlight a few of these in her presentation.
The MSC approach is time-consuming and requires careful facilitation. If people capturing stories lack sufficient time and skills, it is better to select others better suited to use it correctly, or plan to spend significant time training data collectors to record stories—or avoid the method altogether.
People collecting stories should be well trained and supervised so the stories are thoroughly captured with detailed notes.
The MSC approach is easy for participants to understand.
It is worthwhile to allow ample time for the interviews—even if this means gathering fewer stories. It’s important that data collectors can fully explain the questions and that participants have enough time to think about their experiences and articulate their responses.
Follow-up interviews with beneficiaries, program staff, or donors can further strengthen the learning aspect of this approach and complement the MSC data.
The MSC technique is useful when evaluators must systematically narrow down the components of participant observations of what changed.
As mentioned earlier, this method tends to have a bias for positive responses. Specifically asking a question about most significant negative change can address this potential drawback.