Transitions in M&E of SBC Handout

1. What it is Why it should be used for public
health programming
What are its limitations? What
does it obscure? When is it
inappropriate?
Example research questions
Experimental—Originating in the material and
natural sciences, experimental studies--and
especially the kinds of experimental studies that
use randomization to control variance-- are
considered the gold standard. In
social/behavioral research controlled studies
explore the extent to which a difference can be
shown to correlate with introduction of an
independent variable.
Unexpectedly high degrees of correlation
between or among variables can generate
hypotheses different from the ones that inspired
the study.
(Note, however, that correlational studies do not
necessarily require experimentation)
When carefully done, a series of controlled studies
can suggest cause-and-effect relationships.
When specific indicators are directly measurable
(e.g., uptake of ANC services), experimental
methods can also usefully compare outcomes in
the experimental and controlled conditions.
• Expensive and time consuming
• Not possible to control for many
variables readily acknowledged as
critical.
• Correlation does not equate to
causation
• Repeated measures can be
conducted longitudinally but loose
power over time
• By their very nature, they have
limited external validity to other
situations
The more complex programs are, the more
these drawbacks become significant and
interrelated.
• Is activity X performing
better than activity Y on this
specified indicator of
success?
Quasi-experimental research designs, like
experimental designs, test causal hypotheses.
In both experimental (i.e., randomized
controlled trials or RCTs) and quasi-
experimental designs, the program is viewed as
an ‘intervention’ in which a treatment (the
elements of the program) is tested for how well
it achieves its objectives, as measured by a pre-
specified set of indicators. A quasi-experimental
design by definition lacks random
assignment, however. Quasi-experimental
designs identify a comparison group that is as
similar as possible to the treatment group in
terms of baseline (pre-intervention)
characteristics. Assignment to conditions
(treatment versus no treatment or comparison)
is by means of self-selection (by which
participants choose treatment for themselves)
or administrator selection or both of these
routes. (Shadish, et al., 2002)
• Quasi-experimental methods offer practical
options for conducting impact evaluations in
real world settings. By using pre-existing or
self-selected groups such as individuals who
are already participating in a program these
methods avoid the ethical concerns that are
associated with random assignment – for
example, the withholding or delaying of a
potentially effective treatment or the provision
of a less effective treatment for one group of
study participants.
• Quasi-experimental methods can be used
retrospectively, i.e., after the intervention has
taken place. In some cases, especially for
interventions that are spread over a longer
duration, preliminary impact estimates may be
made at mid-term. It is always highly
recommended that evaluation planning begins
in advance of an intervention, however. This is
especially important as baseline data should
be collected before the intended recipients are
exposed to the program activities.
• When you want to avoid the problems of
threats to validity associated with non-designs.
• The largest potential pitfall in quasi-
experimental methods is the risk of
obtaining a poor quality match. The
comparison group needs to be as similar
as possible to the treatment group before
the intervention. Selection bias – the
possibility that those who are eligible or
choose to participate in the intervention
are systematically different from those
who cannot or do not participate.
Observed differences between the two
groups in the indicators of interest may
therefore be due – in full or in part – to an
imperfect match rather than caused by
the intervention.
• Because quasi-experimental methods
lack random assignment, conclusions
made about causality on the basis of
such studies are less definitive than
those elicited by a well conducted
randomized controlled trial (RCT). There
is a tendency to focus on statistically
significant findings to the detriment of
statistically insignificant results.
• What effects does a free food
scheme having on the nutritional
status of children?
• What is the effect of maternal
alcohol use when the mother is
pregnant?
• What the effects of academic
counseling on the academic
achievement of children from
divorced vs. non-divorced
families?

2. What it is Why it should be used for public
health programming
What are its limitations? What
does it obscure? When is it
inappropriate?
Example research questions
Mixed methods refers to an emergent
methodology of research that advances the
systematic integration, or “mixing,” of
quantitative and qualitative data within a single
investigation or sustained program of inquiry.
The basic premise of this methodology is that
such integration permits a more complete and
synergistic utilization of data than do separate
quantitative and qualitative data collection and
analysis.
A pragmatist paradigm - Mixed-Method studies
have emerged from the paradigm wars between
qualitative and quantitative research
approaches to become a widely used mode of
inquiry. (Terrell 2012)
“Integrating quantitative and qualitative data
collection and analysis in a single study or a
program of enquiry.” – Creswell et al 2003.
• Can provide more complete knowledge
• Words, pictures, and narrative can
add meaning to numbers, and vice
versa, numbers can add precision to
words, pictures, and narratives.
• Can answer a broader, more
complete ranges of
evaluation/research questions
because not confined to a single
method, approach, or paradigm
• Complementarity – Each can overcome
weaknesses of the other –
qualitative/quantitative methodologies
• Triangulation – Can provide stronger evidence
base
• Generalizability – Can increase
generalizability of results
• In practice it can get complicated or
require more time and resources
• Needs researcher with equal
ease in both quanti/qualit
methods- very few people are
skilled at both
• Often done concurrently -
added costs of data collection
• Choosing which method as
main focus - balancing depth
versus breadth
• Challenges in writing up
integrated results
• Details of mixed method research are still
being worked out by methodologists
• Theoretical implications of
mixing paradigms
• Qualitative analysis of
quantitative data
• Interpreting conflicting results
• Inappropriate for which contexts: What
are credible results for stakeholders?
• What is the impact of the two
intervention models on infant and
young child feeding practices
among mothers with children 0 to
23 months?
Traditional Qualitative—
• Seeks to understand a topic or
problem from the perspective of the
population
• Asks why, how, and under what
circumstance things occur
• Can include observation of actual
behavior as well as asking about
behavior and relationships
• Enables the research participant’s
answers to help determine the paths
of inquiry
Includes a variety of methods which aim to
learn the unexpected, since answers to
questions are not fixed choices. Qualitative
methods provide the greatest opportunity for
those whose lives are studied to tell their stories
in their own voices, as well as to identify and
explore unanticipated factors, relationships, and
• To understand behavior in its natural context
(participant-observation,)
• To learn why something has happened or did
not happen
• To learn how all categories of people involved
experienced an SBC intervention (e.g.,
mothers, fathers, implementers, children, etc.)
• To gain a holistic understanding of the entire
situation before implementation, during
monitoring, and after completion
• To develop indicators (including quantitative)
that are derived from a holistic understanding
of the local situation and make sense to the
people involved. This results in measurements
of success that are both meaningful and
appropriate.
• To understand the complexity of how a
program, strategy, etc. is working or not and
why (especially with participant-observation).
• It is not possible to find the prevalence or
incidence of practices, disease, beliefs or
conditions in community, health facility,
etc.
• Reliability in participatory qualitative
research can be dubious
• When it’s important to measure the
extent of change in addition to
understanding how things have changed
and why, mixed methods are preferable.
• It is inappropriate to use a qualitative
method for the wrong reasons, e.g., as a
‘fall-back method’ or to speed up the
evaluation or ‘anyone can do it’.
• Participatory qualitative methods are
inappropriate when they would favor one
portion of the community over another or
exacerbate intra-community tensions
(true of any participatory method).
• What is going on here and what
would be best and most feasible
to change or support in order to
improve health? (e.g., the
behavior change may need to be
by those in the health care
system rather than by the
patients)
• How do we identify locally
important indicators to best
measure that change?
• What are the cultural, religious,
class, ethnic and race (as locally
defined), political, logistic, media,
environmental, financial,
emotional factors that influence
the adoption of the recommended
behaviors?
• How can we engage key
influencers who may offer support

3. affected groups. for adopting new behavior?
What is it Why it should be used for public
health programming
What are its limitations? What
does it obscure? When is it
inappropriate?
Example research questions
Complexity-aware-- A group of approaches
well suited to monitor and evaluate complex
projects in which the relationship between
cause and effect (i.e. theory of change) is not
well understood or during which the
implementation process changed from the
original plan. Creating detailed work plans in
advance is difficult. Copes with nonlinearity.
Complex projects are characterized by a high
degree of uncertainty. It is unknown how to go
about solving a problem. Such projects are
also characterized by a low degree of
agreement among stakeholders.
• There are sometimes few or no reliable
quantitative indicators of program processes
and effects.
• There are sometimes multiple or
undetermined pathways (i.e., theories of
change) for change at the onset of program
activity (USAID, 2013).
• Multiple and diverse stakeholders may be
involved at some level in decision making
about the project which makes the project
socially complex (USAID, 2013).
• It provides less clarity on the steps the
change agent took to influence
outcomes.
• Requires significant resource investment
and buy-in from change agent and social
actors.
• Documentation and stakeholder recall,
both of which are affected by subjective
bias, play a role in determining what data
is collected and substantiated.
• The methodology is flexible and so less
standard across different applications
which in turn reflects comparability.
• Most evaluators are still unskilled in using
it
• What changes in organizational
capacity have social and behavior
change (SBC) entities made
since the beginning of the
intervention?
• How have government entities
changed SBC practices or policy
as a result of advocacy efforts.
Scenario for table discussion:
The DC-based NGO Crusade for Good Behavior has recently been awarded a $3.5M contract to implement a broad-spectrum Zika prevention
program for three years with an emphasis on sexual and reproductive health-related behavior change. The project will be delivered to resistant
communities through a weak health system in a large state (12M people) in a Latin American country. The project will be implemented in the entire
state and focus on interpersonal communication and community mobilization. In Year 1 approximately 450 frontline health workers (e.g., public
health nurses, doctors, midwives and community health workers (CHWs)) will be trained as Zika prevention educators. Once a cohort is trained,
community meetings will be held in villages in the state to orient, train, and get the commitment of community leaders to take action in support of
Zika prevention. Each CHW is responsible for 150 homes will visit at least five homes each day for a period of 2 years. All frontline health workers
will personalize messages, introduce priority practices and provide encouragement on ways to prevent Zika during home visits, support group
meetings, and sessions at health centers. Successful strategies will be identified in Year 1 that lend themselves to rapid scale-up in the second year.
The donor is adamant that the evaluation approach be improvement driven and innovative. In particular, the donor is interested in learning about
the interrelationships between actors and the path leading from the programs’ actions to influence on each actor alone and on the way actors
influence each other.