This document discusses monitoring and evaluating the scale-up of the Standard Days Method (SDM) family planning program in multiple countries. It provides background on a 5-year study of SDM scale-up using the ExpandNet/WHO model. The document outlines the importance of monitoring and evaluation to guide the scale-up process and assess outcomes. It presents the SDM scale-up logic model and operational framework. Metrics for monitoring benchmarks and indicators are proposed, along with data sources and tools for collection. Initial monitoring results are reported for some countries. Challenges of scaling up SDM integration across health systems and service coverage are also examined.

1. Monitoring and Evaluating Scale-Up:
Methodological and Programmatic
Challenges
EXPANDING FAMILY PLANNING OPTIONS

2. BACKGROUND

3. Studying SDM Scale Up
(2007-2012)
• 5 year prospective, multi-site,
comparative study of process
and outcomes of scaling up a FP
innovation (SDM)
• Uses ExpandNet/WHO model for
planning, monitoring and
research

4. Scaling up SDM®
Democratic Republic of Congo
Guatemala
India (Jharkhand)
Mali
Rwanda

5. SDM: From Research to Practice
Scale-Up
Integration Case Studies
Studies 2007-2012
Operations 2005 - 2007
Research
2003- 2005
Pilot Studies
2000-2004
Method Concept &
Efficacy Trial
1999-2002

6. Why monitor and evaluate
SDM scale up?
E
• Guide scale-up process
• Maintain stakeholder
Evidence(+) Evidence (+)
momentum and
Practice (-) Practice (+)
accountability
• Assess whether scale up P
is achieved
(outcome/impact) Evidence (-) Evidence (-)
Practice (-) Practice (+)
• Contribute to growing
evidence base of scaling
up, with focus on M&E

7. Beyond SDM…Rigorous monitoring and
evaluation of scale up
Theory-based methods and tool kit to study
scale up process and outcomes, including:
 Research questions and
hypotheses
 Logic model, indicators,
benchmarks
 Access data base/reporting forms
 Baseline/endline instruments
 Quality assurance tools

8. MONITORING & EVALUATION OF
SCALE UP: DESIGN, DATA SOURCES
AND INDICATORS

9. WHO/Expandnet
Scale up
Framework
• Ensures that ‘systems’ are not forgotten
• Evidence to guide strategic choices and adjustments
• Encourages participatory approaches with multiple
stakeholders
• Creates consciousness of rights and equity issues
• Offers common scale up language

10. SDM Scale Up Logic Model Scaling Up
Strategy
Problem: Gap in availability & access to SDM services
INPUTS PROCESS OUTPUTS OUTCOMES
• Staff • TA for systems • Providers • Provider
• Partners adjustment trained competency
• Funds • Advocacy • Clinics offering • Awareness
• Capacity SDM and use
• CycleBeads
Building • Demand • Availability of
• QA – oriented IEC quality
monitoring & • Supportive services
supervision partners/ • Supportive
stakeholders policies
• Systems
Harmonization
Impact: increased sustained availability of SDM

11. Operationalizing Scale Up
1. Iterative,
participatory
process with
stakeholders to
select indicators
BEGIN
WITH
2. Set baselines and THE END
targets based on IN MIND
indicators

12. Defining success in scale up
Availability of quality SDM services at national,
sub-national, organizational level
Availability of quality SDM services at SDPs
Provider capacity

13. Monitoring benchmark
scale-up indicators
Automated country-level and
Develop scale up Develop Access
donor reports for program
indicators data base
management

14. SDM scale-up monitoring data base,
Microsoft Access 2007

15. Data sources
M&E and Case Study
Event tracking
Guided discussions (timelines)
Semi annual with staff (quarterly)
benchmark monitoring
Individual interviews
Community surveys & with stakeholders
Most Significant facility assessments
Change (MSC) story (1-3 times)
collection (1-2 times)
(1-2 times)

16. M&E Tool Kit for SDM Scale Up
• Benchmark tables
• FP service statistics
Monitoring & • Access data base
supervision • Staff discussion guides
tools • Event tracking (timelines)
• Knowledge Improvement Tool
• Client follow-up Interviews
• Household survey instruments
• Facility Assessment tool
Evaluation tools • Provider interview guide
• Most Significant Change (MSC) story
collection

17. Quality Assurance Tools
Provider Refresher Client Follow Up

18. Most Significant Change stories…
start with a question
“Looking back over the last year, what do you
think was the most significant change you have
experienced as a result of SDM being offered in
your community?”
And ask why

19. Most Significant Change (MSC)
Provides different type of
Action
information to document and
improve scale-up
• Scale-up process and outcomes
Learning
not detected by quantitative
monitoring
• Unanticipated processes/effects Stories
of scale up
• Meanings of scale-up process and
outcomes to partners,
stakeholders, communities
• Intangible aspects of scale up
(advocacy, leadership, gender
PROJECT
equity, informed choice)
ACTIVITIES

20. Sample Evaluation Questions:
Scale-Up Outcomes
• What is the experience of women and men with
Client SDM when scaled-up? (Knowledge, attitudes and
use)
• Is SDM offered correctly by providers?
Service • How does SDM introduction influence quality,
provision availability and use of overall family planning
services?
• To what extent has SDM been integrated into
System training, IEC, procurement and distribution, and
integration HMIS? Is it included in norms, protocols and
guidelines?
Resource • What is the level of resources dedicated to SDM?
mobilization

21. Sample Evaluation Questions:
Scale-up Process
Resource team • Do user organizations assume the roles,
responsibilities and ownership of the
resource team during scale-up process?
Advocacy/ • What is the role of SDM champions? What
Dissemination strategies work best?
Organizational choices • Has SDM been offered outside traditional
public sector service delivery?

22. MONITORING & EVALUATION
“RESULTS” TO DATE

23. Baseline Stakeholder Interviews:
Health/FP program managers and policy
makers in Guatemala (n=20)
Political commitment Yes, SDM already integrated (norms, training,
to SDM scale up materials)
Political factors in SDM Some not convinced a natural method can be modern
scale up and effective and demand is sufficient demand. FBOs
and community based NGO networks strong
supporters
SDM Aware of SDM (but lack specifics, esp. efficacy)
knowledge/attitudes
Ability of MOH to Within their mandate. If there is demand, they will
manage SDM scale up support it.
Integration of SDM Not yet. If high SDM ‘demand proved’ it would be
into annual planning/ integrated.
budgeting processes

24. Baseline Provider interviews/facility assessments
in Rwanda (n=155 and n=109)
SDM integration • 2/3 of providers have seen protocols
into • Most unfamiliar with norms (newly introduced in Rw)
norms, guidelines, p
olicies
Status of SDM • 60% of providers have offered SDM (42% in last 3
services months)
• 70% have been offering SDM between 1-5 years
Correctness of • Most providers offer SDM competently, do not find SDM
SDM info counseling difficult
Service delivery • Providers only have 4-10 min for counseling on FP – not
environment enough
Status of SDM • 91% of visited facilities offering FP offered SDM.
services CycleBeads found in most.
• Only 17% of facilities displayed FP info (SDM/LAM are
integrated into IEC)

25. Process Tracking Tool: Events Timeline
FAM project SDM
begins. extended
Rwanda is SDM
included in in UNFPA
picked as
focus country performanc zone (full DHS
e-based Training of integration 2010, incl
finance trainers for of SDM in FP udes SDM
SDM PSI Rwanda) community-
included in mechanism
based
MIS family National
distribution
planning SDM Pre-service training of
registers, starts in
included training trainers with
client cards Rwanda, in
in mini- activities the MOH
and report cluding
DHS begun (1 trainer/2
templates SDM
districts)
March October May July July Novembe Februar Februar March June
2007 2007 2008 2008 2008 r 2008 y 2009 y 2010 2010 2010

26. Jharkhand: Snapshot of
Progress Toward Benchmarks
• SDPs that • Public or private • SDM & LAM in
include FAM as orgs including IEC activities,
part of the FAM in-service materials &
method mix training mass media
1250 4 5
(60%) (67%) (100%)

27. Performance benchmarks: Jharkhand
Selected Indicators as of Jan 2011
Proportion of SDPs with SDM in method mix 1250/2100 (60%)
Providers trained 6700/15,000 (47%)
No. of resource orgs 3/8 (38%)
SDM included in key policies, norms, protocols 2/3 (67%)
SDM in pre-service training Initiated
Public or private training orgs include SDM in in- 4/6 (67%)
service training
Commodities in logistics & procurement systems In progress
SDM in IE&C materials 5/5 (100%)
SDM in HMIS 1/2
SDM in surveys (DHS) Under discussion
Funds leveraged for SDM $196,000 (est’d)

28. Availability of SDM: FP Service Statistics
Jharkhand Six Districts
40000
Addt'l 3 Districts
35000
30000
25000 No Data Av'l for Gumla
20000
15000
First 3 Districts
# users
10000
5000
0
April'08 Sept FEB JULY DEC'09 MAY OCT
Tubectomy NSV I U D (C T) Oral Pill Users Condom SDM LAM
Jharkand service data, through Jan 2011

29. Monitoring SDM Uptake
during Scale Up
Jharkhand Six Districts
4000
3500
3000
2500
Addt'l 3 Districts
2000
# users
1500
First 3 Districts
1000
500
0
April'08 Aug DEC'08 APRIL AUG DEC'09 APRIL AUG DEC'10
SDM LAM
Jharkand service data, through Jan 2011

30. SDM availability: Phased scale up
Jharkhand, India
Phase 1, started Jan 2008
Pop: 3,765,983
Phase 2, started Feb 2010
Pop: 2,755,023
Phase 3, started Nov 2010
Pop: 5,520,869

31. SDM availability: Phased scale up
Democratic Republic of Congo
2003 - 2008
2008 - 2010
None
Health Zones
in the DRC

32. SDM Integration Progress
Policy Environment - Vertical Scale Up
(June 2011)
Norms & Training Supervision Health Info Supply Budget line-
procedures curricula Systems Distribution CBs
DRC
Mali
Rwanda
India
Guatemala

33. SDM Integration Progress
Service Coverage – Horizontal Scale Up
(June 2011)
% SDPs offering 5 year goal
SDM (Jan 2011) (% of country)
DRC 93% 75%
Mali 84% 90%
Rwanda 84% 95%
India 60% 50% of Jharkhand’s 22
million pop)
Guatemala 48% (3 demonstration
departments, 1/6 of country)

34. Proposed indicators for “graduation”
from technical assistance
• Accomplishment of
benchmarks
• Complete transfer of
responsibility to resource
organizations for all vertical
and horizontal elements
• Sufficient level of ownership
within and across key FP
actors/champions and key
subsystems

35. Rebecka Lundgren
Institute for Reproductive Health
Georgetown University
lundgrer@georgetown.edu
www.irh.org