Presented by Lia Florey, MEASURE DHS/ICF International, as part of a symposium organized by MEASURE Evaluation and MEASURE DHS at the 6th MIM Pan-African Malaria Conference.
VIP Call Girls Tirunelveli Aaradhya 8250192130 Independent Escort Service Tir...
Trend Analyses of Nationally-representative Survey Data: What story can be told and what is missing
1. Trend analyses of nationallyrepresentative survey data: What story
can be told and what is missing.
Lia Florey, MEASURE DHS - ICF International
Sixth MIM Pan-African Malaria Conference
October 9, 2013, Durban, South Africa
Symposium 38
2. Acknowledgements
The Multiagency Malaria Control Impact Evaluations are
a joint effort of many partners.
The core team includes members from the following organizations:
•PMI/USAID
– Erin Eckert, Christine Hershey, Rene Salgado
•PMI/CDC
– Achuyt Bhattarai, Carrie Nielsen, Steven Yoon
•MEASURE DHS
– Fred Arnold, Lia Florey, Cameron Taylor
•MEASURE Evaluation
– Ana Claudia Franca-Koh, Samantha Herrera, Jui Shah, Yazoume Ye
Sixth MIM Pan-African Malaria Conference
October 9, 2013, Durban, South Africa
3. Outline
•
•
•
•
•
Introduction to Impact Evaluation Project
General plausibility approach
Challenges to the plausibility approach
Examples
What else can be done
Sixth MIM Pan-African Malaria Conference
October 9, 2013, Durban, South Africa
4. Introduction to Impact Evaluation
Goal: Determine if scale-up of malaria control
interventions has had an impact on malaria outcomes
1. What impact have malaria control interventions had
on malaria-related morbidity and mortality?
2. Can we demonstrate and quantify plausible
association between intervention and impact?
3. What else could have contributed?
Sixth MIM Pan-African Malaria Conference
October 9, 2013, Durban, South Africa
6. Core Analytic
Questions
Input
Process
Output
Outcome
Impact
Question 1: Has the availability of services for malaria prevention and treatment increased and are services
equitably distributed?
a. Funding/spending for malaria programs (from
X
partner organizations and/or domestic funding)
b. Vector management (ITNs, IRS)
X
X
c. Case management
X
X
d. IPTp
X
X
Question 2: Has mortality decreased?
a. All-cause under-five mortality (ACCM)
X
b. Malaria-specific under-five mortality
X
Question 3: Have the malaria incidence and prevalence decreased?
a. Morbidity (anemia prevalence, parasite
prevalence, malaria cases)
b. Is there anecdotal evidence suggesting
additional potential impacts of malaria control
(burden placed on health facilities, etc.)
Question 4: Have other health programs (non-malaria) been scaled up in recent years?
a. Vitamin A, immunizations, etc.
X
X
X
X
X
7. Plausibility approach
• Show trends in scale-up of malaria control
interventions (ITNs, IRS, IPTp, Effective Case Management)
• Show trends in malaria outcomes (Morbidity, Mortality)
• Show trends in other factors that could have influenced
trends in outcomes (Contextual Factors)
• Conclude whether it is plausible that malaria control
interventions reduced malaria-related deaths
Malaria related
interventions
Morbidity
Mortality
Contextual
factors
Contextual
factors
Sixth MIM Pan-African Malaria Conference
October 9, 2013, Durban, South Africa
8. Why use a Plausibility Approach?
• Data on malaria-specific outcomes poor or lacking
• Difficult to measure cause-specific mortality in most
of Africa
– Weak vital registration system
– Cause of death difficult to verify
• We do not have individual-level data needed for
directly measuring causal relationships
– ITN use questions ask about previous night
– Mortality is measured over a five year period
– Exposures to interventions do not always precede outcomes
Sixth MIM Pan-African Malaria Conference
October 9, 2013, Durban, South Africa
9. Impact Model
Sixth MIM Pan-African Malaria
Conference October 9, 2013, Durban,
South Africa
11. Challenges to determining plausibility using survey data
Low levels of coverage throughout evaluation period
– Maybe insufficient to expect impact on mortality
Sixth MIM Pan-African Malaria Conference
October 9, 2013, Durban, South Africa
12. Challenges to determining plausibility using survey data
Mortality trend
Malaria intervention coverage
2000
2010
Lack of baseline data for interventions
– Started measuring half way through evaluation period
– New/improved interventions introduced during period
Sixth MIM Pan-African Malaria Conference
October 9, 2013, Durban, South Africa
13. Example from Angola Impact Evaluation
Sixth MIM Pan-African Malaria Conference
October 9, 2013, Durban, South Africa
14. Challenges to determining plausibility using survey data
Mortality decline began before intervention scale-up
Sixth MIM Pan-African Malaria Conference
October 9, 2013, Durban, South Africa
15. Challenges to determining plausibility using survey data
Mortality trend scenario 1
Malaria intervention coverage
2000
2010
Intervention coverage plateaued but mortality trends
continued
Sixth MIM Pan-African Malaria Conference
October 9, 2013, Durban, South Africa
16. Example from Malawi Impact Evaluation
Sixth MIM Pan-African Malaria Conference
October 9, 2013, Durban, South Africa
17. Challenges to determining plausibility using survey data
Seasonal variation in data collection, Malawi
– DHS low transmission season, MIS high transmission season
– Affects use of interventions as well as outcomes
18. Challenges to determining plausibility using survey data
Ecological fallacy
Sixth MIM Pan-African Malaria Conference
October 9, 2013, Durban, South Africa
19. Challenges to determining plausibility using survey data
Ecological fallacy
Sixth MIM Pan-African Malaria Conference
October 9, 2013, Durban, South Africa
23. What else can be done?
Tell the story with more detail
– Stratifications – by malaria risk, wealth, urban/rural, age
Sixth MIM Pan-African Malaria Conference
October 9, 2013, Durban, South Africa
24. What else can be done?
Tell the story with more detail
– Accessibility, health systems, specific intervention campaigns
25. What else can be done?
Use other methodological approaches
– District-level ecological analyses - Malawi
• Requires large number of sampled districts
• Allows inclusion of contextual factors
• Few national surveys representative at district level
Sixth MIM Pan-African Malaria Conference
October 9, 2013, Durban, South Africa
26. What else can be done?
Use other methodological approaches
– Decomposition analyses - Rwanda
•
•
•
•
Survival models
Individual level
Allows inclusion of contextual factors
Timing issue with exposure data
Decomposition models show that the observed increase in
household bed net ownership, from 8% to 94% could have
explained as much as 45% of the observed decline in ACCM
between 2000 and 2010, equivalent to a reduction of 37
deaths per 1,000 live births.
Sixth MIM Pan-African Malaria Conference
October 9, 2013, Durban, South Africa
27. What else can be done?
• Use other sources of data
– Subnational Anemia & Parasitemia surveys
Sixth MIM Pan-African Malaria Conference
October 9, 2013, Durban, South Africa
28. What else can be done?
• Use other sources of data
– Demographic Surveillance Systems (DSS)
Sixth MIM Pan-African Malaria Conference
October 9, 2013, Durban, South Africa
Timing of the change
Space
Dose-response
Age-pattern
Correspondence malaria morbidity – mortality change
Other factors
LiST deaths averted (magnitude expected)
Reference Rowe paper
Conceptual Framework
These are both plausible patterns for impact. Scenario 1 shows a negative linear relationship between intervention coverage and mortality. Scenario 2 shows a threshold effect.
Timing of collection of intervention data compared to measurement of outcomes
Changing drug policies make it difficult to link trends in treatment with trends in morbidity/mortality
Early surveys did not contain standard questions necessary for calculating some of these indicators (i.e. ITN use).
Difficult to assess a trend with few data points, especially with five-year intervals
Data not always available for the required period
Plausibility versus causality
Time series data on interventions not available
Accounting for possible contextual factors
Alternative scenario are possible with very different implications for the plausibility argument.
Treated nets
Few ITNs in 2000
LLINs distributed en masse ~2007/8
IPTp in 39 countries by 2007
Malawi is an exception (1993)
ACTs
2006/2007
IRS
Usually in targeted areas and not useful as a national measure
ACCM not specific to malaria and therefore cannot be attributed fully to malaria interventions
We see this especially with IPTp, and with ITNs in some countries.
Steep declines during the period before intense intervention began.
Timing of collection of intervention data compared to measurement of outcomes
Changing drug policies make it difficult to link trends in treatment with trends in morbidity/mortality
Early surveys did not contain standard questions necessary for calculating some of these indicators (i.e. ITN use).
Difficult to assess a trend with few data points, especially with five-year intervals
Data not always available for the required period
Plausibility versus causality
Time series data on interventions not available
Accounting for possible contextual factors
This is hypothetical. Can’t test it without disaggregated data.
Significant mortality decline
Significant and rapid intervention coverage
Timing
IPTp not implemented throughout
Data on % of households with ITNs not available from 2000
So successful (look at such high coverage) what next? Will need another approach as coverage gets so high and morbidity so low.
Rapid declines in mortality before uptake of interventions
Rapid uptake of interventions followed by stagnation yet mortality declines continue. How to interpret this?
MIS vs. DHS potential effects
Potential effects of varied epidemiologic conditions (elevation etc.)
Timing of the change
Space, dose-response, age pattern
Correspondence malaria morbidity and mortality change
Other factors
LiST deaths averted (magnitude expected)
Timing of the change
Space, dose-response, age pattern
Correspondence malaria morbidity and mortality change
Other factors
LiST deaths averted (magnitude expected)
Timing of the change
Space, dose-response, age pattern
Correspondence malaria morbidity and mortality change
Other factors
LiST deaths averted (magnitude expected)
Timing of the change
Space, dose-response, age pattern
Correspondence malaria morbidity and mortality change
Other factors
LiST deaths averted (magnitude expected)
Timing of the change
Space, dose-response, age pattern
Correspondence malaria morbidity and mortality change
Other factors
LiST deaths averted (magnitude expected)