A presentation from the 2014 Annual Results and Impact Evaluation Workshop for RBF, held in Buenos Aires, Argentina.

1. Using Operational and HMIS
data for Program Monitoring
and Impact Evaluation - Zambia
2014 Results & Impact Evaluation Workshop
Zambian Delegation
25th March 2014

2. Zambia RBF Model
 One of the few examples of “contracting in” through the public
health sector
 Quasi Provider-Purchaser split by different levels of the Zambian
Health Care Delivery System
 Quantity andQuality data verification
 SteeringCommittees (SCs) as Independent Verifiers
 Periodic External Verification
 Performance-BasedPayments
 “Fee-for-service” on a set of MNCH indicators at Health Centres
 Performance Evaluation Framework for District Medical Offices
 Managerialand financial autonomy of health facilities
2

3. Operational Data
3

4. Trends in PerformanceIndicators:
Q2 2012-Q42013
0
2,000
4,000
6,000
8,000
10,000
12,000
14,000
16,000
Q2 2012 Q3 2012 Q4 2012 Q1 2013 Q2 2013 Q3 2013 Q4 2013
Number
Skilled Deliveries
ANC prenatal and
follow up visits
Postnatal visit
Fully vaccinated
child
Third dose
Fansidar IPT
Preg. women
given Niverapine
and AZT
4

5. Trends in PerformanceIndicators:
Q2 2012-Q42013…
0
100,000
200,000
300,000
400,000
500,000
600,000
700,000
800,000
900,000
Q2 2012 Q3 2012 Q4 2012 Q1 2013 Q2 2013 Q3 2013 Q4 2013
Number
Curative
Consultation
FP users of
modern methods
Preg. women
counseled and
tested for HIV
5

6. Percentage Increase – Quantity Indicators
14%
-1%
23%
-49%
69%
44%
94%
70%
203%
14%
19% 21% 22% 22% 25%
63% 67%
88%
-100%
-50%
0%
50%
100%
150%
200%
250%
Curative
Consultation
HIV Preg.
women
given
Niverapine
& AZT
Pregnant
Women C&T
for HIV
ANC
prenatal and
follow up
visits
Skilled
Deliveries
Third dose
Fansider IPT
Postnatal
visit
Fully
Immunized
Children
FP users of
modern
methods
PercentageChange
Q2 2012 Vs Q4 2013 (Since Project Start) Q4 2012 Vs Q4 2013
6

7. Use of HMIS Data
7

8. Similaritiesand Differences:
OPVs HMIS Data
• OP and HMIS data are collected fromthe same health facilities, and
same data records
• However,OP data is compiled much quicker than HMIS data given the
requirementsfor verification,and linkages to payments
• OP data is only complied frompatient registered as compared to
HMIS which is complied from registersand tally sheets
• Consolidation of OP and HMIS data is doneby differentpersonnelat
district level
• 100% of the OP data is verified on a monthly basis while the HMIS
mainly relies on self-reporteddata which is occasionally verified
• OP and HMIS data MUST show the same trendin indicators,despite
differencesin magnitude
8

9. Using HMIS to compare across the 3 study
arms of the Impact Evaluation
 What can we say before endline?
 Diff-in-diff analysis of trends before and after RBF
 Analysis period – Jan 2011 to Dec 2012: RBF
introduced April 2012
 Not definitive analysis since HMIS is self-reported but:
 Important check on RBF:Are results from the operational
data consistent with HMIS?
 What is the impact of RBF on non-incentivized indicators?
9

10. Impact Evaluation
 Explores whether there is a causal link between the RBF
project and the results
 Baseline – Quantitative andQualitative
 Process Evaluation (interviews,Observations, operational
and HMIS data review)
 End line – Quantitative andQualitative
 Three (3) study arms:
 10 RBF Intervention Districts (RBF)
 10 Input-Based Financing Districts (C1)
 10 PureControl Districts (C2) 10

11. Impact on incentivized indicators
 All measures: per service per facility per month
 Gains in several targeted services, no change in total utilization, and declines in
immunization
 No gains from additional financing to districts
RBF vs. Additional financing
Coef 0.904 0.815 0.696 12.944 2.220 -9.316 -2.783
p-value 0.045 0.231 0.229 0.002 0.019 0.735 0.024
RBF vs. Control
Coef 1.174 1.954 1.586 7.850 2.243 -39.929 -2.761
p-value 0.005 0.002 0.011 0.055 0.031 0.158 0.011
Attendance
outpatient
total (calc)
Immunised
fully <1 year
new
Antenatal
1st visit
before 20
weeks
IPT 3rd
dose to
pregnant
woman
Postnatal
care within
6 days
Attendance
Family
Planning
total (Calc)
Delivery by
skilled
personnel
Additional financing vs. Control
Coef 0.035 1.117 0.882 -5.022 0.027 -31.420 0.035
p-value 0.979 0.107 0.114 0.194 0.976 0.247 0.979
11

12. Trend in Skilled delivery
(Jan 2011-Dec 2012)
0
200
400
600
800
1000
1200
1400
1600
1800
2000
Input Based
Pure Control
DataSource:HMIS, MoH
RBF implemented
12
RBF

13. Impact on non-incentivized indicators
 Little spill-over to the non-incentivized
 Additional financinghired more staff (but no changein service
measures)
RBF vs. Additional financing
Coef -0.945 3.046 0.136 -0.326 -3.578 -1.018
p-value 0.780 0.090 0.203 0.027 0.563 0.773
RBF vs. Control
Coef -3.847 2.147 0.046 0.181 8.057 0.298
p-value 0.306 0.191 0.799 0.289 0.032 0.759
Additional financing vs. Control
Coef -3.302 -0.840 -0.084 0.511 10.542 1.359
p-value 0.327 0.679 0.588 0.001 0.138 0.748
Supportive
supervision
visits this
month
Vitamin A
supplement
to 6-11
months
New TB
patient total
(calc)
TB patient
completed
treatment
Support
staff newly
recruited
Nurse
midwife
workdays
on duty
13

14. Use of OP and HMIS data has made it
possible to:
 TriangulateOP data with HMIS data i.e. Check on
consistency of OP data with HMIS
 Independently verify the OP data
 Monitor trends in incentivizedand non-incentivized
indicators across the three (3) researcharms
 Monitor the utilization of funds by indicators,and the
overall amount allocatedto the RBF Project
 Make adjustments to the RBF design, as well as to provide
more capacity building and technical support
14

15. Examples of how the Emerging
Information has been used
 Application of the quality tool changed to reward for quality
improvementsinstead of penalizing for quality deficits
 Set investmentcomponent at a minimum of 40%, and staff
performance incentivesat a maximum of 60%
 Increase assessment fees for hospitals doing quality audits
 Revise TA package to draw on local capacities
 Enhanced technical support to underperforming health
facilities
 Introductionof supervision fees for Provincial RBF Steering
Committees
15

16. Challenges
• Late transmission of HMIS data
• Poor quality of HMIS data due to migration to a web-
based DHIS-2
• Inadequate data entry clerks at health facilities
particularly in the control districts
• Costly to conduct a process evaluation involving
observations at health facilities, and interviews with
service providers, patients, and community members
16

17. Thank You
17