SlideShare une entreprise Scribd logo
1  sur  20
Design and implementation
of operational components
of Result-Based Financing
schemes
The case of the verification in Benin
Matthieu Antony
Agence Européenne pour le Développement et la Santé
European Agency for Development and Health
mantony@aedes.be
Dar es Salaam / 25.11.2015
Introduction
• Few evaluations have been published and many Randomized Controlled Trials
(RCTs) are currently ongoing to assess the effects of RBF
- Basinga and al, 2011; Falisse and al, 2014; Bonfrer and al, 2014; Huillery and al, 2015
• Quantitative methods are often silent on the paths and processes through which
results are achieved, and on wider health system effects
- Witter and al, 2013
• Our study aims at analyzing how implementation challenges can modify the
original design, affect RBF’s theory of change and therefore the scheme’s potential
for results
– We focus on the ‘verification of results’ in Benin
– We also draw some lessons on the design and implementation of key operational
components of RBF (such as verification), and on verification itself
Introduction
Role of verification in RBF’s Theory of Change
Verification
- Calculate rewards  pay bonus in a transparent, timely and regular manner
- Detect fraud & signal a threat of sanctions to providers
- Channel the “voice” of communities and patients, and improve providers’ accountability
- Improve governance & stewardship through DHMTs’ involvement
- Provide reliable data  data analysis  feedback and “coaching”
* Should be financially viable
Performance
(quantity + quality of outputs)
Payment
Context
The RBF project pilot
• Started in October 2011 with WB
funds – in 8 districts (“zones de
santé”)
• Population covered: 2,377,559
• Focus on health facilities’
productivity and quality of health
services
• On-going RCT
• Scale up to 21 districts with the
support of GF and GAVI (April
2015)
Context
Key roles in the scheme’s management
World Bank Funding
A Project Coordination Unit at MoH in charge of
signing contracts, purchasing services and
payment transfers.
An implementation agency in charge of technical
assistance, coaching and verification
Context
Design of the RBF scheme
• Contracts signed, and verification performed, with facilities in
both control and intervention arm (n = 188)
• Quantity indicators
– 8 indicators at community level
– 28 indicators at health centre level
– 14 indicators at hospital level
• Quality checklist
– 400 items
• “Results validation meeting” at central level on quarterly basis
Context
Design of the verification of results
Quantity verification
Monthly basis
Technical quality evaluation
Quarterly basis
Community verification
Quarterly basis
• Counting health services produced
from facilities’ registries
– performed by implem. agency
• Assessing quality against checklist
– carried out by DHMTs for health centres & by
peers for hospitals
– under the supervision of implem. agency
• Tracing patients in communities
• Patients’ satisfaction survey
– carried out by contracted Community Based
Organizations (CBOs)
– implem. agency in charge of sampling of
patients to track, CBOs supervision, reports’
validation
Research methods
Secondary data
Project documents
(reports, budgets) 2012-2015
TA activities daily timesheets
(n=20)
RBF specific data series
2012-2015 (n=188)
Focus Groups with CBOs (n=5)
Document review
Participant
observation
Findings
Results of verification
Indicators Timeframe Results
Difference between declared
and verified data on num. of
services provided
Duration of the project
Varies between 4% (new
users of family planning) and
51% (patient referred to
hospital)
Quality scores Third quarter of 2015 Between 4% and 96%
Num. of patients missing
from community tracing*
Third quarter of 2013 (first
community survey)
Between 12% and 47%
Patients’ satisfaction* - -
* Data collected, but not systematically analyzed
Findings
Implementation issues in verification processes
Planned Observed
Provide data, analyzed for feedback
and coaching to facilities
Little time for data analysis and coaching activities.
No analysis at all for community verif. data
Avoid fraud and provide a threat of
sanctions to providers
Sanctions for frauds and discrepancies in quantity
reported are rarely applied. Rewards for higher
patients’ satisfaction, or lower discrepancies in data
reported are also not implemented
Transparent, regular and timely
payment
Delays in the payment of bonus
Improve governance and stewardship
DHMTs rarely involved (they don’t have time, nor
resources and RBF verif. is not «motivating» enough)
Channel the «voice» of communities
and patients
- Elite appropriation of CBOs
- No analysis (rewards/sanctions) based on patients’
satisfaction survey
Financially viable Relatively costly, esp. community verification
Findings
Little time for data analysis, feedback and coaching
Proportion of time spent on different activities for imp. agency
staff in the field, based on timesheets of implem. agency staff
Quantitative
verification
46%
Community
verification
26%
Quality
evaluation
18%
Data
recording
10%
Verification
67%
Others
16%
Data analysis
and Meeting at
central level
9%
Training and
Coaching
8%
Findings
verification implementation issues
Planned Observed
Provide data, analyzed for feedback
and coaching to facilities
Little time for data analysis and coaching activities. No
analysis at all for community verif. data
Avoid fraud and provide a threat of
sanctions to providers
Sanctions for frauds and discrepancies in quantity
reported are rarely applied. Rewards for higher
patients’ satisfaction, or lower discrepancies in
data reported are also not implemented
Transparent, regular and timely
payment
Delays in the payment of bonus
Improve governance and stewardship
DHMTs rarely involved (they don’t have time, nor
resources and RBF verif. is not «motivating» enough)
Channel the «voice» of communities
and patients
- Elite appropriation of CBOs
- No analysis (rewards/sanctions) based on patients’
satisfaction survey
Financially viable Relatively costly, esp. community verification
Findings
Verification implementation issues
Planned Observed
Provide data, analyzed for feedback
and coaching to facilities
Little time for data analysis and coaching activities. No
analysis at all for community verif. data
Avoid fraud and provide a threat of
sanctions to providers
Sanctions for frauds and discrepancies in quantity
reported are rarely applied. Rewards for higher
patients’ satisfaction, or lower discrepancies in data
reported are also not implemented
Transparent, regular and timely
payment
Delays in the payment of bonus
Improve governance and stewardship
DHMTs rarely involved (they don’t have time, nor
resources and RBF verif. is not «motivating» enough)
Channel the «voice» of communities
and patients
- Elite appropriation of CBOs
- No analysis (rewards/sanctions) based on patients’
satisfaction survey
Financially viable Relatively costly, esp. community verification
Findings
Payment delays
Quarter Quarter End
Invoice
Transmission
Delay
due to verif.
Transfer
Delay
due to transfer
Total Delay
(month)
Q2 2012 Jun-12 Sep-12 3,5 Nov-12 1,5 5
Q3 2012 Sep-12 Dec-12 3,5 Apr-13 3 6,5
Q4 2012 Dec-12 Apr-13 4,0 Jul-13 3,5 7,5
Q1 2013 Mar-13 Jul-13 4,5 Nov-13 3,5 8
Q2 2013 Jun-13 Sep-13 3,5 Dec-13 3 6,5
Q3 2013 Sep-13 Dec-13 3,5 Feb-14 2 5,5
Q4 2013 Dec-13 Apr-14 4,0 May-14 0,5 4,5
Q1 2014 Mar-14 Jun-14 3,5 Jul-14 0 3,5
Q2 2014 Jun-14 Sep-14 3,5 Jan-15 3,5 7,0
Q3 2014 Sep-14 Dec-14 3,0 Feb-15 2 5
Findings
Verification implementation issues
Planned Observed
Provide data, analyzed for
feedback and coaching to facilities
Little time for data analysis and coaching activities. No
analysis at all for community verif. data
Avoid fraud and provide a threat of
sanctions to providers
Sanctions for frauds and discrepancies in quantity reported
are rarely applied. Rewards for higher patients’
satisfaction, or lower discrepancies in data reported are
also not implemented
Transparent, regular and timely
payment
Delays in the payment of bonus
Improve governance and
stewardship
DHMTs rarely involved (they don’t have time, nor
resources and RBF verif. is not «motivating» enough)
Channel the «voice» of
communities and patients
- Elite appropriation of CBOs
- No analysis (rewards/sanctions) based on patients’
satisfaction survey
Financially viable Relatively costly, esp. community verification
Findings
Verification implementation issues
Planned Observed
Provide data, analyzed for feedback
and coaching to facilities
Little time for data analysis and coaching activities. No
analysis at all for community verif. data
Avoid fraud and provide a threat of
sanctions to providers
Sanctions for frauds and discrepancies in quantity
reported are rarely applied. Rewards for higher
patients’ satisfaction, or lower discrepancies in data
reported are also not implemented
Transparent, regular and timely
payment
Delays in the payment of bonus
Improve governance and stewardship
DHMTs rarely involved (they don’t have time, nor
resources and RBF verif. is not «motivating» enough)
Channel the “voice” of communities
and patients
- Elite appropriation of CBOs
- No analysis of (no rewards/sanctions based on)
patients’ satisfaction survey
Financially viable Relatively costly, esp. community verification
Findings
Verification implementation issues
Planned Observed
Provide data, analyzed for feedback
and coaching to facilities
Little time for data analysis and coaching activities. No
analysis at all for community verif. data
Avoid fraud and provide a threat of
sanctions to providers
Sanctions for frauds and discrepancies in quantity
reported are rarely applied. Rewards for higher
patients’ satisfaction, or lower discrepancies in data
reported are also not implemented
Transparent, regular and timely
payment
Delays in the payment of bonus
Improve governance and stewardship
DHMTs rarely involved (they don’t have time, nor
resources and RBF verif. is not «motivating» enough)
Channel the «voice» of communities
and patients
- Elite appropriation of CBOs
- No analysis (rewards/sanctions) based on patients’
satisfaction survey
Financially viable Relatively costly, esp. community verification
Findings
Verification processes are costly
Funds to implem
agency for verification
activities (only)
Funds to CBOs
0.33
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Funds for RBF bonus to facilities Total funds for verification activities
1,595,644 USD
718,295 USD
0.12
Discussion & conclusions
• Methodologically
What are Impact Evaluation really testing, if the theory of change is
modified by implementation challenges?  it is essential to include an
analysis of the implementation processes in the RBF schemes’
evaluations
• Design
The design of the key elements of RBF schemes should be adapted to
the context and iteratively modified and improved during implementation
• Verification
Our analysis leads us to question whether the rational for three-pronged
(incl. community verif.) and thorough (i.e. non-random) verification is
valid under all conditions.
Acknowledgment
• Thanks to all the Technical Assistants of the Implementation Agency
who provided time, insight and expertise that greatly assisted the
research
• To Maria Paola Bertone and Olivier Barthes for assistance with
methodology, and all the AEDES team for comments that greatly
improved this presentation
• Many thanks also to Dr Akpamoli and his team and to Maud Juquois
and Ibrahim Magazi from the World Bank for their constant support
during the implementation of the project and with this research.

Contenu connexe

Plus de COP_HHA

Session 10 What have we learned so far ?
Session 10 What have we learned so far ? Session 10 What have we learned so far ?
Session 10 What have we learned so far ?
COP_HHA
 
Session 9 Expérience de renforcement du système de santé au Bénin
Session 9 Expérience de renforcement du système de santé au BéninSession 9 Expérience de renforcement du système de santé au Bénin
Session 9 Expérience de renforcement du système de santé au Bénin
COP_HHA
 
Session 8 Human Factors Predicting Failure and Success in Hospital Informatio...
Session 8 Human Factors Predicting Failure and Success in Hospital Informatio...Session 8 Human Factors Predicting Failure and Success in Hospital Informatio...
Session 8 Human Factors Predicting Failure and Success in Hospital Informatio...
COP_HHA
 
Session 7 a Vérification communautaire et satisfaction des patients
Session 7 a Vérification communautaire et satisfaction des patientsSession 7 a Vérification communautaire et satisfaction des patients
Session 7 a Vérification communautaire et satisfaction des patients
COP_HHA
 
Session 6 c Renforcer la participation communautaire à travers les médias soc...
Session 6 c Renforcer la participation communautaire à travers les médias soc...Session 6 c Renforcer la participation communautaire à travers les médias soc...
Session 6 c Renforcer la participation communautaire à travers les médias soc...
COP_HHA
 
Session 6 b Les TIC pour faire entrer la santé au sein des foyers : l'exemple...
Session 6 b Les TIC pour faire entrer la santé au sein des foyers : l'exemple...Session 6 b Les TIC pour faire entrer la santé au sein des foyers : l'exemple...
Session 6 b Les TIC pour faire entrer la santé au sein des foyers : l'exemple...
COP_HHA
 
Session 5 b Hospital Information Management with OpenClinic in Africa
Session 5 b Hospital Information Management with OpenClinic in AfricaSession 5 b Hospital Information Management with OpenClinic in Africa
Session 5 b Hospital Information Management with OpenClinic in Africa
COP_HHA
 
Session 5 a Les NTIC au service de la relation Soignant-Soigné en 1° ligne, e...
Session 5 a Les NTIC au service de la relation Soignant-Soigné en 1° ligne, e...Session 5 a Les NTIC au service de la relation Soignant-Soigné en 1° ligne, e...
Session 5 a Les NTIC au service de la relation Soignant-Soigné en 1° ligne, e...
COP_HHA
 
Session 5 Réforme du système d'information sanitaire en Côte d'Ivoire, quelle...
Session 5 Réforme du système d'information sanitaire en Côte d'Ivoire, quelle...Session 5 Réforme du système d'information sanitaire en Côte d'Ivoire, quelle...
Session 5 Réforme du système d'information sanitaire en Côte d'Ivoire, quelle...
COP_HHA
 
Session 5 Implementation and USE OF central Health information systems suppo...
Session 5 Implementation and USE OF central  Health information systems suppo...Session 5 Implementation and USE OF central  Health information systems suppo...
Session 5 Implementation and USE OF central Health information systems suppo...
COP_HHA
 
Session 4 c Cas du district de Ngoumou, Camaroun
Session 4 c Cas du district de Ngoumou, Camaroun Session 4 c Cas du district de Ngoumou, Camaroun
Session 4 c Cas du district de Ngoumou, Camaroun
COP_HHA
 
Session 4 b Défi de réussite de solutions TIC dans les systèmes de santé
Session 4 b Défi de réussite de solutions TIC dans les systèmes de santéSession 4 b Défi de réussite de solutions TIC dans les systèmes de santé
Session 4 b Défi de réussite de solutions TIC dans les systèmes de santé
COP_HHA
 
Session 4 a Situation du CMA Congo II Douala, Cameroun
Session 4 a Situation du CMA Congo II Douala, Cameroun Session 4 a Situation du CMA Congo II Douala, Cameroun
Session 4 a Situation du CMA Congo II Douala, Cameroun
COP_HHA
 

Plus de COP_HHA (14)

Session 10 What have we learned so far ?
Session 10 What have we learned so far ? Session 10 What have we learned so far ?
Session 10 What have we learned so far ?
 
Session 9 Expérience de renforcement du système de santé au Bénin
Session 9 Expérience de renforcement du système de santé au BéninSession 9 Expérience de renforcement du système de santé au Bénin
Session 9 Expérience de renforcement du système de santé au Bénin
 
Session 8 Human Factors Predicting Failure and Success in Hospital Informatio...
Session 8 Human Factors Predicting Failure and Success in Hospital Informatio...Session 8 Human Factors Predicting Failure and Success in Hospital Informatio...
Session 8 Human Factors Predicting Failure and Success in Hospital Informatio...
 
Session 7 a Vérification communautaire et satisfaction des patients
Session 7 a Vérification communautaire et satisfaction des patientsSession 7 a Vérification communautaire et satisfaction des patients
Session 7 a Vérification communautaire et satisfaction des patients
 
Session 6 c Renforcer la participation communautaire à travers les médias soc...
Session 6 c Renforcer la participation communautaire à travers les médias soc...Session 6 c Renforcer la participation communautaire à travers les médias soc...
Session 6 c Renforcer la participation communautaire à travers les médias soc...
 
Session 6 b Les TIC pour faire entrer la santé au sein des foyers : l'exemple...
Session 6 b Les TIC pour faire entrer la santé au sein des foyers : l'exemple...Session 6 b Les TIC pour faire entrer la santé au sein des foyers : l'exemple...
Session 6 b Les TIC pour faire entrer la santé au sein des foyers : l'exemple...
 
Session 6 a DHIS2 : Overview and Implementation in West Africa
Session 6 a DHIS2 : Overview and Implementation in West AfricaSession 6 a DHIS2 : Overview and Implementation in West Africa
Session 6 a DHIS2 : Overview and Implementation in West Africa
 
Session 5 b Hospital Information Management with OpenClinic in Africa
Session 5 b Hospital Information Management with OpenClinic in AfricaSession 5 b Hospital Information Management with OpenClinic in Africa
Session 5 b Hospital Information Management with OpenClinic in Africa
 
Session 5 a Les NTIC au service de la relation Soignant-Soigné en 1° ligne, e...
Session 5 a Les NTIC au service de la relation Soignant-Soigné en 1° ligne, e...Session 5 a Les NTIC au service de la relation Soignant-Soigné en 1° ligne, e...
Session 5 a Les NTIC au service de la relation Soignant-Soigné en 1° ligne, e...
 
Session 5 Réforme du système d'information sanitaire en Côte d'Ivoire, quelle...
Session 5 Réforme du système d'information sanitaire en Côte d'Ivoire, quelle...Session 5 Réforme du système d'information sanitaire en Côte d'Ivoire, quelle...
Session 5 Réforme du système d'information sanitaire en Côte d'Ivoire, quelle...
 
Session 5 Implementation and USE OF central Health information systems suppo...
Session 5 Implementation and USE OF central  Health information systems suppo...Session 5 Implementation and USE OF central  Health information systems suppo...
Session 5 Implementation and USE OF central Health information systems suppo...
 
Session 4 c Cas du district de Ngoumou, Camaroun
Session 4 c Cas du district de Ngoumou, Camaroun Session 4 c Cas du district de Ngoumou, Camaroun
Session 4 c Cas du district de Ngoumou, Camaroun
 
Session 4 b Défi de réussite de solutions TIC dans les systèmes de santé
Session 4 b Défi de réussite de solutions TIC dans les systèmes de santéSession 4 b Défi de réussite de solutions TIC dans les systèmes de santé
Session 4 b Défi de réussite de solutions TIC dans les systèmes de santé
 
Session 4 a Situation du CMA Congo II Douala, Cameroun
Session 4 a Situation du CMA Congo II Douala, Cameroun Session 4 a Situation du CMA Congo II Douala, Cameroun
Session 4 a Situation du CMA Congo II Douala, Cameroun
 

Dernier

Circulation through Special Regions -characteristics and regulation
Circulation through Special Regions -characteristics and regulationCirculation through Special Regions -characteristics and regulation
Circulation through Special Regions -characteristics and regulation
MedicoseAcademics
 

Dernier (20)

Is Rheumatoid Arthritis a Metabolic Disorder.pptx
Is Rheumatoid Arthritis a Metabolic Disorder.pptxIs Rheumatoid Arthritis a Metabolic Disorder.pptx
Is Rheumatoid Arthritis a Metabolic Disorder.pptx
 
NCLEX RN REVIEW EXAM CONTENT BLUE BOOK PDF
NCLEX RN REVIEW EXAM CONTENT BLUE BOOK PDFNCLEX RN REVIEW EXAM CONTENT BLUE BOOK PDF
NCLEX RN REVIEW EXAM CONTENT BLUE BOOK PDF
 
Introducing VarSeq Dx as a Medical Device in the European Union
Introducing VarSeq Dx as a Medical Device in the European UnionIntroducing VarSeq Dx as a Medical Device in the European Union
Introducing VarSeq Dx as a Medical Device in the European Union
 
Integrated Neuromuscular Inhibition Technique (INIT)
Integrated Neuromuscular Inhibition Technique (INIT)Integrated Neuromuscular Inhibition Technique (INIT)
Integrated Neuromuscular Inhibition Technique (INIT)
 
DEVELOPMENT OF OCCLUSION IN PEDIATRIC DENTISTRY
DEVELOPMENT OF OCCLUSION IN PEDIATRIC DENTISTRYDEVELOPMENT OF OCCLUSION IN PEDIATRIC DENTISTRY
DEVELOPMENT OF OCCLUSION IN PEDIATRIC DENTISTRY
 
Compare home pulse pressure components collected directly from home
Compare home pulse pressure components collected directly from homeCompare home pulse pressure components collected directly from home
Compare home pulse pressure components collected directly from home
 
End Feel -joint end feel - Normal and Abnormal end feel
End Feel -joint end feel - Normal and Abnormal end feelEnd Feel -joint end feel - Normal and Abnormal end feel
End Feel -joint end feel - Normal and Abnormal end feel
 
A thorough review of supernormal conduction.pptx
A thorough review of supernormal conduction.pptxA thorough review of supernormal conduction.pptx
A thorough review of supernormal conduction.pptx
 
TUBERCULINUM-2.BHMS.MATERIA MEDICA.HOMOEOPATHY
TUBERCULINUM-2.BHMS.MATERIA MEDICA.HOMOEOPATHYTUBERCULINUM-2.BHMS.MATERIA MEDICA.HOMOEOPATHY
TUBERCULINUM-2.BHMS.MATERIA MEDICA.HOMOEOPATHY
 
Multiple sclerosis diet.230524.ppt3.pptx
Multiple sclerosis diet.230524.ppt3.pptxMultiple sclerosis diet.230524.ppt3.pptx
Multiple sclerosis diet.230524.ppt3.pptx
 
Muscle Energy Technique (MET) with variant and techniques.
Muscle Energy Technique (MET) with variant and techniques.Muscle Energy Technique (MET) with variant and techniques.
Muscle Energy Technique (MET) with variant and techniques.
 
Anuman- An inference for helpful in diagnosis and treatment
Anuman- An inference for helpful in diagnosis and treatmentAnuman- An inference for helpful in diagnosis and treatment
Anuman- An inference for helpful in diagnosis and treatment
 
The Orbit & its contents by Dr. Rabia I. Gandapore.pptx
The Orbit & its contents by Dr. Rabia I. Gandapore.pptxThe Orbit & its contents by Dr. Rabia I. Gandapore.pptx
The Orbit & its contents by Dr. Rabia I. Gandapore.pptx
 
DIGITAL RADIOGRAPHY-SABBU KHATOON .pptx
DIGITAL RADIOGRAPHY-SABBU KHATOON  .pptxDIGITAL RADIOGRAPHY-SABBU KHATOON  .pptx
DIGITAL RADIOGRAPHY-SABBU KHATOON .pptx
 
hypo and hyper thyroidism final lecture.pptx
hypo and hyper thyroidism  final lecture.pptxhypo and hyper thyroidism  final lecture.pptx
hypo and hyper thyroidism final lecture.pptx
 
DECIPHERING COMMON ECG FINDINGS IN ED.pptx
DECIPHERING COMMON ECG FINDINGS IN ED.pptxDECIPHERING COMMON ECG FINDINGS IN ED.pptx
DECIPHERING COMMON ECG FINDINGS IN ED.pptx
 
Antiplatelets in IHD, Dose Duration, DAPT vs SAPT
Antiplatelets in IHD, Dose Duration, DAPT vs SAPTAntiplatelets in IHD, Dose Duration, DAPT vs SAPT
Antiplatelets in IHD, Dose Duration, DAPT vs SAPT
 
CNN-based plastic waste detection system
CNN-based plastic waste detection systemCNN-based plastic waste detection system
CNN-based plastic waste detection system
 
Why invest into infodemic management in health emergencies
Why invest into infodemic management in health emergenciesWhy invest into infodemic management in health emergencies
Why invest into infodemic management in health emergencies
 
Circulation through Special Regions -characteristics and regulation
Circulation through Special Regions -characteristics and regulationCirculation through Special Regions -characteristics and regulation
Circulation through Special Regions -characteristics and regulation
 

Why should we pay more attention to the operational components of RbF schemes ? A case study on the design and implementation of the "verification" in Benin

  • 1. Design and implementation of operational components of Result-Based Financing schemes The case of the verification in Benin Matthieu Antony Agence Européenne pour le Développement et la Santé European Agency for Development and Health mantony@aedes.be Dar es Salaam / 25.11.2015
  • 2. Introduction • Few evaluations have been published and many Randomized Controlled Trials (RCTs) are currently ongoing to assess the effects of RBF - Basinga and al, 2011; Falisse and al, 2014; Bonfrer and al, 2014; Huillery and al, 2015 • Quantitative methods are often silent on the paths and processes through which results are achieved, and on wider health system effects - Witter and al, 2013 • Our study aims at analyzing how implementation challenges can modify the original design, affect RBF’s theory of change and therefore the scheme’s potential for results – We focus on the ‘verification of results’ in Benin – We also draw some lessons on the design and implementation of key operational components of RBF (such as verification), and on verification itself
  • 3. Introduction Role of verification in RBF’s Theory of Change Verification - Calculate rewards  pay bonus in a transparent, timely and regular manner - Detect fraud & signal a threat of sanctions to providers - Channel the “voice” of communities and patients, and improve providers’ accountability - Improve governance & stewardship through DHMTs’ involvement - Provide reliable data  data analysis  feedback and “coaching” * Should be financially viable Performance (quantity + quality of outputs) Payment
  • 4. Context The RBF project pilot • Started in October 2011 with WB funds – in 8 districts (“zones de santé”) • Population covered: 2,377,559 • Focus on health facilities’ productivity and quality of health services • On-going RCT • Scale up to 21 districts with the support of GF and GAVI (April 2015)
  • 5. Context Key roles in the scheme’s management World Bank Funding A Project Coordination Unit at MoH in charge of signing contracts, purchasing services and payment transfers. An implementation agency in charge of technical assistance, coaching and verification
  • 6. Context Design of the RBF scheme • Contracts signed, and verification performed, with facilities in both control and intervention arm (n = 188) • Quantity indicators – 8 indicators at community level – 28 indicators at health centre level – 14 indicators at hospital level • Quality checklist – 400 items • “Results validation meeting” at central level on quarterly basis
  • 7. Context Design of the verification of results Quantity verification Monthly basis Technical quality evaluation Quarterly basis Community verification Quarterly basis • Counting health services produced from facilities’ registries – performed by implem. agency • Assessing quality against checklist – carried out by DHMTs for health centres & by peers for hospitals – under the supervision of implem. agency • Tracing patients in communities • Patients’ satisfaction survey – carried out by contracted Community Based Organizations (CBOs) – implem. agency in charge of sampling of patients to track, CBOs supervision, reports’ validation
  • 8. Research methods Secondary data Project documents (reports, budgets) 2012-2015 TA activities daily timesheets (n=20) RBF specific data series 2012-2015 (n=188) Focus Groups with CBOs (n=5) Document review Participant observation
  • 9. Findings Results of verification Indicators Timeframe Results Difference between declared and verified data on num. of services provided Duration of the project Varies between 4% (new users of family planning) and 51% (patient referred to hospital) Quality scores Third quarter of 2015 Between 4% and 96% Num. of patients missing from community tracing* Third quarter of 2013 (first community survey) Between 12% and 47% Patients’ satisfaction* - - * Data collected, but not systematically analyzed
  • 10. Findings Implementation issues in verification processes Planned Observed Provide data, analyzed for feedback and coaching to facilities Little time for data analysis and coaching activities. No analysis at all for community verif. data Avoid fraud and provide a threat of sanctions to providers Sanctions for frauds and discrepancies in quantity reported are rarely applied. Rewards for higher patients’ satisfaction, or lower discrepancies in data reported are also not implemented Transparent, regular and timely payment Delays in the payment of bonus Improve governance and stewardship DHMTs rarely involved (they don’t have time, nor resources and RBF verif. is not «motivating» enough) Channel the «voice» of communities and patients - Elite appropriation of CBOs - No analysis (rewards/sanctions) based on patients’ satisfaction survey Financially viable Relatively costly, esp. community verification
  • 11. Findings Little time for data analysis, feedback and coaching Proportion of time spent on different activities for imp. agency staff in the field, based on timesheets of implem. agency staff Quantitative verification 46% Community verification 26% Quality evaluation 18% Data recording 10% Verification 67% Others 16% Data analysis and Meeting at central level 9% Training and Coaching 8%
  • 12. Findings verification implementation issues Planned Observed Provide data, analyzed for feedback and coaching to facilities Little time for data analysis and coaching activities. No analysis at all for community verif. data Avoid fraud and provide a threat of sanctions to providers Sanctions for frauds and discrepancies in quantity reported are rarely applied. Rewards for higher patients’ satisfaction, or lower discrepancies in data reported are also not implemented Transparent, regular and timely payment Delays in the payment of bonus Improve governance and stewardship DHMTs rarely involved (they don’t have time, nor resources and RBF verif. is not «motivating» enough) Channel the «voice» of communities and patients - Elite appropriation of CBOs - No analysis (rewards/sanctions) based on patients’ satisfaction survey Financially viable Relatively costly, esp. community verification
  • 13. Findings Verification implementation issues Planned Observed Provide data, analyzed for feedback and coaching to facilities Little time for data analysis and coaching activities. No analysis at all for community verif. data Avoid fraud and provide a threat of sanctions to providers Sanctions for frauds and discrepancies in quantity reported are rarely applied. Rewards for higher patients’ satisfaction, or lower discrepancies in data reported are also not implemented Transparent, regular and timely payment Delays in the payment of bonus Improve governance and stewardship DHMTs rarely involved (they don’t have time, nor resources and RBF verif. is not «motivating» enough) Channel the «voice» of communities and patients - Elite appropriation of CBOs - No analysis (rewards/sanctions) based on patients’ satisfaction survey Financially viable Relatively costly, esp. community verification
  • 14. Findings Payment delays Quarter Quarter End Invoice Transmission Delay due to verif. Transfer Delay due to transfer Total Delay (month) Q2 2012 Jun-12 Sep-12 3,5 Nov-12 1,5 5 Q3 2012 Sep-12 Dec-12 3,5 Apr-13 3 6,5 Q4 2012 Dec-12 Apr-13 4,0 Jul-13 3,5 7,5 Q1 2013 Mar-13 Jul-13 4,5 Nov-13 3,5 8 Q2 2013 Jun-13 Sep-13 3,5 Dec-13 3 6,5 Q3 2013 Sep-13 Dec-13 3,5 Feb-14 2 5,5 Q4 2013 Dec-13 Apr-14 4,0 May-14 0,5 4,5 Q1 2014 Mar-14 Jun-14 3,5 Jul-14 0 3,5 Q2 2014 Jun-14 Sep-14 3,5 Jan-15 3,5 7,0 Q3 2014 Sep-14 Dec-14 3,0 Feb-15 2 5
  • 15. Findings Verification implementation issues Planned Observed Provide data, analyzed for feedback and coaching to facilities Little time for data analysis and coaching activities. No analysis at all for community verif. data Avoid fraud and provide a threat of sanctions to providers Sanctions for frauds and discrepancies in quantity reported are rarely applied. Rewards for higher patients’ satisfaction, or lower discrepancies in data reported are also not implemented Transparent, regular and timely payment Delays in the payment of bonus Improve governance and stewardship DHMTs rarely involved (they don’t have time, nor resources and RBF verif. is not «motivating» enough) Channel the «voice» of communities and patients - Elite appropriation of CBOs - No analysis (rewards/sanctions) based on patients’ satisfaction survey Financially viable Relatively costly, esp. community verification
  • 16. Findings Verification implementation issues Planned Observed Provide data, analyzed for feedback and coaching to facilities Little time for data analysis and coaching activities. No analysis at all for community verif. data Avoid fraud and provide a threat of sanctions to providers Sanctions for frauds and discrepancies in quantity reported are rarely applied. Rewards for higher patients’ satisfaction, or lower discrepancies in data reported are also not implemented Transparent, regular and timely payment Delays in the payment of bonus Improve governance and stewardship DHMTs rarely involved (they don’t have time, nor resources and RBF verif. is not «motivating» enough) Channel the “voice” of communities and patients - Elite appropriation of CBOs - No analysis of (no rewards/sanctions based on) patients’ satisfaction survey Financially viable Relatively costly, esp. community verification
  • 17. Findings Verification implementation issues Planned Observed Provide data, analyzed for feedback and coaching to facilities Little time for data analysis and coaching activities. No analysis at all for community verif. data Avoid fraud and provide a threat of sanctions to providers Sanctions for frauds and discrepancies in quantity reported are rarely applied. Rewards for higher patients’ satisfaction, or lower discrepancies in data reported are also not implemented Transparent, regular and timely payment Delays in the payment of bonus Improve governance and stewardship DHMTs rarely involved (they don’t have time, nor resources and RBF verif. is not «motivating» enough) Channel the «voice» of communities and patients - Elite appropriation of CBOs - No analysis (rewards/sanctions) based on patients’ satisfaction survey Financially viable Relatively costly, esp. community verification
  • 18. Findings Verification processes are costly Funds to implem agency for verification activities (only) Funds to CBOs 0.33 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Funds for RBF bonus to facilities Total funds for verification activities 1,595,644 USD 718,295 USD 0.12
  • 19. Discussion & conclusions • Methodologically What are Impact Evaluation really testing, if the theory of change is modified by implementation challenges?  it is essential to include an analysis of the implementation processes in the RBF schemes’ evaluations • Design The design of the key elements of RBF schemes should be adapted to the context and iteratively modified and improved during implementation • Verification Our analysis leads us to question whether the rational for three-pronged (incl. community verif.) and thorough (i.e. non-random) verification is valid under all conditions.
  • 20. Acknowledgment • Thanks to all the Technical Assistants of the Implementation Agency who provided time, insight and expertise that greatly assisted the research • To Maria Paola Bertone and Olivier Barthes for assistance with methodology, and all the AEDES team for comments that greatly improved this presentation • Many thanks also to Dr Akpamoli and his team and to Maud Juquois and Ibrahim Magazi from the World Bank for their constant support during the implementation of the project and with this research.

Notes de l'éditeur

  1. Put it very simply, RBF schemes aim at improving health coverage and quality by linking payments to providers to desired outputs. This linkage is done by establishing contracts, clarifying roles and tasks, and defining rewards and sanctions for providers. It is envisaged that this would generate behavioural changes at individual and organizational level which would improve health outcomes. In this theory of change, verification plays a key role. First of all, it allows to calculate the rewards for the providers and therefore pay them a bonus in a transparent manner (which enhances their trust) and to pay them promptly and regularly based on their effort/performance. Etc etc etc etc (point on data). Finally, the verification process provides reliable data, which can be analyzed to provide feedback to providers and “coaching”. This is another key element of RBF as with increase autonomy, providers need more data and (initially) support for decision-making. (point of financial viability): importantly, verification processes should be financially viable, so that the benefits of them outweigh the costs
  2. Implementation agency, led by AEDES and with which I have been involved since 2012, is in charge of running the project on a daily base, including providing technical assistance, coaching (and data analysis) and verification.
  3. Three verification axes Consists in …. Carried out by …. Role of implementation agency
  4. As part of the implementation agency, we were a participant observer to the processes of implementation of the scheme since its beginning in 2011. For this study we also reviewed the existing literature and used documents relating to the project, such as manuals, reports, budgets. We also used secondary data on RBF, such as data on service outputs and quality (Open RBF), time sheets from implenting agency’s TA and FGD with CBOs.
  5. (in summary/briefly), the results of the verification of the RBF scheme are the following: ……………….. However, in this presentation we also look at the implementation processes of the verification, to explore how the actual implementation differs from the original design.
  6. Stress that there is little time for coaching and data analysis
  7. 2 – The design of the verification function (and of other key elements of RBF schemes) should be adapted to the context. We think that toolkits and the CoPs, may lead to a «standardization» of approaches across contexts even when certain RBF elements may not be relevant (in the case of Benin, we can wonder what is the role of the community verification?). Also, verification and other key elements should be constantly modified and improved during implementation – for example, in Benin verification processes are now being updated based on these findings.