2. Purpose of Presentation
Rationale for the community indicator project
Donor organization consultations
On-line technical forum series
Field tests in Vietnam and Kenya
Description of recommended indicators
3. Rationale
Reduce reporting burdens for community
based organizations
Track prevention, care, testing, and linkages
at the community level
4. Aim
Harmonize PEPFAR and Global Fund
community program indicators
Improve the technical merit of community
indicators
Highlight a few of the emerging indicators that
would fill gaps in knowledge about community
programs
5. Extensive International Consultation
2009 meeting Community Based Information Systems
MEASURE Evaluation in D.C.
2010 meeting Community Program Indicators
MEASURE Evaluation in D.C.
2010 meeting Community Systems Strengthening
Global Fund in Geneva
2010 internal review of Community Program Indicators
all PEPFAR TWGs in D.C.
6. Community of Practice
“The overall goal of the Indicators for Community HIV
Programs CoP is to gather expert opinion on the best
metrics for community HIV program performance and to
gain consensus on minimum standards for monitoring
community HIV services and health outcomes.”
http://knowledge-gateway.org/registration/PreviewInvitation.ashx?c=24643e57-e93e-
49df-ae1c-f798ea662a16&i=61fe624d-0aaa-49ab-abeb-83d0ef89e10a
7. Indicator Selection Criteria
Represent services that address the continuum of
prevention, treatment, care, and support
Fully defined and currently used
Collected/used at the community level and
aggregated/used at national level
Collected by community-based information system
(non-clinic based)
Not include indicators from national level surveys or
special studies
8. CoP Recommendations Oct 2011
Outreach
encounters
Completed DEBIs
PwP
Condom stock-outs
Needle/Syringe
stock out
Received HTC
Completed referral
One care service
Undernourished
PLHIV
Children assessed
for individual needs
9. Field Tests in Vietnam and Kenya
To assess the need and utility of the
recommended indicators for both CBOs and
national authorities.
To determine the feasibility of collecting and
analyzing data for the recommended indicators.
To assess whether or not the draft
recommended indicators are fully defined
10. Mixed Methods Approach
Individual and small group consultations
In the style of semi-structured interviews
Review program records, documents and data
management systems
In the style of Data Quality Assessments
Request copies of relevant documents and forms
11. Field Test Participants
Kenya
7 community organizations
35 interviews
Vietnam
7 community organizations
20 interviews
12. Field Test Recommendations (I)
“Minimum Package of Services” should be
illustrative at the international level
Focus on developing national guidelines for
community services is important
Feasibility of data collection per local context
Validity, i.e. more closely represents the service
Reliability, i.e. less open to interpretation from one
site to another
13. Field Test Recommendations (II)
Proposed prevention indicators are feasible
and useful
encounters, standardized prevention interventions,
stock outs, community PwP services
One care service important and useful; there
is overlap with PwP
Completed referrals to a specific service is
possible, but ambitious
HIV testing, ART defaulters
14. Field Test Recommendations (III)
Consider counting households receiving care
and support
Right now a mix of individuals and head of
household under one care service
The option to count individuals and/or households
HTC and PWP indicators are from clinic-
based information systems
Disaggregate by type of site would very useful to
local and national level
15. Field Test Recommendations (IV)
Uncommon community services; indicators
not required at this time
Nutrition assessments for PLHIV
Individual needs assessments for children or
adults
Supporting or tracking all/various completed
referrals
16. Field Test Recommendations (V)
Paper-based systems still relevant
Possible to track # individuals in a reporting
period
Managing longitudinal records is not possible,
therefore difficult to track:
needs assessments
completed referrals
individual level outcomes
17. Field Test Recommendations (VI)
Community services found during field test
not represented in indicator list
HIV prevention with people who injected drugs
Finding ART defaulters in the community
ART adherence support
18. Final Indicators for Prevention Services
Number of HIV prevention outreach encounters
conducted during the reporting period
Number of people who completed a standardized
HIV prevention intervention during the reporting
period
Number of people living with HIV/AIDS (PLHIV)
reached with a minimum package of community-
based or home-based interventions for
Prevention with Positives (PwP) during the
reporting period
19. Final Indicators for Prevention Materials
Number and proportion of days during the
reporting period that stock outs of condoms
occurred at prevention sites
Number and proportion of days during the
reporting period that stock outs of needles and/or
syringes occurred at prevention sites
20. Final Indicator for Care Services
Number of adults and children infected and/or
affected by HIV/AIDS who received a minimum
of one community-based or home-based care
and support service during the reporting period
PLHIV
OVC
Caretaker of OVC(s)
Head of household
All members of household
21. Final Indicator for Testing and Linkages
Number and proportion of clients referred to HIV
testing and counseling services by a community-
based program and received their test results in
the reporting period
Notes de l'éditeur
Reduce reporting burdens for community based organizationsTrack prevention, care, testing, and linkages at the community level
Harmonize PEPFAR and Global Fund community program indicatorsImprove the technical merit of community indicatorsHighlight a few of the emerging indicators that would fill gaps in knowledge about community programs
Ground work for the standardization of indicators to measure community HIV program performance began in 2009 with the international consultation on Community Based Information Systems hosted by MEASURE Evaluation Project. Among the recommendations was a request to develop a core set of indicators across donors to reduce reporting burdens. In the fall 2009, an inventory of all published community indicators (totaling 123) was developed. The indicator list included input, output and outcome indicators using both survey and program data. The list also included indicators that collected information on clinical services that relied on community support. In January 2010, an international consultation on community indicators hosted by MEASURE Evaluation Project in Washington D.C. recommended focusing on the standardization of indicators that track service delivery, referrals for services, quality of services, and health outcomes with data collected at the community level only. The previously mentioned criteria were developed during this meeting and later validated by our community of practice members. A follow up consultation in Geneva provided the Global Fund recommendation to include indicators that measure community-led activities to decrease stigma, decrease gender based violence, or support an enabling environment. Finally in the summer of 2010, PEPFAR completed an internal review, which included all headquarter TWGs. TWGs reviewed the indicator inventory,previous consultation recommendations, and gave support for the development of an on-line forum to further gather expert opinion around indicator standardization.
In 2011, a 12 week on-line forum series designed by MEASURE Evaluation Project was held from April to June 2011. Technical experts from the field and from multilateral headquarters joined a community of practice (CoP) on Knowledge Gateway in order to participate. The community of practice remains a dissemination and feedback tool, with 165 members representing 26 countries. Lead experts were recruited to facilitate a 2-week long forum on a special topic designed to forward the goals of The Community. There were 5 topics:outcomes for OVC, service provision for HIV prevention, monitoring community participation in HTC, tracking referrals and linkages, and standardizing existing care indicators. MEASURE staff facilitated a formal launch for one week and then a wrap-up of the forum series for another week. The wrap up aimed to gain consensus on the overall conclusions of the forum series and on a final set of indicators.
This criteria was proposed during the January 2010 meeting in D.C. and was validated by community of practice participants using an online poll.Indicators represent services that address the continuum of prevention, treatment, care, and supportIndicators are fully defined and currently used Indicators are collected (and used) at the community level and aggregated (and used) at the national levelIndicators require data collected from a community-based information system and not collected from a clinic-based information systemIndicators do not require data from surveys or special studies administered by the national authorities
10 indicators using a poll during the “wrap-up” forum.Each indicator is based on existing foundational indicatorsMembers were not successful in addressing the metrics for community-led activities to decrease stigma, decrease gender based violence, or support an enabling environment. This remains a gap.Prevention programs across the board struggle to monitor one to one and small group contacts and generally do not report on the number of unique individuals. The lack of evidence-based preventions programs implemented on a global level leaves managers unable to fully define the intervention to be counted. The only exceptions cited during an on-line forum were multi-session youth education programs and outreach worker facilitated referrals to HIV testing. Generally, the indicators currently required by Global Fund and PEPFAR are not feasible to collect and lack technical merit. Members suggested revising these indicators to better reflect information HIV prevention programs are currently able to collect, with the aim to improve the quality of the measures overtime. We think the prevention with positives (PwP) indicator introduced by PEPFAR requiring a minimum package of services is important to include in this set. As written, the indicator can be collected from clinical and community programs. The field test will take a closer look at how community programs report against this indicator.Monitoring condom and clean needle distribution and/or promotion is important, but the current practice to report the number of materials distributed is not sufficient. We propose tracking stock outs at the prevention site level. Generally HIV care, support, and nutrition programs can successfully track the total number of unique individuals receiving at least one care service (case loads) in a reporting period. During the field test, we are most interested in disaggregation such as type of services, community services versus clinical services at the national level, etc.Individual assessments for adults and children are necessary to ensure that appropriate care, support, and nutrition services are provided to each person. USAID advisor Amie Heap has spear headed a “Nutrition and HIV Indicator Set” that has been submitted to the UNAIDS Registry this year. We have borrowed from the Nutrition and HIV Indicator Set by adapting the nutrition assessment indicator to be a general care and support assessment indicator for children. We are also using the proportion of undernourished people living with HIV (PLHIV) is an essential community indicator from that set. Again, as written, the indicator can be collected from clinical and community programs. The field test will take a closer look at how community programs report against this indicator or if they report it at all.HIV testing programs are successfully tracking the total number of people receiving HIV test results in a reporting period. However, a theme across all areas, including HIV testing and counseling, was the need to understand who is receiving what service from whom. We would like to now what type of support Civil Society Organizations and National AIDS Program managers require to use the indicator data disaggregated by Risk/Need, Age, Sex, Type of Service, Location, and Type of Provider.Another way to improve comprehensive prevention, care and support is to prioritize improving linkages and referrals for clients. Community program participation in national referral guidelines and/or the development referral guidelines of their own is important. We know that referrals and linkages are weak and that an indicator measuring counter-referrals or completed referrals would be a good start in monitoring linkages.
Purpose: To engage key stakeholders in harmonizing a core set of community-based HIV program indicators that can be collected (and used) at the community level and be aggregated (and used) at the national level.The field test objectives are based on the 2010 UNAIDS Indicator Standards: Operational Guidelines for Selecting Indicators for the HIV ResponseThe field test was comprised of three phases: community HIV intervention mapping and site selection, CBO interviews, NAP interviews.Note that the field test is not a review or evaluation of current indicators or monitoring and evaluation systems. Rather, the field test conducted in Vietnam and Kenya was to generate lessons learned that informed the revision and finalization of the set of recommended indicators for community HIV programs. As a set, the CBOs should provide information for all nine indicators, but each CBO is not required to provide services and collect information for all nine indicators. The set of CBOs should include organizations with small staff and large staff. All CBOs should have well developed service delivery and strong to fair data collection systems. The set of CBOs does not have to be geographically representative of the country in which the field test occurs.
Consultations will be conducted with CBO directors, program managers, M&E directors, and database managers. Consultations will also be requested with National AIDS program directors and managers, M&E directors and database managers. Group consultations will be requested after individual interviews have been completed to help “connect the dots.”The consultations requested the following from the participant:An overview of their community program Their opinion on the usefulness of current indicatorsA description of their data collection processesThen we would introduce the draft indicator and ask forFeedback on the draft indicator’s usefulness and clarity of definitionAll consultations used a semi-structured interview guide that was founded on data quality assurance tools.
Kenya12 organizations in site selection phase6 local, 1 ING, and 2 natlgovt organizations in interview phase35interviews including the community and national levelsNairobi, nearby Central Province was included so that urban/rural variation,Coast Province for EBIsVietnam10 local organizations in site selection phase7 organizations in interview phase20 interviews including community and national levelHanoi area
Given the results from the field tests conducted in Vietnam and Kenya, a “package of services” outlined in any of the community program indicators should be illustrative not required at the international level. A focus on establishing national guidelines or a national package of services would improve the feasibility of every indicator, facilitating less reporting burden for the community level and improving validity and reliability across community sites. Of note, clear and specific guidance on care and support packages versus PwP packages is an important first step to better understanding the types of services PLHIV receive and from whom. It should also be noted that any information collected for community program indicators would require a paper-based system. This is relevant for both Vietnam and Kenya. Any data management systems adapted or developed for these indicators should reflect that need. While personnel have experience collecting and reporting on a wide array of indicators similar to those recommended, there are still issues regarding limited capacity of personnel to manage electronically-based systems.Strong evidence in both countries points to the feasibility and usefulness at the community and national level for the prevention indicators and the stock out indicators. Although, many experts have called for better tracking of individuals receiving care and support services, the field test points to the continuation of tracking the “number of adults and children infected and/or affected by HIV/AIDS who received a minimum of one care and support service during the reporting period.” As mentioned earlier, great caution is recommended in order to avoid duplication when tracking the care and support indicator and PwP indicator. A gap, not fully explored during this field test, may be the number households receiving care and support. The number of households may better reflect care and support programs in countries with high prevalence rates.The indicators for HIV testing services, nutrition assessments, needs assessments for children, and completed referrals were not feasible and are therefore not included in the final set of indicators. One system issue that arose was the lack of CBO capacity to assess and track progress of unique individuals over long periods of time. Tracking the number of people who received a service within a designated reporting period is possible in both countries. Managing individual, long-term records at the CBO, which would include assessment results, is not feasible in either country. Therefore, nutrition and OVC individual assessment indicators were not included in the final set. A general “completed referral” indicator was also eliminated from the indicator set for similar reasons. CBOs have limited capacity to document and track individual cases over an extended time. Another system issue that arose is how to best represent the supportive role community programs play in people’s access to clinical services. Although community programs are essential to linking community members to HIV testing, including HIV tests provided by CBO personnel or on CBO grounds, the management and reporting responsibilities fall on clinical systems in both countries. Therefore, the “number of people tested” and the “number test kit stock outs” were not included in the indicator set. We strongly recommend that clinic-based information systems track the number of people tested at community-based sites in order to improve HIV testing strategies. In the event that the PwP indicator is reported via the clinic-based information system only, such as in Kenya, we strongly recommend that the number of people reached with PwP services by CBOs is tracked within that system and used for program management. Conversely, if a country does not require PwP program data to be tracked by a clinic-based information system only, we recommend collecting a community-based PwP indicator, which is included in the recommended set.In order to document the work CBOs contribute to HIV testing programs and to invest in client referrals and program linkages, we recommend adding an indicator used in Vietnam. “Number and proportion of clients referred to HIV testing services that were tested and received their results,” can be found on page 16 of this report and is included in the final recommended set.Given the programming and epidemic situations found in Vietnam and Kenya, there are gaps in the indicator set presented outlined below. In Vietnam for example, one of the current gaps is the need for indicators that address programs for IDUs within the proposed indicator set (please refer to Annex 6 for a list of IDU related indicators currently used in Vietnam). This is particularly important for countries with concentrated epidemics such as that found in Vietnam. It is also relevant to some extent for the diverse epidemic found in Kenya where pockets of IDUs in certain areas, such as the coastal region, mimic concentrated epidemics. An additional gap highlighted by the Kenya field test is the need for indicators that address ART adherence and the tracing of ART defaulters. Kenyan respondents felt that the contributions that communities make toward tracing of ART defaulters should be documented and monitored. An indicator for the proportion of defaulters traced would be useful for the purpose of assessing quality of treatment services. In addition, the inclusion of such an indicator would sensitize community partners to the importance of adherence and could bring up issues around why people default. At least one INGO is piloting a model in which a list of defaulters is generated by CCC staff; CHWs trace those individuals in the community and attempt to persuade them to return to the clinic. Introducing a defaulter tracing indicator could help to formalize and expand this process.The final recommended indicators for community HIV programs are listed below. Reference sheets for each indicator follow the list.
Completed referral to a service like HIV testing may be closer to a an estimate in the beginning. Coupon system is working but new and imperfect. Also consider peer worker logs.
Peer educator to small group participant ratioNumber people reached with mass mediaPwP could be written as a community versus a clinical package. Or it could be written as a completed referral to a clinical service. Or vice a versa.
Also count numbers distributed
PLHIVOVCCare taker of OVCHead of householdAll members in the householdDependent on the nature of the service or the participation requirements
Completed referrals to a specific service would be usefulHTCPwPARTNutritionDrug RehabilitationThis completed referral to HTC represents all of the work CBOs put into outreach to vulnerable people. CBOs would count this if the test happens at their sponsored event, on their property, with their community worker. They would also count this indicator if the test happened at a clinic. A coupon system or community worker log would be used.The number of people tested would be a separate indicator reported by the clinical site using the official testing form.