This document summarizes an upcoming LQAS session that will discuss common mistakes in LQAS and ways to avoid them. The session will include an overview of LQAS, experiences from organizations that use it, and small group work. While LQAS can be useful for monitoring, it is controversial and best not used for evaluation. Common mistakes include incorrectly interpreting findings, improper descriptions of actions taken, and not collecting adequate information. The American Statistical Association recommends carefully stating LQAS conclusions to avoid mistakenly drawing conclusions about a supervision area's performance.

1. LQAS: Pitfalls, Controversies, and Addressing Concerns CORE Spring Meeting Jennifer Luna; Todd Nitkin; Bill Yaggy May 10, 2011

2. LQAS is useful However, common mistakes can lead to problems including: Not collecting adequate information Incorrectly interpreting findings Improperly describing actions taken as a result of findings

3. LQAS session Purpose: Discuss common mistakes and practical ways to avoid them Agenda: Overview of LQAS, resources and discussion of current controversies Experience, advice from AMREF Practical advice from MTI Small group work Brief plenary on results of small group work

4. LQAS Origin in quality control for industry light bulbs Promoted for use in small population based health surveys Useful for monitoring because it provides information used to identify sub-divisions (supervision areas) of project area that are sub-standard More attention can be given to these areas. Does not provide coverage levels for supervision areas

5. Steps Divide project area into supervision areas (SAs) Decide on level below which a supervision area is identified as priority for special attention; e.g. <85% Randomly select sample of 19 per SA Use a decision rule table to determine what number of correct responses is the cut-off for identifying SAs that need priority attention <20%, <1 correct response out of 19 <50%, <7 correct responses out of 19 85%, <14 correct responses out of 19

6. Steps Indicators must be dichotomous (yes/no) Was child exclusively breastfed? Not for how many months was the child exclusively breastfed?

7. Supervision Area (SA) “Lots” that you divide project area into Should have meaning: i.e. health facility catchment areas, sub-districts or other MOH administrative areas A B E D C

9. Parallel Sampling Conducting multiple surveys at the same time using the same logistic system Necessary for surveys that collect information on indicators with different age groups as denominators: i.e. IYCF (denominator children 6-23 months); use of ORS for diarrhea treatment (denominator children 0-23 months with diarrhea in the last two weeks) Parallel sampling ensures that: Each indicator will have 19 responses/supervision area

10. LQAS – Coverage estimates Information from supervision areas combined to obtain coverage for entire project area LQAS with information combined from multiple SAs is a form of stratified sampling where supervision areas are strata Requires a weighted average of all supervision areas; confidence intervals Total of 95 responses (5x19 = 95) Remember parallel sampling

11. LQAS Useful for monitoring of implementation Can be controversial Best not to use for evaluation

12. Controversy - Example Concern raised that use of LQAS to identify SAs as having reached the target, might result in mistakenly identifying an SA as one in for which resources can be reduced. If this is a mistaken classification, then the population will not receive needed resources

13. American Statistical Association (ASA) Recommendation Report to the American Statistical Association Board of Directors on Lot Quality Assurance Sampling, August 19, 2010 “We believe that LQAS conclusions should be carefully stated to ensure that the user will never mistakenly draw a conclusion about the performance of the SA. We propose the following as a better interpretation of the conclusions that can be drawn from LQAS: Based on considerations related to the power of the test employed, given the sample size used:

14. ASA recommendation continued If 13 or more of the 19 samples of men in a supervision area (SA) can correctly name the ways to prevent sexual transmission of HIV, then we classify the SA as not requiring priority intervention at this time or If 12 or fewer of the 19 sampled men in the SA can correctly name the ways to prevent sexual transmission of HIV, then we classify the SA as substandardrequiring immediate intervention.

15. Articles Rhoda, Dale A., Soledad A. Fernandez, David J. Fitch, and Stanley Lemeshow (2010), LQAS: User Beware. International Journal of Epidemiology, 39:60-68 Olives, C., M. Pagano, and J.J. Valadez (2010), Commentary: Understanding practical lot quality assurance sampling. International Journal of Epidemiology, 39: 69-71.

16. Resources: MCHIP NGO/PVO support website: www.mchipngo.net Rapid Health Surveys Handbook (Public Health Institute) CORE website: (www.coregroup.org) KPC Trainer of Survey Trainers (TOAST) LQAS trainers guides LQAS protocol for parallel sampling LQAS FAQs