Paula Lorgelly, ISPOR European Congress, Vienna 2016

1. Panelist:
Paula Lorgelly
Deputy Director, OHE
ISPOR European Congress, Vienna, 2016
Anchoring Vignettes:
identifying response bias and DIF in self
assessed health

2. Anchoring Vignettes:
identifying response bias and DIF in self assessed health
Background
• Individual and household surveys often rely on self-assessed
measures of health
• In general, would you say your health is: excellent, very good,
good, fair or poor?
• Analyses using measures of self-assessed health (SAH) rely on
the measure being an accurate reflection of the true health of the
groups or individuals concerned
• But responses to questions on subjective scales will be inaccurate
if groups of individuals systematically differ in their use and/or
interpretation of the response categories
• Systematic variation in the use of response categories is known
as reporting heterogeneity or response scale heterogeneity or
differential item functioning (DIF)
2

3. Anchoring Vignettes:
identifying response bias and DIF in self assessed health
DIF and the EQ-5D
• The EQ-5D is the most commonly used instrument for measuring
preference-based health-related quality of life (HRQoL)
• Commonly used in economic evaluations, but increasingly
collected via routine data collection in health care systems
(PROMs programmes in England, Sweden and Canada) and
included in health surveys as a measure of population health
status
• When used to measure and compare health profiles or utilities
across sub-groups of the population, the results will be
misleading if groups systematically differ in use of response
categories
• Could the EQ-5D suffer from DIF like other SRH measures?

4. Anchoring Vignettes:
identifying response bias and DIF in self assessed health
4

5. Anchoring Vignettes:
identifying response bias and DIF in self assessed health
Differential Item Functioning
τ4
τ3
τ2
τ1
No problems
Slight problems
Moderate problems
Severe problems
Unable to walk
Group 2
Underlyinglatenthealthscaleformobility
τ4
τ3
τ2
τ1
No problems
Slight problems
Moderate problems
Severe problems
Unable to walk
Group 1High mobility
Low mobility
Group 2’s
mean health
Group 1’s
mean health

6. Anchoring Vignettes:
identifying response bias and DIF in self assessed health
Anchoring vignettes
• In order to obtain any meaningful comparison between the health
of groups 1 and 2 it is essential to adjust for DIF
• Anchoring vignettes (King et al. 2004) can be used to adjust for
DIF
• Previously been used to address DIF in political efficacy,
job/income/life satisfaction, general/specific health measures
• Vignette - a brief health description of a hypothetical individual
• Respondents are asked to rate the health state described by the
vignette using the same ordered categories they use to rate their
own health
• Since the actual level of health of the people in the vignettes is
the same for all respondents, the variation in ratings can be used
to identify and correct for DIF

7. Anchoring Vignettes:
identifying response bias and DIF in self assessed health
Anchoring vignettes
• Example of a vignette for the mobility domain:
Belinda walks for one or two kilometres and climbs three flights
of stairs every day without tiring.
Select the one option that best describes Belinda’s mobility:
 She has no problems with walking around
 She has slight problems with walking around
 She has moderate problems with walking around
 She has severe problems with walking around
 She is unable to walk around

8. Anchoring Vignettes:
identifying response bias and DIF in self assessed health
Anchoring vignettes
• Typically, a series of vignettes are presented for each health
construct of interest, at varying levels of severity
• Suppose we give groups 1 and 2 two vignettes to rate, of
differing severity:
• Vignette 1 – limited problems in walking around
• Vignette 2 – more problems in walking around

9. Anchoring Vignettes:
identifying response bias and DIF in self assessed health
Anchoring vignettes
τ4
τ3
τ2
τ1
No problems
Slight problems
Moderate problems
Severe problems
Unable to walk
Group 2
Underlyinglatenthealthscaleformobility
Vignette 2
Vignette 1
High mobility
Low mobility
τ4
τ3
τ2
τ1
No problems
Slight problems
Moderate problems
Severe problems
Unable to walk
Group 1

10. Anchoring Vignettes:
identifying response bias and DIF in self assessed health
Necessary assumptions
• Vignette equivalence (VE) holds if all respondents interpret the
health states described by the vignettes in the same way and on
the same uni-dimensional scale, aside from random error.
• VE is demonstrated in the example above by the horizontal
dotted lines
• Response consistency (RC) is where respondents rate the health
of the hypothetical people described in the vignettes in the same
way or using the same underlying scale that they would rate their
own health.
• RC would be violated if, for example, respondents rated the
health described by the vignettes either more or less harshly
than they did their own health

11. Anchoring Vignettes:
identifying response bias and DIF in self assessed health
What do we know so far?
• Au and Lorgelly (2014) Quality of Life Research
• Evidence that vignettes for the EQ-5D-5L are feasible
• Suggested improvements required in the wording in order to
improve response consistency
• Knott et al (2016) Health Economics
• Considers some of the issues of using vignettes
• Reviews benefits of operationalising the approach
• Knott et al (2016) HEDG York Working paper (16/14)
• Vignettes can be used identify DIF in the EQ-5D-5L (at least in
certain age groups)
• Failure to adjust for DIF can lead to conclusions that are
misleading
• Further work is needed to achieve vignette equivalence

12. Anchoring Vignettes:
identifying response bias and DIF in self assessed health
DIF adjusted indices
0.6
0.65
0.7
0.75
0.8
0.85
0.9
0.95
1
EQ-5DIndex
Index based on self-reports DIF-adjusted index
0.6
0.65
0.7
0.75
0.8
0.85
0.9
0.95
1
EQ-5DIndex
Index based on self-reports DIF-adjusted index
Difference=0.049
Difference=0.095

13. Anchoring Vignettes:
identifying response bias and DIF in self assessed health
What do we know so far?
• Knott & Lorgelly (2016) HESG Paper - summer
• It is possible to correct for DIF using responses to anchoring
vignettes that are collected externally to the main dataset of
interest
• Resulting QALY measures can be considered comparable across
different population groups

14. Anchoring Vignettes:
identifying response bias and DIF in self assessed health
DIF adjustment – group differences
-0.004
0.054
0.038
0.093
0.065
0.08
-0.02
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
Male - Female High educ - Low educ Migrant - Born Aus Employed - Not employed Married - Alone Aged 65 plus - Under 65
DifferenceinEQ-5D-5Lindices
Unadjusted scores DIF-adjusted scores
0.016
0.079
0.037
0.141
0.096 0.097
-0.02
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
Male - Female High educ - Low educ Migrant - Born Aus Employed - Not employed Married - Alone Aged 65 plus - Under 65
DifferenceinEQ-5D-5Lindices
Unadjusted scores DIF-adjusted scores
MID=0.074

15. Anchoring Vignettes:
identifying response bias and DIF in self assessed health
Where to next?
• More research to better understand the vignette equivalence
failure issue
• Will there always be a trade-off with response consistency?
• Is there value in exploring DIF cross-culturally?
• Multi-national clinical trials, often apply one country’s tariff as
if all respondents are within that country
• Is the external adjustment as good as (or a close substitute for)
collecting them within a study?
• What does this mean for economic evaluations and the decisions
they inform?
• Could response behaviour change over time?

16. Anchoring Vignettes:
identifying response bias and DIF in self assessed health
For enquiries relating to this presentation, please contact
Paula Lorgelly at plorgelly@ohe.org
To keep up with the latest news and research, subscribe to our blog, OHE News
Follow us on Twitter @OHENews, LinkedIn and SlideShare
Office of Health Economics (OHE)
Southside, 7th Floor
105 Victoria Street
London SW1E 6QT
United Kingdom
+44 20 7747 8850
www.ohe.org
OHE’s publications may be downloaded free of charge from our website.