Measuring the right outcomes in mental health

Outcome Measurement in Mental
Health
John Brazier
Director of the Economic Evaluation of Health and Care Interventions
Policy Research Unit (EEPRU)

School of Health and Related Research
The University of Sheffield, UK

CEMPH Conference 7 November 2013

Background
Growing use of generic outcome measures
in people with mental health problems:
• Surveys (e.g. ESEMed, PMS in UK)
• Use in clinical trials

• Assessing cost-effectiveness to inform
reimbursement decisions (e.g. NICE)
• Routine outcome monitoring (e.g. IAPT,
PROMS programme, PBR)
Things have changed since the Gilbody review!
27/11/2013 © The University of Sheffield

2

3

What should we be measuring?
• Quality of life (QoL) or well-being are ill-defined and
there are different ways of conceptualising them:
functionings, capabilities, wellbeing etc.
• The World Health Organization (1948) declared
health to be

“A state of complete physical, mental and social wellbeing, and not merely the absence of disease and
infirmity”
• QoL means different things to different people and
this is why we need the views of mental health
service users in developing and testing measures

Types of measure
• Generic measures: those instruments
designed for use on any population (EQ-5D,
ICECAP etc.)
• Condition specific measure (CSM): those
instruments designed for use in a specific
population (CORE-10, PHQ9, GAD etc.)
Both types of measure are standardised and
come with a scoring algorithm (that may or
may not be ‘preference based’ for calculating
QALYs)

Quality Adjusted Life years
(QALY)

• QALYs combines both quantity and quality of life into a
single measure
• QALYs can be used across all health care interventions
for all patient groups

6

By placing a tick in one box in each group below, please indicate which statements
best describe your own health state today.
Mobility
I have no problems in walking about
I have some problems in walking about
I am confined to bed





Self-Care
I have no problems with self-care

•

•
•

I have some problems washing or dressing myself

Scored using UK TTO
values ( Dolan et
al,1997) with a range
minus 0.54 (worst
impairment) to 1 (full
health)
Preferred by NICE and
used in DH PROMS
programme
5 level version now
available

I am unable to wash or dress myself





Usual Activities (e.g. work, study, housework, family or
leisure activities)
I have no problems with performing my usual activities
I have some problems with performing my usual activities
I am unable to perform my usual activities





Pain/Discomfort
I have no pain or discomfort
I have moderate pain or discomfort
I have extreme pain or discomfort





Anxiety/Depression
I am not anxious or depressed
I am moderately anxious or depressed
I am extremely anxious or depressed





How do we know if a measure like EQ-5D is
measuring the right thing?
• Does it cover the important dimensions of (mental) health
related quality of life and relevant ranges (Content
validity)

• Does it reflect known group differences and correlate with
other indicators of quality of life (Construct validity)
• Does respond to known changes quality of life
(Responsiveness)
Assessing validity is problematic due to the absence of a gold
standard and validity being a question of degree

8

Overview of talk
Based on the findings of an MRC funded study looking at the
appropriateness of EQ-5D and other generic measures in people with
mental health problems

• Psychometric evidence – based on a systematic
review and further analyses
• Qualitative evidence – based on a review and indepth interviews with mental health service
users
• Implications for existing measures: EQ-5D, SF6D and the new kid on the block the SWEMWBS

Systematic review
To assess the appropriateness of the EQ-5D and the SF36 family in terms of their validity and responsiveness in
five mental health conditions:
• Schizophrenia, Bipolar disorder, Personality disorders,
Depression and anxiety

• Ten health databases were searched
• Studies were appraised and data extracted using a
standardised template.
• Performance:
• Construct validity: known group differences with general
population control or between severity groups and convergent
validity
• Responsiveness to known changes
27/11/2013

9

10

PRISMA FLOW DIAGRAM
Citations retrieved by
search of electronic
databases (n=7,095)
Unique citations
(n=4,115)

Titles and/or abstracts
potentially relevant (n=266)

Studies satisfying
inclusion criteria (n=154)

PDs (n=9)

Schizophrenia
(n=33)


Bipolar disorder
(n=22)

Duplicate citations
(n=2,980)

Citations excluded
after screening
(n=3,849)

Full papers excluded
after screening
(n=124)
12 references from
reference list check

Depressive &
anxiety disorders
(n=23 – not SF-36)

11

Depression and Anxiety
EQ-5D
√√√√x √√

SF-6D
√√

Known group: Casecontrol
Convergent validity

-

√√

√√√√√

√√√

Responsiveness

√√√√x√√√√√√√√√√

√√

Known group: Severity

Where √ indicates results in support of validity or responsiveness and x
indicates an inconsistent or non-significant results by test (and not an
individual study);

12

• EQ-5D and SF-6D: good known group and
convergent validity, and responsiveness.
• More highly correlated with depression scales in
patients with anxiety than with clinical anxiety
scales
• Convergent validity in patients with anxiety may
be driven by aspects of depression within
anxiety disorder and presence of co-morbid
depression.

13

Personality Disorders
EQ-5D
√

Known group:
severity
Known group:
√
case-control
Convergent validity √
Responsiveness

√√√

SF-36
x

SF-12
√√

SF-6D
-

√√

-

-

√

-

-

√

-

-

14

Personality disorders
• EQ-5D: responsive, KGV and CV
• Limited and mixed SF-36 evidence
(related to two studies only)
• Very little evidence on SF-12 and
none for SF-6D

15

Schizophrenia
EQ-5D

SF-36

Known group:
Severity

√

√

-

-

Know group:
Case-control

-

√√√√√√√√√√
√

√

-

√ √ √ x x √ x √ √ √√√√√√√√√ x
x √√√√ x x
xxxx

-

√

-

x

Convergent
validity
Responsiveness

√ √√ x x

√√√ x x x x

SF-12 SF-6D

16

Schizophrenia
• KGV: Yes, but crude measures
• CV & R: Mixed results
• Clinical assessment of symptoms e.g. Positive and
Negative Symptoms Scale (PANSS), functioning
measures (e.g. Global Assessment of Functioning)
and schizophrenia specific measures of HRQL (e.g.
QLS).

• Some evidence that EQ-5D reflected depression
rather than other symptoms.

17

Bipolar disorder
EQ-5D
SF-36
√√
√√√x√√

Known group:
Severity
Know group: Case- x
control

Convergent validity
Responsiveness

SF-12 SF-6D
√
-

√√√√√√√ √√x

-

√x√ x √ √ √ √ √ √ √ √ √
x
√√√√x
-

-

18

Bipolar Disorder
•

Majority evidence on the SF-36
• Crude KGV
• CV good

•

EQ-5D (4 studies)
• Mixed results for CV and R
• EQ-VAS performed better than the index.

• Properties demonstrated for depression but not
mania

Overview of quantitative
evidence

19

 EQ-5D and SF-36 achieved adequate levels of performance in
depression and to some extent anxiety and personality disorder (but
limited evidence)
 Results were more mixed in schizophrenia and bi-polar with a
suggestion that results in some studies may reflect differences in
depression
 These findings were supported by analyses of further data sets
undertaken as part of the research

But:
 Need more evidence using better indicators for testing validity and
responsiveness is required.
 Need to assess content validity using qualitative research to better
understand the findings


21

Why Qualitative Research?
• Content and face validity
• Perspective of individual important

• Required by FDA for measures being used
to support labelling claims (and generally
good practice)


22

Two research studies
• Systematic Review and synthesis of
qualitative research
• Primary research – interviews of service
users


23

Study 1: Systematic review
• 13 studies
• Canada/UK/Sweden/USA/Australia/NZ
• Occupational Therapy (5); Nursing (4); Psychology (2);
Psychiatry (1) Social worker (1)

• Schizophrenia (3); Bi-polar (3); Panic Disorder (1); Mixed
(6)
• Framework analysis used to identify common and
variable patterns of themes
• Limitations: focus on severe mental health schizophrenia/bi-polar

24

Study 2: Service User Interviews
• 19 interviews

• Broad range diagnosis and severity
• Severe and enduring (CMHT x 2)
-Schizophrenia; Bipolar; Personality Disorder;
-Severe Depression/Anxiety; PTSD
• Mild to Moderate (IAPT)
Anxiety/Depression
• Themes from review made up initial themes of
framework
• Limitations – gatekeepers/diagnostic range

25

Conceptual Difficulties
• Quality of life as a positive concept
• Responses tend to focus on negative aspects
of condition including clinical symptoms
• Symptoms being separate from QoL
• Review - overlap of QoL with other concepts

• Recovery/lived experience
• Overlap of domains within QoL
• e.g. Self-stigmatization (Belonging or self
perception)

26

Findings – QoL domains

Autonomy

Well-Being
Ill-Being

Self
Perception

Belonging
Activity

Hope
Physical
health


27

Well-Being – Ill-Being
Positive – adds quality

Negative – takes quality away

Overall sense of well-being

Overall feelings of distress

Feeling calm/relaxed

Anxiety/worry/fear

Feeling safe

Low mood/boredom

Enjoyment

Lack of energy/feeling tired

(Happiness)

Lack of concentration

(If there was one thing you could change to improve your life, what would
it be) I would lose this anxiety that I seem to be constantly carrying with me.
I don’t know why, I don’t know how, although I can remember when me and
my sister spoke she says, at that time, they always used to say I was very
highly strung, it’s a term that you don’t really hear now, but I was always a
bit like that when I was very young anyway and I seem to have carried this
anxiety and nervousness with me ever since (IAPT Panic attacks)

28

Physical Health
• Physical health problems affect mental health
• Mental health affects physical health
• Presence of both makes life difficult to cope with
it feels physical as well as mental …. my body aches and like
I think I just become really tense and that is what makes my
body ache and I feel like erm I feel like my chest is being
crushed and erm I can’t breath and things like that and erm
I just want to be asleep all the time to escape but I can’t
sleep (CMHT Severe depression/Socio-affective PD)

29

Self Perception


Coherent sense of self

Lack of self identity

Positive self identity

Low self esteem

Self acceptance

Lack of confidence
Feelings of failure/uselessness

The worse part is the lack of self worth and having to
accept that one is disabled one has a stigma which
stops you from doing anything else really other than
being worthless (CMHT Schizophrenia)

30

Relationships and Belonging


Accepted and understood

Lack of understanding

Support

Stigma

Companionship/camraderie

Rejection/exclusion

Love and affection

Loneliness/isolation

Trust

Abuse

Feeling part of society

Feeling alien to society

I have feelings of erm not belonging to the human race, like, I feel very-, it’s not an
outcast, I just don’t feel a connection erm I don’t know how else to describe that, it’s
being like an alien, that’s the only way I can describe that, and I know that sounds
weird but that’s the only way I can describe the feeling of it, I don’t feel akin with
anybody, I am very guarded and things like that …I would just like to be supported by
other people all working to a common cause err helping other people, that’s all I have
ever wanted to do (CMHT Severe Depression/Anxiety)

31

Autonomy


Choice

Lack of choice

Opportunity

Lack of opportunity

Control/self determination

Control (excessive)

Coping

Not coping - overwhelmed

I seem a bit of a control freak, I want everything to be worked out before I
decide to do a certain thing, you know, I want everything to be fairly
straightforward and I mean, you can’t, in a way, you can’t live life like that,
and yet I still want to live life like that, do you know what I mean? … … it’s
about the stress, erm having faith or taking this, stepping out of your comfort
zone, whatever you want to call it, yeah (Schizophrenia)


32

Activity


Enjoyment – well being

Stress (exacerbates symptoms)

Rewarding - esteem

Failure - esteem

Meaning and purpose
Belonging – social activity
Structure

Distraction from problems
I went on a year’s course at engineering and I was absolutely scared about going on that,
but I did it and I did it, you know, quite well … and at that time, I was really happy in my
life and I thought well I’ve done something, I’ve achieved something here doing this … I
think that’s because I had the drive and a purpose of getting up and going out every
morning and doing what, you know, normal people do, sort of, you know, I got into a
routine which was very good, so I was happy and more stable, I just felt a little bit more
worthwhile, you know. I don’t like to feel useless really (Anxiety/panic attacks)

33

Hope and Hopelessness


Positive view of future

Negative view of future

Optimism

Pessimism

Dreams and goals

Feelings of loss

Thinking ahead

Demoralization

The one thing that I used to do a lot is not think about the future, I’d think a
couple of days ahead and then not think about, you know, any further than
that … cos now, one thing that’s different from when I started going through
this process, is that I’m more willing to think further ahead, you know, I’m
more willing to say, well in a year’s time I’d like to be at this place, before I
did this I wouldn’t, but there’s still a long way to go, and still a lot of obstacles
that I’ve got to overcome (Depression/trans-gender)

34

Overview of qualitative findings
• Measuring QoL in mental health complex
• QoL covers negative as well as positive aspects
• Similar themes to ‘personal recovery’ – also service
user driven:
• Boardman review for DH
• Connectedness, Hope and optimism, Identity,
Meaning and Empowerment (CHIME) (Leamy et
al, 2011)


Content validity of EQ-5D
Well-being/Ill-being

- negative only, excludes calm, happiness,
energy , safety etc

Physical health

Mobility, self care, usual activity and pain or
discomfort

Self-perception

None

Relationships and
belonging

Usual activities?
- Excludes affection, trust, support and loneliness,
stigma, abuse etc. (the quality of contact)

Autonomy

None

Activity

Usual activity (and perhaps self-care)
- Nothing on stress/failure/meaning/ purpose etc.

Hope and hopelessness

None


36

Content validity of SF-6D

Depression and anxiety, energy
- Calmness and happiness in SF-36

Physical health

Physical functioning, role limitation, pain

Self-perception

None

Relationships and
belonging

Social functioning
- Excludes affection, trust, support and loneliness,
stigma, abuse etc. (the quality of contact)

Autonomy

None

Activity

Role limitation


None
- General health perception in SF-36 asks about
future health expectation, but very limited


37

38


39

Adoption of
SWEMWBS for PBR


The Short Warwick-Edinburgh Mental Well-being Scale (S-WEMWBS)
Below are some statements about feelings and thoughts. Please tick the box that best describes your
experience of each over the last 2 weeks

None
of the
time

Rarely

I’ve been feeling optimistic about
the future

1

2

3

4

5

I’ve been feeling useful

1

2

3

4

5

I’ve been feeling relaxed

1

2

3

4

5

I’ve been dealing with problems
well

1

2

3

4

5

I’ve been thinking clearly

1

2

3

4

5

I’ve been feeling close to
other people

1

2

3

4

5

I’ve been able to make up my
own mind about things

1

2

3

4

5

STATEMENTS

Some
of
Often
the time

All of
the
time

“Short Warwick Edinburgh Mental Well-Being Scale (SWEMWBS)
© NHS Health Scotland, University of Warwick and University of Edinburgh, 2008, all rights
reserved.”

40

Content validity of S-WEMWBS

Feeling relaxed, thinking clearly

Physical health
Self-perception

Feeling useful

Relationships and
belonging

Feeling close to other people

Autonomy

Make up my own mind about things

Activity

Dealing with problems


Feeling optimistic


41

Implications for
research

42

• Further testing of construct validity and
responsiveness of EQ-5D and SF-6D using
better indicators of HRQL
• Comparative testing of SWEMWBS on
populations with mental health problems
• Extend qualitative research to conditions not well
covered (e.g. OCD), recruit through different
channels and extend to other countries and
cultures
• Develop a mental health specific generic
preference-based measure?

43

Developing a generic preferencebased mental health measure
Questionnaire
• Confirm dimensions
• Develop items to reflect dimensions from qualitative work

• Test properties of items in range of populations
Scoring
• Value using Time trade-off or discrete choice
experiments with duration
• General public and people with mental health problems

44

References:
•

Brazier JE, Connell J, Papaioannou D, Mukuria C, Mulhern B, O’Cathain A, Barkham
M, Knapp M, Byford S, Gilbody S, Parry G. Validating generic preference-based
measures of health in mental health populations and estimating mapping functions for
widely used specific measures. Health Technology Assessment (in press)

•

Connell J, Brazier JE, O'Cathain A, Lloyd-Jones M, Paisley S. Quality of life of people
with mental health problems: a synthesis of qualitative research
Health and Quality of Life Outcomes 2012, 10:138.

•

Papaionnou D, Brazier JE, Parry G. How to measure quality of life for cost
effectiveness analyses in personality disorders? A systematic review. Journal of
Personality Disorder 2013; 27(3):383-401

•

Papaionnou D, Brazier J, Parry G. How valid and responsive are generic health
status measures, such as the EQ-5D and SF-36, in schizophrenia? A systematic
review. Value in Health 2011, 14(6):907-920.


Measuring the right outcomes in mental health

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Measuring the right outcomes in mental health

Similar to Measuring the right outcomes in mental health (20)

Recently uploaded

Recently uploaded (20)

Measuring the right outcomes in mental health

Editor's Notes