Heterogeneity in meta-analysis

Saudi Board of Preventive Medicine, Riyadh Ministry of Health, KSA Lecture 07/10
Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course
Heterogeneity in
meta-analysis
Dr. S. A. Rizwan M.D.,
Public Health Specialist & Lecturer,
Saudi Board of Preventive Medicine – Riyadh,
Ministry of Health, KSA
25.11.2019 1
With thanks to Martin Bland

Outline
• What is heterogeneity?
• Sources of heterogeneity
• Impact of heterogeneity on different parameters
• Quantification of heterogeneity
• Dealing with heterogeneity
25.11.2019 2

What do we mean by Heterogeneity?
25.11.2019 3
It’sthe variance in trueeffects(not observed effects) that we careabout

Where does it arise from?
• Studies differ in terms of
• Patients
• Interventions
• Outcome definitions
• Design
• This is called Clinical heterogeneity
• Variation in true treatment effects in magnitude or direction
• This is called Statistical heterogeneity
25.11.2019 4

Where does it arise from?
• Statistical heterogeneity may be caused by
• clinical differences between trials
• methodological differences between trials
• unknown trial characteristics
• Even if studies are clinically homogeneous there may be statistical
heterogeneity
25.11.2019 5

Why estimate heterogeneity?
It affects weight, mean, SE and utility of effects
25.11.2019 6

Impact on Weight
25.11.2019 7

It affects the weights
25.11.2019 8

W = 1/ (V1)
Weights when T2 = 0
25.11.2019 9

Weights when T2 > 0
25.11.2019 10
1W = 1/ (V +T 2
)

Impact on Standard Error
25.11.2019 11

It affects the standard error
25.11.2019 12
… the confidence interval, and thep-value

Dr. S. A. Rizwan, M.D.Demystifying statistics series: Meta-analysis course 25.11.2019 13

140
25.11.2019 16

Precision ofthe
mean effect
25.11.2019 17

Precision ofthe
mean effect
25.11.2019 18

Precision ofthe
mean effect
25.11.2019 19

Impact on Mean ES
25.11.2019 20

It affects the mean ES
25.11.2019 21

It affects the mean ES
• As heterogeneity increases, mean effect shifts away from larger
studies and towards smaller studies
25.11.2019 22

Impact on substantive utility
25.11.2019 25

It affects the substantive utility
25.11.2019 26
Is the treatment effectivefor everyone, or
effectivefor some and harmful for others?

150
25.11.2019 29

Dispersion ofthe
individual effects
25.11.2019 34

Dispersion ofthe
individual effects
25.11.2019 35

How do we quantify
heterogeneity?
25.11.2019 36

Two-step process
• Isolate the real dispersion
• Translate this into useful indices
25.11.2019 37

1. Isolate the real dispersion
25.11.2019 38

What we’d like to see if the true effect is the
same in all studies
25.11.2019 39

What we might see if the true effect is the
same in all studies
25.11.2019 40

Is there real dispersion?
25.11.2019 41

It depends on the precision
25.11.2019 42

It depends on the precision
25.11.2019 43

Key point
• We can easily compute variance of observed effects
• But this is due partly to real differences in effects and partly to
sampling error within studies
• We need to isolate the between-studies variance
25.11.2019 44

170
To assess heterogeneity
• Compute observed variance
• Estimate how much variance would be expected if true effect is
identical in all studies
• Observed minus expected is estimate of true variance
25.11.2019 48

Q df Q-df
Isolating the real dispersion
25.11.2019 49

2. Translate into useful indices
25.11.2019 50

Indices related to heterogeneity
25.11.2019 51
Q−df Basis for all indices
p Test of null
T Standard deviation of true effects
T2 Variance of true effects
I2 Proportion of true/total variance

P-value
• Can we conclude that there is some variance in true effects
• Depends on amount of excess variance and the amount of evidence
25.11.2019 52

Statistics apply to both models
25.11.2019 56

I square – an alternative of Q
• Q is statistically under-powered when the number of studies is low
and when the sample size within the studies is low
• Larger values of I2, the more heterogeneity
• 75%: large heterogeneity
• 50%: moderate heterogeneity
• 25%: low heterogeneity
25.11.2019 57

Caution!
• I2 is NOT a measure of absolute heterogeneity
• I2 tells us what proportion of the observed dispersion reflects
differences in true scores rather than random sampling error
25.11.2019 58

What proportion
of the observed variance is real?
25.11.2019 59
I2

Test for heterogeneity: χ2 = 4.91, df = 2, P=0.086
Heterogeneity
25.11.2019 60

What happens when heterogeneity is not
significant
• No statistical evidence for difference between trials
• But, test for heterogeneity has low power – the number of studies is
usually low - and may fail to detect heterogeneity as statistically
significant when it exists.
• This cannot be interpreted as evidence of homogeneity.
• To compensate for the low power of the test a higher significance
level is sometimes taken, P < 0.1 for statistical significance.
25.11.2019 61

What happens when heterogeneity is
significant
• Differences between trials exist
• It may be invalid to pool the results and generate a single summary
result
• Describe variation
• Investigate sources of heterogeneity
• Account for heterogeneity
25.11.2019 62

Dealing with heterogeneity
• Do not pool — narrative review.
• Ignore heterogeneity and use fixed effect model:
• confidence interval too narrow,
• difficult to interpret pooled estimate,
• may be biased.
• Explore heterogeneity, can we explain it and remove it?
• Allow for heterogeneity and use random effects model
• Change the ES measure
• Exclude extreme outlier studies (caution!)
25.11.2019 63

Investigating sources of heterogeneity
• Subgroup analysis:
• subsets of trials,
• subsets of patients,
• subsets should be pre-specified to avoid bias.
• Relate size of effect to characteristics of the trials, e.g.:
• average age,
• proportion of females,
• intended dose of drug,
• baseline risk.
• ‘Meta-regression’ can be used.
25.11.2019 64

Investigating heterogeneity – Subgroups
• Split into 26 sub-studies with
more uniform age groups.
• Before adjustment for age: X2 =
127, df=9, P<0.001.
• After adjustment for age: X2 =
45, df=23, P=0.005.
• A considerable improvement,
but still some heterogeneity
present.
25.11.2019 65

Investigating heterogeneity – Subgroups
• We want to see if the mean effect sizes for studies grouped by age of
student differ from each other
• Question: What are our assumptions about τ2 ?
• Option 1: We assume that each group has its own underlying distribution
of effect sizes, so that we are really estimating μ and separate variance
components for each category of the age.
• Usually we don’t have a large number of studies within each group, and our estimates
of the variance components will not be precise
• Option 2: Assume that we have a common variance component τ2
25.11.2019 66

Investigating heterogeneity – meta-regression
• Line fitted by meta-regression.
• Odds ratios of ischemic heart
disease (and 95% CI) according to
the average extent of serum
cholesterol reduction achieved in
each of 28 trials.
• Overall summary of results is
indicated by sloping line.
• Results of the nine smallest trials
have been combined.
25.11.2019 67

Investigating heterogeneity - Galbraith plot
• Alternative graphical representation to forest plot.
• Horizontal axis: 1/standard error.
• Horizontal axis will be zero if standard error is infinite, a study of zero size.
• Vertical axis: effect/standard error.
• This is the test statistic for the individual study.
• For 95% of studies, we expect this to be within 2 units of the true effect.
25.11.2019 68

• Corticosteroids for severe sepsis
and septic shock (Annane et al.,
2004)
-4
-2
0
2
Difference/standarderror
0 2 4 6
1/standard error
Long, low dose trials
25.11.2019 69
Galbraith plot for log OR
Low doses & long duration trials

• We can add a line representing
the pooled effect.
• Plot (pooled effect)/se against
1/se.
-4
-2
0
2
0 2 4 6
1/standard error
Pooled effect
25.11.2019 70

• We can add a line representing the
pooled effect.
• Plot (pooled effect)/se against 1/se.
• 95% limits will be 2 units above and
below this line.
• We expect 95% of points to be
between these limits if there is no
heterogeneity.
• This is true for low dose, long duration
trials.
-4
-2
0
2
0 2 4 6
1/standard error
Pooled effect 95% limits
25.11.2019 71

• We have two points outside the
95% limits and one on the line.
• We can investigate them to see
how these trials differ from the
others.
• The pooled effect is smaller so the
line is less steep. -4
-2
0
2
0 2 4 6
1/standard error
25.11.2019 72
All trials

• We could reanalyse taking
dosage and duration separately.
• These trials are all of high dose
or short duration treatments.
-4
-2
0
2
0 2 4 6
1/standard error
Long, low dose Short, high dose Other
25.11.2019 73
All trials

Galbraith plot or Forest plot?
Wagner
CSG
Klatersky
Schumer
Lucas
Sprung
Bone
VASSCSG
Luce
Slusher
Bollaert
Briegel
Chawla
AnnaneYildiz
-4
-2
0
2
0 2 4 6
1/standard error
Pooled
Yildiz
Annane
Chawla
Briegel
Bollaert
Slusher
Luce
VASSCSG
Bone
Sprung
Lucas
Schumer
Klatersky
CSG
Wagner
.01 .1 1 10 100
Odds ratio of death
25.11.2019 74
“Conventional meta-analysis diagrams . . . are not very useful for investigating heterogeneity. A
better diagramfor this purpose was proposed by Galbraith . . .” (Thompson, 1994). Is this really
true?

Galbraith plot or Forest plot?
Wagner
CSG
Klatersky
Schumer
Lucas
Sprung
Bone
VASSCSG
Luce
Slusher
Bollaert
Briegel
Chawla
AnnaneYildiz
-4
-2
0
2
0 2 4 6
1/standard error
Pooled
Yildiz
Annane
Chawla
Briegel
Bollaert
Slusher
Luce
VASSCSG
Bone
Sprung
Lucas
Schumer
Klatersky
CSG
Wagner
.01 .1 1 10 100
Odds ratio of death
25.11.2019 75
Trials outside the Galbraith limits will be trials where the 95% confidence interval
does not contain the pooled estimate. We can spot them from the forest plot.

L’ abbe plot
25.11.2019 76
• Ideally a L'Abbé plot should have the
symbols appropriate to the size of the
trials.
• There is an inset for the symbol size, and
the two colours show trazodone used for
erectile dysfunction in two different
conditions (and with clear clinical
heterogeneity).
Trazodone for erectile dysfunction in psychogenic
erectile dysfunction (dark symbols) and with
physiological or mixed aetiology (light symbols)

• Cannot always explain
heterogeneity
• Example: Effect of breast
feeding in infancy on blood
pressure in later life (Owen
et al., 2003)
• *In parenthesis: age at
which blood pressure
measured
25.11.2019 77

• Cannot always explain
heterogeneity
• X2=59.4, 25df, P<0.001
• Three age groups: P=0.6.
• Born before or after 1980: P=0.8.
• Have to accept it and take it into
account by using a random effects
model.
25.11.2019 78

Take home messages
• Heterogeneity is not a bad thing
• It should not be ignored
• It should be understood, explored and quantified
• And it should be properly handled in analysis
• Discussion of meta-analysis findings should include a major note on
heterogeneity of effects
25.11.2019 79

Thank you
Kindly email your queries to sarizwan1986@outlook.com
25.11.2019 80

Heterogeneity in meta-analysis

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Heterogeneity in meta-analysis

Similaire à Heterogeneity in meta-analysis (11)

Plus de Rizwan S A

Plus de Rizwan S A (18)

Dernier

Dernier (20)

Heterogeneity in meta-analysis