Meta-Analysis

Critical appraisal of a meta-analysis study
Samir Haffar M.D.
Associate Professor of Gastroenterology
Al-Mouassat University Hospital – Damascus – Syria

Hierarchy of evidence in quantitative studies
McGovern D, Summerskill W, Valori R, Levi M. Key topics in EBM.
BIOS Scientific Publishers, 1st Edition, Oxford, 2001.

Gene Glass
American statistician – University of Colorado
Involved in social science research
He coined the term meta-analysis in 1976

Logo of Cochrane collaboration
http://www.cochrane.org
Database available free online in many countries

Number of publications about MA (1986 - 1999)
Results from Medline search using MeSH
“meta-analysis” & text word “systematic review”
Egger M et all. Systematic reviews in health care: Meta-analysis in context.
BMJ Publishing Group, London, 2nd edition, 2001.

Greenhalgh T. How to read a paper - The basics of evidence based medicine.
BMJ Publishing Group -2nd Edition - London - 2001.
Trisha Greenhalgh
“ If I had to pick one word which exemplifies
the fear felt by so many students, clinicians, &
consumers towards evidence-based medicine,
that word would be meta-analysis”

1- Ask
2- Acquire
4- Apply
5- Assess
Patient
dilemma
Principles
of EBP
Evidence alone does not
decide – combine with other
knowledge & values
3- Appraise
Hierarchy
of evidence
5 A

Clinical history
• 60-year-old man with acute biliary pancreatitis
• Ranson‟s score: 4 – No fever – Normal WBCs
• CECT* on day 7: CT grading system of Balthazar 3
Necrosis score 2
CT severity index 5
• You wonder if prophylactic antibiotics prevents infection
of non-infected pancreatic necrosis & decreases mortality
*CECT: Contrast-Enhanced Computed Tomography

Ranson’s score for gallstone pancreatitis
Age > 70 yr
Blood glucose >220 mg/dl
WBC >18,000/mm3
LDH > 400 IU/L
ASAT > 250 IU/L
At presentation
1 point for each positive factor
Severe acute pancreatitis: ≥ 3
Ranson JHC. Am J Gastroenterol 1982;77:633.
During initial 48 hr
Ht >10% decrease
Serum calcium < 8 mg/dl
Base deficit > 5 mEq/L
BUN > 2 mg/dl increase
Fluid sequestration > 4 L

CT grading system of Balthazar
Grade Description Points
A Normal pancreas 0
Balthazar EJ et al. Radiology 1990 ; 174 : 331 – 6.
B Pancreatic enlargement 1
C Inflammation of pancreas or peripancreatic fat 2
D Single peripancreatic fluid collection 3
E ≥ 2 fluid collections or retroperitoneal air 4

Necrosis score
Necrosis Points
No pancreatic necrosis 0 points
One third of pancreas 2 points
One half of pancreas 4 points
> one half of pancreas 6 points

CT severity index
The index ranges from 0 to 10
Severe acute pancreatitis ≥ 3
CT grading of Balthazar
(0 – 4 points)
Necrosis score
(0 – 6 points)
+
Morgan DE. Clin Gastroenterol Hepatol 2008 ; 6 : 1077 – 1085.

CT Severity Index (CTSI)
Localized fluid collection adjacent to tail: CT grading (3 points)
Lack of enhancement of pancreatic tail: Necrosis <30 % (2 points)
Absence of retroperitoneal air

Key components of your clinical question
PICO
P Patient Severe AP with CT-proven necrosis
I Intervention Prophylactic antibiotics
C Comparaison Placebo or no treatment
O Outcome Infected pancreatic necrosis – Mortality
Prophylactic antibiotics in pancreatic necrosis

PubMed translation of query into search terms
PICO Element Search terms for PubMed
P Acute necrotizing pancreatitis “acute necrotizing pancreatitis” [MeSH]
* MeSH: Medical Subject Headings in PubMed
I Prophylactic antibiotics “antibiotic prophylaxis” [MeSH term]
C Placebo
No treatment
“placebo” [MeSH term]
O Infected necrosis
Mortality
“infection” [MeSH term]
“necrosis” [MeSH term]
“mortality” [MeSH term]
Other Meta-analysis SR in PubMed Clinical Queries

Systematic review & meta-analysis
Systematic reviews
(SR)
Meta-analyses
(MA)
MA may, or may not, include a SR

Definition of meta-analysis
“Statistical analysis that combines or integrates
the results of several independent clinical trials
considered by the analyst to be combinable”
Proceedings of biopharmaceutical section of American statistical association.
1988 ; 2 : 28 – 33.

Rationale for a meta-analysis
By combining the samples of individual studies,
the overall sample size is increased, thereby
improving the statistical power of the analysis as
well as precision of estimates of treatment effects

Steps of meta-analysis
 Formulation of the problem to be addressed
 Data collection
 Data recording
 Data analysis
 Reporting the results (Forest plot)
Researchers should write in advance a detailed protocol

 Formulation of the addressed problem
PICO
P Patient Severe AP with CT-proven necrosis
I Intervention Prophylactic antibiotics
C Comparaison Placebo or no treatment
O Outcome Infected pancreatic necrosis -Mortality
Study design: RCTs

 Formulation of the addressed problem
• Controlled trials
• Randomization of patients
• Intention to treat principle (ITT)
• Preferably blinded
• Outcome assessment: p – RR – OR – CIs – NNT
Guyatt G, et al. User’s guide to the medical literature.
Essentials of evidence based clinical practice. Mc Graw Hill, 2nd ed, 2008.
Specify inclusion & exclusion criteria

Basic structure of a RCT / Parallel trial
Petrie A, Sabin C. Medical statistics at a glance. Blackwell Publishing, 2nd edition, 2005.
Most frequently used design

Randomization
• Simple randomization
• Random table
• Block randomization
• Stratified randomization
• Minimization method
• Unequal randomization
• Allocation concealment
Inacceptable
Preferred

Intention to treat analysis
Quality control rather than analytic tool
• Strategy in conduct & analysis of RCT ensuring that all
patients allocated to treatment or control groups are
analyzed together as representing that treatment arm
whether or not they received the prescribed treatment or
completed the study
BIOS Scientific Publishers, 1st ed, Oxford, 2001.
Randomized participants = Analyzed participants

Blinding or Masking
• Participants
• Investigators who administer interventions
• Investigators taking care of the participants
• Investigators assessing the outcomes
• Data analyst
• Investigators who write results of the trial
Blinding can be implemented in at least 6 levels in RCTs
Usually
the same

 Data collection
Finding all studies (Is there an existing SR?)
• Electronic search
Initial search PubMed – Cochrane Review
Others databases: EMBASE, CINAHL
Further search References of relevant reviews
Find terms you didn’t use (MeSH*)
Search again Snowballing
• Supplementary search
Hand search
Write to researchers
* MeSH: Medical Subject Headings in MEDLINE

Studies included in meta-analysis
Studies reviewed
Gray
literature
All studies published
All studies conducted

Why using multiple sources?
Papers identified in a SR of near patient testing
Unique
Not unique
Glasziou P et al. Systematic reviews in health care: a practical guide.
Cambridge University Press, 1st edition, 2001.

 Data collection
• Electronic databases – MEDLINE
– EMBASE
– CCTR
– Cochrane Library
– Science Citation Index
• Hand search – References from published trials
– Major conference abstracts
CCTR: Cochrane Controlled Trials Register
Bai Y et al. Am J Gastroenterol 2008 ; 103 : 104 – 110.

 Data recording
• 2 independent observers extract the data
• Quality of the studies may be rated with specially
designed checklist or scales
• Blinding observers to names of authors, institutions,
names of journals, funding & acknowledgments

Existing tools to assess trial quality
• Several components grouped in
Scales Each item scored numerically
Overall quality score is generated
Checklists Components evaluated separately
No numerical scores
• Systematic search of literature in 1995 identified
25 scales & 9 checklists for assessing trial quality*
* Moher D et all. Controlled clinical trials 1995 ; 16 : 62 – 73.

• Quality assessment performed independently by 2
authors using empirical evidence 1-2
• Disagreement resolved by discussion between 2 reviewers
• Low risk of bias Generation of allocation sequence
Allocation concealment
Blinding
• High risk of bias 1 or more component inadequate
 Data recording
1 Schulz KF et al. JAMA 1995 ; 273 : 408 – 12.
2 Moher D et al. Lancet 1998 ; 352 : 609 – 13.

Antibiotic prophylactic in pancreatic necrosis
Flow diagram

Characteristics of RCTs included in MA
Bai Yu & al. Am J Gastroenterol 2008 ; 103 : 104 – 110.
467 patients included in 7 trials

 Data analysis
2 stage statistical process of MA
Statistical power of MA is often very high
• Treatment effect for each study
p value (p)
Relative Risk (RR) or Odds Ratio (OR)
Confidence Intervals (CIs)
Number Needed to Treat (NNT)
• Overall treatment effect
Calculated as weighted average of individual statistics

• p > 0.05 Statistically insignificant
• p < 0.05 Statistically significant
Probability value (p value)
Statistically
significant
Clinically
significant
Doesn't
mean

Risk & Odds
Risk
a
a + b
Risk =
RR: risk patients/risk controls
Odds
a
b
Odds =
OR: odds patients/odds controls

Interpretation of RR & OR
OR or RR should be accompanied by CI
RR or OR > 1
Increased likelihood of outcome in treatment group
RR or OR < 1
Decreased likelihood of outcome in treatment group
RR or OR = 1
No difference of outcome between tt & control group

Odds ratio or relative risk?
OR will be close to RR if endpoint occurs infrequently (<15%)
If outcome is more common, OR will differ increasingly from RR

Confidence intervals
Value 95 % CI are commonly used
90 or 99% CI are sometimes used
Width of CI Indicates precision of the estimate
Wider the interval, less the precision
CI includes 1 No statistically significant difference
CI doesn‟t include 1 Statistically significant difference

Statistical significance & CI
(a) Statistically significant , low precision
(b) Statistically significant, high precision
(c) Not statistically significant, low precision
(d) Not statistically significant, high precision
Glasziou P et al. Evidence based practice workbook. Blackwell, 2nd edition, 2007.

Number Needed to Treat (NNT)
• Relative risk (RR)
Risk in treatment group / risk in control group
• Absolute risk reduction (ARR)
Risk in control group – risk in treatment group
• NNT (expressed in clinically relevant way)
1 /ARR

Statistical methods/overall treatment effect
Larger trials have more influence than smaller ones
Fixed effects model 1
Random effects model 2
Bayesian models3 Controversial
Fixed & random effects
1 Prog Cardiovasc Dis 1985 ; 17 : 335 – 71.
2 Stat Med 1992 ; 11 : 141 – 58.
3 BMJ 1996 ; 313 : 603 – 7.
No single
correct method

 Data analysis
Bai Yu & al. Am J Gastroenterol 2008 ; 103 : 104 – 110.
• Treatment effect for each study
• Overall treatment effect
Random effects model only
Inherited heterogeneity between the studies
More conservative estimate of effect by using wider CIs
p value (p)
Relative risk (RR)
95% confidence intervals (CIs)

 Reporting the results
The typical graph for displaying results
of a meta-analysis is called a „„forest plot‟‟

Antibiotic prophylaxis & pancreatic necrosis
Forest plot

Horizontal line
Scale measuring the treatment effect

Vertical line or line of no effect
Treatment & control groups have the same effect

Point estimate & CIs for each study

Point estimate (RR or OR) & CI
Gallin JI, Ognibene FP. Principles & practice of clinical research.
A Press, 2nd ed, 2005.

Diamond

The diamond
Perera R, Heneghan C, Badenoch D. Statistics Toolkit.
Blackwell Publishing Ltd, Oxford, 1st edition, 2008.
Shows combined point estimate (OR or RR)
& CI for the meta-analysis

Diamond in meta-analysis
Diamond on Left of the line of no effect
Less episodes of outcome of interest in treatment group
Diamond on Right of the line of no effect
MoRe episodes of outcome in treatment group
Diamond touches the line of no effect
No statistically significant difference between groups
Diamond does not touch the line of no effect
Difference between two groups statistically significant

The diamond
Shows the overall result of MA

Antibiotic prophylactic effect on mortality
The diamond

Interpretation of forest plot
Names on left First authors of primary studies
Black squares RR or OR of individual studies
Black square size Weight of each trial in MA
Horizontal lines 95% confidence intervals
Vertical line Line of no effect (OR or RR = 1)
Diamond Overall treatment effect
Diamond Center Combined treatment effect
Tips of diamond 95% CI

Meta-analytic analyses are prone to bias
& need to be interpreted with caution
Bias: difference between study results & truth

Bias in meta-analysis (1)
• Publication bias: studies never published
Studies with no beneficial effect of treatment
Studies sponsored by pharmaceutical industry
Studies from a single centre versus multiple centers
• English language bias:
Positive findings published in a international journal
Negative findings published in a local journal
• Database bias:
Journals not indexed in major databases

Language bias
40 pairs of trials published by the same author
Controlled trials with statistically significant results was
higher among reports published in English
Egger M et all. Lancet 1997 ; 350 : 326 – 9.

Bias in meta-analysis (2)
• Multiple publication bias
Studies with significant results lead to multiple publications
• Bias in provision of data
Additional data not reported in print needed for MA
• Biased inclusion criteria
Selective inclusion of studies with positive findings
Exclusion of studies with negative findings

Explaining heterogeneity
In language of meta-analysis
- Homogeneity means results of each individual trial
are compatible with the results of any of the others
- Heterogeneity means results of each individual trial
are incompatible with results of any of the others

Do the pieces fit together?
Simon SD. Statistical evidence in medical trials: What do the data really tell us?
Oxford University Press, Oxford, 1st edition, 2006

How to measure heterogeneity in MA?
• Qualitative
Forest plot Visual evidence of heterogeneity
Funnel plot Visual evidence of heterogeneity
• Quantitative
X-squared
I-squared Based on Cochran‟s Q

Heterogeneity & forest plot
Hypothetical MA
Some trials with lower C.I. above upper C.I. of other trials
Some lines do not overlap
BIOS Scientific Publishers, 1st Edition, Oxford, 2001.

Funnel plots
Bias detected by simple graphical test
• Plot for each trial RR or OR on x axis
Sample size on y axis
• Absence of bias
Plot should resemble inverted funnel or Christmas tree
• Presence of bias
Plot shows asymmetrical & skewed shape

Ideal funnel plot
The smaller the trial, the larger the distribution of results
Cleophas TJ et all. Statistics applied to clinical trials.
Springer, The Netherlands , 3rd edition, 2006.

Cut Christmas tree
Cleophas TJ et all. Statistics applied to clinical trials.
Springer, The Netherlands , 3rd edition, 2006.
Negative trials not published (missing)
Suspicion of considerable publication bias in this MA

Funnel plot
Publication bias of antibiotics for infected necrosis

Funnel plot
Publication bias of trials of antibiotics for mortality

Quantitative measure of heterogeneity
Many prefer not to use quantitative measure
• X-squared:
Degree of freedom (df) Number of trials in MA – 1
X2 ≈ df No heterogeneity
X2 much greater than df Serious heterogeneity
• I-squared (0 – 100%)
< 25% No heterogeneity
50% – 75% Serious heterogeneity

Heterogeneity
X2 = 7.82 (df 6 – No heterogeneity)
I2 = 23.2% (No or little heterogeneity)

Antibiotic prophylactic effect on mortality
Heterogeneity
X2 = 4.66 (df 6 – No heterogeneity)
I2 = 0 % (No or little heterogeneity)

Appraising & applying meta-analysis
Heneghan C, Badenoch D. EBM toolkit. BMJ Books, London, 1st edition 2002.

Questions for appraising MA – 1
Critical Appraisal Skills Programme. Appraisal Tools. Oxford, UK.
http://www.phru.nhs.uk/casp/appraisa.htm (accessed 10 Dec 2004).
 Clearly focused question Focused question
 Identification of all relevant studies Good search
 Inclusion the right type of study Yes (RCTs)
 Assessment quality of all studies Yes but no blinding
 Reasonable to combine study results Yes (good X2 & I2)

Questions for appraising MA – 2
Critical Appraisal Skills Programme. Appraisal Tools. Oxford, UK.
http://www.phru.nhs.uk/casp/appraisa.htm (accessed 10 Dec 2004).
 Precision of the results No (wide 95% CI)
 Result presentation & main result RR (95% CI)
No difference
 Results applied to local population Mainly alcoholic
 Change practice as result of MA No?
All important outcomes considered Antibiotic SE?

 Assess
Limitations of this MA
• Timing of initiation of antibiotics
• Subgroup analysis Age
Etiology of pancreatitis
Presence of organ failure
• Wide 95% CI Infected necrosis 0.81 (0.54 –1.22)
Mortality 0.70 (0.42 – 1.17)
Further large scale better design RCTs are needed

* Altman DG et al. Ann Intern Med 2001 ; 134 : 663 - 94.
Improving quality of reports
Consolidated
Standards of
Reporting Trials
RCTs
CONSORT*
Diagnostic
accuracy study
STARD**
Standards for
Reporting of
Diagnostic Accuracy
*** Bossuyt PM et all. BMJ 2003; 326 : 41 – 44.
Quality of
Reporting of
Meta-analyses
Meta-analysis
QUOROM**
** Moher D et al. Lancet 1999 ; 354 : 1896 - 900.

QUOROM* statement
Targeted authors of MA rather than readers
• Experts 30 experts (epidemiologists, clinicians, editors,
statisticians, researchers)
• Date Oct 2–3, 1996 (Chicago – USA)
• Aim Improve quality of reporting MA & may be SR
• Results Flow diagram: progress through stages of MA
Checklist: 21 headings & subheadings
* Quorom: Quality of Reporting of Meta-analyses
Moher D et al. Lancet 1999 ; 354 : 1896 - 900.

The QUOROM checklist
Heading Subheading Descriptor Reported
Page No
Title Identify report as MA or SR of RCTs
Abstract
Objectives
Data sources
Review methods
Results
Conclusion
Use a structured format
Clinical question explicitly
Databases (list) & other information sources
Selection criteria, validity assessment, data synthesis
Characteristics of RCTs, point estimates, CI
Main results
Introduction Clinical problem, rationales for intervention & review
Methods Searching
Selection
Validity assessment
Data abstraction
Study characteristics
Data synthesis
Information sources in detail, precise restrictions
Inclusion & exclusion criteria
Criteria & process used (masked conditions, ..)
Process used (completed independently, in duplicate)
Study design, intervention, outcome & heterogeneity
Measures of effect (RR), method of combining results
(statistical testing & CI), missing data; statistical
heterogeneity, assessment of publication bias
Results Trial flow
Study characteristics
Data synthesis
Profile summarizing trial flow
Data for each trial (age, sample size, dose, follow-up)
Agreement, summary results, effect sizes & CI in ITT
Discussion Key findings, internal & external validity, biases, …

How much work is a meta-analysis?
• Analysis of 37 MA by Allen & Olkin of MetaWorks*
• Hours Average 1139 (216 – 2518)
• Breakdown 588 Protocol, searching, & retrieval
44 Statistical analysis
206 Report writing
201 Administration
• Total time depends on number of citations
* Company based in Massachusetts (USA) specializes in doing SR
Allen, I.E. Olkin, I. JAMA 1999; 282 : 634 – 5.

“Doing a meta-analysis is easy,
doing one well is hard”
Ingram Olkin

Importance of meta-analysis
• For some clinicians
MA is seen as exercises in "mega-silliness"
• For other clinicians
MA left no place for narrative review article
• The truth
Is likely to lie somewhere between these 2 extremes

References
Cambridge Press
2001
BMJ Publishing Group
2001
John Wiley & Sons
2009

Meta-Analysis

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Meta-Analysis

Similaire à Meta-Analysis (20)

Plus de Samir Haffar

Plus de Samir Haffar (20)

Dernier

Dernier (20)

Meta-Analysis

Notes de l'éditeur