A Critique of Cochrane Collaboration. Practical Considerations for Revision of Systematic Review

1. Ninian Peckitt
FRCS FFD RCS FDS RCS FACCS
Oral and Maxillofacial Surgeon / Facial Plastic Surgeon
Adjunct Associate Professor of Engineering Assisted Surgery
Massey University, New Zealand

2. Cochrane Collaboration
• group >28,000 volunteers
• >100 countries
• Review health care interventions tested in biomedical
randomized controlled trials.[3]
• + more non-randomized, observational studies.
• systematic reviews published as "Cochrane Reviews"
• in the Cochrane Library.

3. Goals and Prinicples
The goal of the collaboration is to help people make well informed
decisions about health care by preparing, maintaining and ensuring the
accessibility of systematic reviews of the effects of health care
interventions. The principles of the Cochrane Collaboration are:
• collaboration
• building on the enthusiasm of individuals
• avoiding duplication
• minimizing bias
• keeping up to date
• striving for relevance
• promoting access
• ensuring quality
• continuity
• enabling wide participation

4. Forest Plot - (Blobbogram)
• Common - 2 columns
• The left-hand column lists:
– the names of the studies
(e.g. RCTs / epidemiol studies)
– chronological order
– from the top downwards.

5. Forest Plot - (Blobbogram)
• The right-hand column
• a plot of the measure of effect
– e.g. an odds ratio for each of these studies
– (often represented by a square)
– incorporating confidence intervals
– represented by horizontal lines.
• The graph may be plotted on a natural logarithmic
scale when using odds ratios or other ratio-based
effect measures
• confidence intervals are symmetrical about the means
from each study
• ensures undue emphasis is not given to odds ratios
greater than 1 when compared to those less than 1

6. Forest Plot

7. Meta Analysis
• combines the results of several studies
• that address a set of related research hypotheses
• In its simplest form, this is normally by identification of a common measure of
effect size,
• for which a weighted average might be the output of a meta-analyses.
• Here the weighting might be related to sample sizes within the individual studies.
• More generally there are other differences between the studies that need to be
allowed for, but the general aim of a meta-analysis is to more powerfully estimate
the true "effect size" as opposed to a smaller "effect size" derived in a single study
under a given single set of assumptions and conditions.

8. Advantages of Meta Analysis
• Shows if the results are more varied than expected from sample diversity
• Derivation and statistical testing of overall factors / effect size parameters in related studies
• Generalization to the population of studies
• Ability to control for between-study variation
• Including moderators to explain variation
• Higher statistical power to detect an effect than in ‘n=1 sized study sample’
• Deal with information overload: the high number of articles published each year.
• combines several studies less influenced by local findings than single studies will be.
• May show if a publication bias exists.

9. Steps in Meta Analysis
1. Formulation of the problem
2. Search of literature
3. Selection of studies (‘incorporation criteria’)
– Based on quality criteria, e.g. the requirement of randomization and blinding in a clinical trial
– Selection of specific studies on a well-specified subject, e.g. the treatment of breast cancer.
– Decide whether unpublished studies are included to avoid publication bias
4. Decide which dependent variables or summary measures are allowed.
– Differences (discrete data)
– Means (continuous data)

10. Steps in Meta Analysis
• Hedges' g
– is a popular summary measure for continuous data
– that is standardized in order to eliminate scale differences, but
incorporates an index of variation between groups:
μt is the treatment mean, μc is the control mean, σ2 the pooled
variance
• For reporting guidelines, see QUOROM statement [6][7]

11. Metaregression Models
5. Model selection
• simple regression
• fixed effect meta-regression
• random effects meta-regression

12. Simple Regression
Where yj is the effect size in study j and
β0 (intercept) the estimated overall effect size
The variables specify
• different characteristics of the study,
• specifies the between study variation
• Note that this model does not allow specification of within study variation.

13. Fixed Effect Meta-regression
• assumes that the true effect size θ
• is normally distributed - within study variance of the effect size
• allows for within study variability
• but no between study variability
• because all studies have the identical expected fixed effect size θ,
• i.e.***Note that for the "fixed-effect" no plural is used (in contrast
to "random-effects") as only ONE true effect across all datasets is
assumed.***

14. Fixed Effect Meta-regression
• variance of the effect size in study j
• Fixed effect meta-regression ignores between study variation
• parameter estimates biased if between study variation can not be ignored
• Generalizations to the population are not possible

15. Random effects meta-regression
• assumption that θ in is a random variable
• following a (hyper-)distribution
• Here is the variance of the effect size in study j.
• Between study variance is estimated using common
estimation procedures for random effects models
(restricted maximum likelihood (REML) estimators).

16. Applications in Modern Science
• Modern statistical meta-analysis
• does more than just combine the effect sizes of a set of studies.
• test if studies show more variation than expected (e.g. sampling different research participants)
• study characteristics such as :
– Measurement
– instrument used
– population sampled
– or aspects of the studies' design are coded.
• These characteristics are then used as predictor variables :
– to analyze the excess variation in the effect sizes
– Studiesweaknesses can be corrected statistically (e.g. bias in size or codings)

17. Applications in Modern Science
• Meta-analysis can be done with
– single-subject design
– group research designs
• much research on low incidents populations
– single-subject research designs
– Considerable dispute exists for the most appropriate
meta-analytic technique for single subject research.[8]

18. Single Subject Design
Continuous assessment:
• Individual behaviour observed repeatedly over the intervention.
• Insures Rx effects observed long enough to convince that Rx produces a lasting effect.
Baseline assessment:
• Before Rx is implemented, look for behavioral trends.
• If Rx reverses a baseline trend (e.g., getting worse [baseline ]vx [Rx] reversed this trend)
• powerful evidence suggesting (though not proving) a treatment effect.
Variability in data:
• Ability to observe how Rx changes behavior from day-to-day
• Large-group statistical designs do not typically provide this information
• because repeated assessments are not usually not taken
• and the behavior of individuals in the groups are not scrutinized - group means are reported

19. Phases within Single Subject Design
Phases within single-subject design
• Baseline:
– data on the dependent variable without any intervention in place
• Intervention:
– introduce independent variable (the intervention)
– collect data on dependent variable
• Reversal:
– removes the independent variable (reversal)
– collects data on the dependent variable
• Data must be Stable (steady trend / low variability) before move to the next phase
• Single-subject designs produce or approximate three levels of knowledge:
– (1) descriptive
– (2) correlational
– (3) causal[4]

20. Flexibility of Single Subject Design
• highly flexible
• highlight individual differences
• in response to intervention effects[5]
• In general
– reduce interpretation bias
– for counselors when doing therapy[6]

21. Data Interpretation
Independent variable on the dependent variable,
• graph the data collected / visually inspect the differences between phases
• If there is a clear distinction between baseline and intervention, and then the data returns to the
same trends/level during reversal, a functional relation between the variables is inferred.[7]
• Sometimes, visual inspection of the data demonstrates results that statistical tests fail to find[8][9]
• Begin with Graphic analysis
– During the baseline, data are repeatedly collected and then graphed on the behavior of interest.
– Visual representation of the subject’s behavior before application of the intervention
– Several (e.g. 3-5 ) baseline data points for description of effects on the target behavior during intervention
• Subject as their own control
• Baseline behavior would match its behavior in the intervention phase unless the intervention does
something to change it.
• This logic then holds to rule out confound

22. Goals
• Prediction
• Verification
• Replication

23. Meta-analysis of single subject research
• Meta-analysis, like all research, has the ability to change a profession.
• functional analysis < effect sizes than contingency management
• Debate meta-analysis of single-subject designs.
• The two choices being debated are
– the percentage nonoverlapping data(PND) vs.
– data points exceeding the median(PEM) method.[13][14][15]
– Noorgate and colleagues have argued that meta-analyses that analyze all
linear trends in data don’t work since they don’t distinguish between effects
on level and slope.[14][16]

24. Limitations Single Subject Designs
• Preplanning of Designs often made as the data are collected.[17]
• No widely agreed upon rules for: altering phases / conflicting ideas / conduct
• Major criticisms of single-subject designs are:
– Carry-over effects: results from the previous phase carry-over into the next phase,
– Order effects: the ordering (sequence) of the intervention / treatment affects what results
– Irreversibility:
• once a change in the independent variable occurs, the dependent variable is affected
• This cannot be undone by simply removing the independent variable.
– Ethical problems:
• Withdrawal of treatment - ethical and feasibility problems

25. Statistical Significance
• unlikely to have occurred by chance
• Stat hypothesis tests that answer the question
– Assuming that the null hypothesis is true
– what is the probability of a value as extreme as
the value observed?

28. • Systematic Review: takes 23 months from protocol to publication
• Timespan: Hundreds / Thousands of hours.
• Problems: Production and Updating of review
• <40% data is up-to-date
Out of Date Systematic Reviews
• 2009 - 2,383
• 2012 - 3,149
• Terrible figures made worse by increase in
spending on Cochrane
http://blog.tripdatabase.com/2013/04/a-critique-of-cochrane-collaboration.html#sthash.vioz56f0.dpuf

29. Funding Cochrane
• £100 million over last 7 years
• >£150 million – $250 billion US Dollars over 20 years of its
existence
http://blog.tripdatabase.com/2013/04/a-critique-of-cochrane-collaboration.html#sthash.vioz56f0.dpuf

30. • Current system is unsustainable
• Current System is not fit for purpose.
• The methodology
• has reduced some bias,
• but resulted in a huge financial cost increase
• a huge cost in opportunity
• The Tamiflu fiasco highlights a flawed methodology
http://blog.tripdatabase.com/2013/04/a-critique-of-cochrane-collaboration.html#sthash.vioz56f0.dpuf

31. Tamiflu - Tom Jefferson Review Leader stated:
“…I personally believe and my colleagues believe with me that Cochrane
Reviews based on publications should really be a thing of the past…”
Cochrane systematic review relied on published journal articles
• Large amounts of data were missed, most of which was made available
for the regulatory agencies e.g. EMA, FDA.
• The updated, 2012, review was a huge undertaking
• Jack Cuzick - Evidence Live 2013 made a general call for reviews to be based
on individual patient data (IPD)
http://blog.tripdatabase.com/2013/04/a-critique-of-cochrane-collaboration.html#sthash.vioz56f0.dpuf

32. Peer opinion Cochrane Methodology
• Incapable of making an accurate assessment of an interventions ‘worth’
• The seriousness of this challenge should not be underestimated,
• This challenge attacks at the very heart of the Cochrane Collaboration.
http://blog.tripdatabase.com/2013/04/a-critique-of-cochrane-collaboration.html#sthash.vioz56f0.dpuf

33. Doing things more quickly can give you the same or similar results to the
Cochrane methodology.
1) Can we rely on the best trial?
A comparison of individual trials and systematic reviews
• Random sample of Cochrane systematic reviews
• Was largest RCT in agreement with the subsequent meta-analysis?
• Yes - 81% of the meta-analyses examined and if the largest RCT was
positive and significant it was around 95%.
• In other words, using the largest RCT can give a broad hint as to the likely
result of a subsequent meta-analysis
Is Systematic Review Really Required?
http://blog.tripdatabase.com/2013/04/a-critique-of-cochrane-collaboration.html#sthash.vioz56f0.dpuf

34. 2) McMaster Premium LiteratUre Service (PLUS) performed well for identifying
new studies for updated Cochrane reviews.
• Authors compared the performance of McMaster Premium LiteratUre Service
(PLUS) and Clinical Queries (CQs) to that of the Cochrane Controlled Trials
Register, MEDLINE, and EMBASE for locating studies added during an update of
reviews.
• They concluded that PLUS included less than a quarter of the new studies in
Cochrane updates, but most reviews appeared unaffected by the omission of
these studies.
• In other words, you do not necessarily need to get all articles to arrive at an
accurate effect size (compared to the Cochrane systematic review).
Is Systematic Review Really Required?
http://blog.tripdatabase.com/2013/04/a-critique-of-cochrane-collaboration.html#sthash.vioz56f0.dpuf

35. Is Systematic Review Really Required?
3) A pragmatic strategy for the review of clinical evidence.
• Authors compared a research strategy based on the review of a selected number
of core journals, with that derived by an SR in estimating the efficacy of
treatments.
• Conclusion: “We verified in a sample of SRs that the conclusion of a research
strategy based on a pre-defined set of general and specialist medical journals is
able to replicate almost all the clinical recommendations of a formal SR.
http://blog.tripdatabase.com/2013/04/a-critique-of-cochrane-collaboration.html#sthash.vioz56f0.dpuf

36. The Future
• Reduce the cost per review.
• Recognise laws of diminishing returns
• Major challenge to do a modification of a systematic review in a month (or less)
 rapid systematic reviews
 A more detailed/costly sytematic review including regulatory data and/or IPD.
• Reduce cost of Review by 90%
Is Systematic Review Really Required?
http://blog.tripdatabase.com/2013/04/a-critique-of-cochrane-collaboration.html#sthash.vioz56f0.dpuf

37. But might we get the wrong answer with streamlining?
Yes !
But detailed Cochrane systematic reviews also have given us the wrong answer !
Is Systematic Review Really Required?

38. References
1. The Cochrane Oversight Committee. Measuring the performance of The Cochrane Library. 2012
2. Allen IE, Olkin I. Estimating time to conduct a meta-analysis from number of citations retrieved. JAMA. 1999 Aug
18;282(7):634-5.
3. Cochrane Collaboration Annual Report & Financial Statements 2010/11
4. Payne D. Tamiflu: the battle for secret drug data. BMJ 2012;345:e7303
5. HAI Europe - Dr. Tom Jefferson on lack of access to Tamiflu clinical trials
6. Jefferson TO, Demicheli V, Di Pietrantonj C, Jones M, Rivetti D. Neuraminidase inhibitors for preventing and treating influenza
in healthy adults. Cochrane Database Syst Rev. 2006 Jul 19;(3):CD001265
7. Jefferson T, Jones MA, Doshi P, Del Mar CB, Heneghan CJ, Hama R, Thompson MJ. Neuraminidase inhibitors for preventing
and treating influenza in healthy adults and children. Cochrane Database Syst Rev. 2012 Jan 18;1:CD008965. doi:
10.1002/14651858.CD008965.pub3
8. Glasziou PP, Shepperd S, Brassey J. Can we rely on the best trial? A comparison of individual trials and systematic reviews.
BMC Med Res Methodol. 2010 Mar 18;10:23. doi: 10.1186/1471-2288-10-23
9. Hemens BJ, Haynes RB. McMaster Premium LiteratUre Service (PLUS) performed well for identifying new studies for updated
Cochrane reviews. J Clin Epidemiol. 2012 Jan;65(1):62-72.e1
10.Sagliocca L, De Masi S, Ferrigno L, Mele A, Traversa G. A pragmatic strategy for the review of clinical evidence. J Eval Clin
Pract. 2013 Jan 15. doi: 10.1111/jep.1202
11.Greenhalgh T. Why do we always end up here? Evidence-based medicine's conceptual cul-de-sacs and some off-road
alternative routes. J Prim Health Care. 2012 Jun 1;4(2):92-7.J Prim Health Care. 2012 Jun 1;4(2):92-7.