3. Methodology Reviews
• A "methodology study" is a study of the
methods used in randomized trials, other
healthcare evaluations or systematic reviews.
– Consent procedures
– Recruitment methods
– Association of allocation concealment with
estimates of treatment effect
• A "methodology review" is a systematic
review of methodology studies
4. Study Designs for CER/ PCOR
•head to head randomized trials, • observational study analysis
•cluster randomized trials, approaches employing so-called
•adaptive designs, causal inference techniques,
which can include instrumental
•practice / pragmatic trials, variables, marginal structural
• PBE-CPI “practice based evidence models, propensity scores,
for clinical practice improvement,” among others.
•natural experiments,
•observational or cross-sectional • “NEW” design terms such as
studies of registries and databases “observational randomization
including electronic medical records, study”
•meta-analysis,
•network meta-analysis,
•modelling and simulation
Source: IOM-National Priorities Committee 2009
4
5. “Best “ study design for CER?
•A number of reviews comparing the effect sizes and biases in
randomized and non-randomized studies have been conducted.
•Most compared randomized to non-randomized trials.
•Most often limited the comparison with observational
studies to cohort studies, or the types of observational
designs included were not specified.
•Most published between 1982 and 2003
•Compared to RCTs, observational designs have been found to
overestimate treatment effects ; underestimate treatment
effects; or show no difference.
6. Protocol: in press, Cochrane Library
Health care outcomes assessed with non-
experimental designs compared with
those assessed in randomized trials
Lisa Bero
Andrew Anglemyer
Tara Horvath
San Francisco Branch of the US Cochrane Center
HIV / AIDS Cochrane Review Group
UCSF
Funding: Clinical and Translational Sciences Institute (CTSI), University of California,
San Francisco (UCSF), USA
7. Objectives
• To assess the impact of study design--to include
RCTs vs observational study designs, different
types of observational studies, and/or choice of
analytic techniques -- on the effect measures
estimated in observational and randomized
studies
• To explore methodological variables that might
explain any differences identified
• To identify gaps in the existing research
comparing study designs
8. Inclusion criteria
• Systematic or non-systematic reviews
designed as methodological studies to
compare study designs
• Clinical outcomes: efficacy or harms of
alternative interventions to prevent or treat a
clinical condition or improve the delivery of
care
9. A priori subgroup analyses
• Comparisons of drug interventions
• Clinical topic
• Heterogeneity of included methodological
studies
11. PRELIMINARY DATA:
9 studies in meta-analysis
• Included 19 – 276 studies
• Evaluated a mix of interventions
– Lower back pain
– Digestive surgery
– Various interventions
• One focused on drug –drug comparisons
(Naudet)
• One focused on adverse events from (mostly)
pharmacological treatements (Golder)
13. Preliminary Findings
• Differences in effect measures not associated
with study design – explore other reasons
• Conduct subgroup analyses
• Need methodological studies comparing trials
with other observational designs (not just
cohorts, case control) and different analytic
methods for observational data