This is an updated version of my Comparing Research Designs lecture, which now includes discussions on: (1) common considerations with research design such as bias, reliability, validity, and confounding; and (2) expanded discussion of RCT designs including factorial and cross-over designs.
2. On the Agenda
Important considerations in research design
Reliability & validity
Biases & confounding
Strength of evidence
Observational Research Designs
Cross-sectional study
Case-control study
Cohort study
Experimental Research Designs
The Basics of Factorial and CrossoverTrials
4. Reliability &Validity
Reliability Validity
Refers to the consistency of an
instrument/measurement.
Thought of as an individual’s “true
score” on the phenomenon you aim to
measure minus “measurement error”
Two common types of reliability
Internal consistency: Cronbach’s
alpha, KR20
Inter-Rater: Kappa statistic
Necessary but not sufficient in
determining validity.
Refers to the accuracy of an
instrument/measurement
In other words, “the degree to which
you’re measuring what you claim to
measure”
Two broad types of validity
Internal validity
External validity
5. Internal vs. ExternalValidity
One of the strengths of randomized designs are that they have
substantially higher internal & external validity.
InternalValidity: refers to the integrity of the experiment itself. It is
the ability to draw a causal link between your treatment and the
dependent variable of interest.
ExternalValidity: refers to the ability to generalize your study findings
to the population at large. In other words, are your findings from a
sample of UTMCK patients with HTN going to apply to all patients with
HTN?
6. Threats to InternalValidity
Concerns the accuracy of measurement within the
study
Shadish, Cook & Campbell (2002) summarized a
number of possible threats to internal validity, which
can severely jeopardize the findings of a study. In
particular:
History, Mortality, & Maturation
Repeated Testing
Confounding
Diffusion & Compensatory Rivalry
7. Threats to InternalValidity
Diffusion & Compensatory Rivalry
Diffusion: Treatment effects can “spill over” or “spread” across
treatment groups. EX: Patients from different groups live near each
other and discuss / share their experiences or treatments.
Compensatory Rivalry: Patients perform in a certain way because
they know they’re in the control / experimental groups.
8. Threats to InternalValidity
History, Mortality, & Maturation
History: events external to the experiment influence the participants’. EX:
Superstorm Sandy hits during a crossover trial in New Jersey.
Mortality: Patients either die (mortality) or drop out of the study (attrition) at
different rates.
Maturation: Patients change over the course of the treatment, which
influences results. EX: Children grow up during the course of a pediatric clinical
trial.
RepeatedTesting
Patients can become “test-wise” if given the same subjective test multiple
times, or they become conditioned to being tested (EX: patient’s pulse
increases before a needle stick).
9. ExternalValidity
The ability to generalize the findings of your study to the
relevant population.
Threatened by
Bias
Confounding
Non-experimental design (i.e. case-control vs. RCT)
Lack of randomization
External validity is the strongest when a true experimental
design is used.
10. Confounding
A confounder is a variable that is causally associated with
the outcome (DV) and may or may not be causally
associated with the exposure (IV)
Causes spurious conclusions & inferences to be made
about a set of variables
Reduced through
Randomization
Matching
Statistically controlling (covariates)
12. Bias in Research
The result of systematic
error in the design or
conduct of a study
Can artificially “trend”
results
Toward the Null hypothesis
Toward the Alternative hypothesis
A major problem to
consider when planning any
study
13. Common Biases
Selection bias: one relevant group in the population (e.g. cases
positive for predictor variable) has a higher probability of being
included in the sample
Misclassification can be either unsystematic (random) or
systematic (bad)
Information: bias from erroneously classifying people in
exposure/outcome categories
Recall/Response: bias associated with inaccurate recall of
exposure or representation of true exposure (self-report)
Experimenter/Interviewer bias: Differential treatment of
participants in treatment and control groups
Publication: the tendency to publish only “positive” or
“significant” findings.
14. Strength of Evidence
The Bradford Hill Criteria
Provides researchers with seven criteria for assessing
strength of evidence.
Strength of association (i.e. effect size)
Consistency (i.e. reliability)
Specificity
Temporal relationship
Biological gradient
Plausibility
Coherence
Experiment (reversibility)
Analogy (consideration of alternate explanations)
15. Pyramid of Clinical Evidence
RCT
Cohort
Studies
Case Control
Studies
Case Series
Case Reports
Ideas, Editorials, Opinions
Animal research
In vitro (‘test tube’) research
Systematic Reviews
& Meta-analyses
Evidence
Summaries
Level 2 Evidence
Level 1 Evidence
Level 3 Evidence
Cross-Sectional
Studies: Level 2.3
17. Cross-Sectional Studies
“Snapshot” of a population.
People are studied at a
“point” in time, without
follow-up.
Strength of evidence…
What are some research
questions that can be
answered with cross-
sectional designs?
18. Advantages and Disadvantages of Cross-
Sectional Studies
Advantages Disadvantages
Fast and inexpensive
No loss to follow-up
Springboard to
expand/inform research
question
Can target a larger sample
size
Can’t determine causal
relationship
Impractical for rare diseases
“Garbage in, garbage out”
Risk for nonresponse
19. Case-Control Studies
Always retrospective
Prevalence vs. Incidence
A sample with the disease from a population is selected
(cases).
A sample without the disease from a population is selected
(controls).
Groups are compared using possible predictors of the
disease state.
20. Advantages and Disadvantages of Case-
Control Studies
Advantages Disadvantages
High information yield
with few participants
Useful for rare outcomes
Cannot estimate incidence
of disease
Limited outcomes can be
studied
Highly susceptible to
biases
21. Strategies for Sampling Controls
Population versus hospital/clinic-based controls
Matching
Individual level
Group level
Using two or more control groups
22. Cohort Studies
A “cohort” is a group of individuals who are followed or
traced over a period of time.
A cohort study analyzes an exposure/disease relationship
within the entire cohort.
Groups selected based on exposure to a risk factor.
Level of evidence?
24. Prospective vs. Retrospective
Cohort Studies
Exposure Outcome
Prospective
Assessed at the
beginning of the
study (present)
Followed into
the future for
outcome
Retrospective
Assessed at some
point in the past
Outcome has
already occurred
25. Advantages and Disadvantages of
Cohort Studies
Advantages Disadvantages
Establish population-based incidence
Temporal relationship inferred
Time-to-event analysis possible
Used when randomization not possible
Reduces biases (selection, information)
Lengthy and costly
Not suitable for rare/long-latency
diseases
May require very large samples
Nonresponse, migration and loss-to-
follow-up
Sampling, ascertainment and observer
biases
27. Experimental Designs
What areThey?
Considered to be the “gold standard” of clinical evidence
because:
Randomization is used to reduce the effect of biases and
confounding variables
Patients (single) and researchers (double) can be blinded to
the intervention
High internal and external validity allow for assessing cause
and effect relationships.
The most basic experimental design is a “Parallel
trial.”
Patients are randomized into one of two groups, and
remain in the same group throughout the study.
“Double-blind trials”
28. Factorial Designs
What areThey?
Factorial designs allow for researchers to
test multiple interventions or treatment
combinations in a single study.
For example: drug A or Drug B and 3x per week or
everyday dose cycle.
The simplest form of this design is a 2x2
factorial design.
Allows researchers to test individual
treatment effects and/or interactions
between different treatments.
Looks like a “grid”
29. Factorial Designs
Why areThey Used?
Factorial design are commonly used to effectively test
multiple treatments or “Main effects” in a single study.
More efficient and more statistically powerful than multiple single intervention studies
Especially useful for testing interactions among different
interventions or treatments
Main Effects
Interactions
31. Cross-over Designs
What areThey?
A cross-over trial design involves giving the two or
more interventions/treatments to a single group of
patients.
At its most basic, this trial tests the efficacy of two
treatments where each patient spends a period of time
under both treatment options.
Patients are randomized into which treatment they
receive first, and then swap to the other treatment
after a predetermined time.
33. Cross-Over Designs
Why areThey Used?
Cross-over trials are useful because they reduce confounding
factors associated with between-subjects designs.
Patients serve as their own controls
Useful for time-dependent research questions
Higher statistical power than between subjects designs due to no between-subjects
error (i.e. need less patients to find statistical significance).
35. Disadvantages of RCT Designs
Extremely time and
resource demanding
Unethical in many
situations
Poor external validity if the
RCT is too highly controlled
Difficult to study rare
events
Therapeutic misconception
36. In Pairs…
Work together to brainstorm an example of how your topic
could be addressed using 1) a Cross-Sectional design, 2) a
case-control design, 3) a prospective or retrospective cohort
design, and an RCT (Parallel, factorial, or cross-over).
Be prepared to share your responses