This document discusses epistemic problems in critical care medicine research. It notes that for interventions with less obvious or immediate effects, evidence from clinical trials is important due to uncertainty. However, many such trials in critical care have problems with non-repeatable positive results and inadequate study power to detect realistic treatment effects. This leads to a situation where the null hypothesis of no treatment effect has stochastic dominance. The document argues that commonly used research practices like setting a low threshold for statistical significance, aiming for high study power, and accurately estimating expected treatment effects are often not followed, undermining the strength of evidence even from large trials.

1. There Is (NO) Evidence
For That:
Epistemic Problems in Critical Care
Medicine
SCOTT K. ABEREGG, MD, MPH
SALT LAKE CITY, UTAH
WWW.MEDICALEVIDENCEBLOG.COM
WWW.STATUSIATROGENICUS.BLOGSPOT.COM

2. What Constitutes Knowledge (A
Justified True Belief)?
From Therapeutic Agnosticism: Stochastic Dominance of the Null Hypothesis
Category 1:
 ARR high [NNT low]
 “Visible” & immediate effects
 Causal Pathways “Obvious”
 Type I diabetes  DKA
 Insulin  Resolution of DKA
 Trials “unethical” – No Equipoise –
High Prior Probability for Ha
Category 2:
 ARR low(er) [NNT high(er)]
 “Invisible” & delayed effects
 Associations Prevalent, CPs Obscure
 ICU Hyperglycemia  ???
 Insulin  resolution of hyperglycemia
 Trials imperative – Equipoise - Low(er) Prior
Probability for Ha

3. Examples:
Category 1: No Evidence?
 Parachutes for Gravitational Challenge
 Oxygen for severe hypoxia
 Mechanical Ventilation
 Antibiotics for sepsis
 IVF for dehydration
 Insulin in DKA
 Knee Replacement
Category 2: Evidence?
 Efficacy of Parachute A versus Parachute B
 Normocapnia; Heliox
 Low tidal volume ventilation
 Duration of antibiotic therapy
 “Goal Directed Therapy”, fluids for sepsis
 Anytensive insulin therapy in CCM
 Arthroscopy scams and shams

4. There Is Evidence for…

5. Two Problems in Category 2 Therapies
in Critical Care
Non-repeatability of Positives
 “Journal Club Biases”
 Conflict of Interest
 Single Center (Crabbe Effect)
 Early Stopping (benefit/futility)
 Multiple Comparisons
 Regression to mean/Decline effect
 Lack of blinding
 Publication bias
 “Flexibility”
Nothing Works in CCM
 Stochastic Dominance of the Null
Hypothesis
 Inadequate Study Power/Delta Inflation

6. Stochastic Dominance of the Null
Hypothesis: The ARDSnet Population of Studies
 KARMA, n=234, standard , β, δ
 Stopped for futility n=234, δ(observed) =
1.0%, p=.85
 ARMA, n=861, standard , β, δ
 Stopped for efficacy, n=861, δ = 8.8%,
p=0.007
 LaSRS, n=180, standard , β; δ 15%,
revised mid-study to 20% because of
low enrollment
 Observed δ 0.6%, P=1.0
 FACTT, n=1000, standard , β, δ
 Observed δ 2.9%, p=0.30
 ALVEOLI, n=549, standard , β, δ
 Stopped early; Observed δ 2.6, p=0.48
 ARDSnet II
 ALTA
 EDEN
 OMEGA
 SAILS
 This “Population” of hypotheses
is dominated by Ho; Ho not
rejected 90% of the time

7. Prior probability of Ho = 1-
Ha
Held, BMC Medical
Research Methodology 2010,
10:21
“A nomogram for p-values”
http://www.biomedcentral.co
m/1471-2288/10/21
Minimum posterior
probability of Ho; 1-Ho =
Maximum posterior
probability of Ha

8. Inadequate Study Power

9. What Ought to be
 Type I error rate selected –
significance threshold - (alpha)
 Type II error rate selected – Study
power – 1-β
 Estimate of baseline event rate in
control group
 Estimate of treatment effect size – δ
(delta)
What is
 0.05 by convention (Statistical
Methods for Research Workers; Fisher,
1925)
 Usually 80 +/- 10% - dual significance
hypothesis testing
 Prior data consulted for baseline
event rate estimate
 Estimate made of how many patients it
is feasible to enroll. Delta back-
extrapolated from this number –
power calculation in reverse. 10%
generally used

10. Delta Inflation is to Research as Grade
Inflation is to Academics

11. NICE SUGAR36
Van Den Berghe9
Bernard11 Brower6
Fagon4
Esteban18
Rivers10
Ronco5
Schiffl13
Predicted Delta (%)

12. Conclusions
 Knowledge has many forms and heirarchies of knowledge are artificial and
epistemically dubious (Sorry, User’s Guides.)
 We face a dual problem of methodologically inadequate positive studies
and the “intractability of mortality” in adequate studies
 Ho has stochastic dominance in critical care research
 Bayesian analysis reveals that the “strength” of our evidence is much less
than we presume
 The solution to inadequate power, double significance hypothesis testing,
has been subverted by delta inflation (both premeditated and
subconscious)
 Saying “There is no evidence for that” is meaningless
 Saying “There is evidence for XYZ” is equally meaningless