Symposium Data-driven risk-based decisions; PhD defense Tom Hueting
Wednesday 22 juni 2022
Universiteit Twente
I present some issues on improving prediction models by updating and adding markers
1. June 22, 2022
Towards better performance of prediction models:
updating and extension with markers
: updating and marker
Ewout W. Steyerberg, PhD
Professor of Clinical Biostatistics and
Medical Decision Making
Dept of Biomedical Data Sciences
Leiden University Medical Center
Thanks to many, including Ben van Calster, Leuven
2. Key question: how to improve prediction models?
1. Better development + validation
a) Sample size
b) Methods
2. Updating of existing models
a) Local settings
b) Continuous learning
3. Extension with markers
4. Machine learning
22-Jun-22
2 Insert > Header & footer
3. Subquestion 1: How to assess model improvement?
1. Calibration (A + B, intercept + slope)
2. Discrimination (C, concordance)
3. Clinical usefulness (D, decision-analytic)
22-Jun-22
3 Insert > Header & footer
8. Decision-analytic perspectives
If we are serious about “using different thresholds that allow the
operator of the model to trade-off concerns in the errors made by
the model” we need a decision-analytic perspective
1. Define threshold
2. Evaluate quality of classification
Decision Curve
22-Jun-22
8 Insert > Header & footer
11. 3 statements on Decision Curve Analysis (DCA)
1. A classic idea (1884 or older)
2. A good link with clinical context:
benefit of treatment vs harm by overtreatment to define thresholds
3. A good graphic because thresholds are ‘subjective’
22-Jun-22
11 Insert > Header & footer
12. Youden index and Net Benefit; Peirce, Science 1884
Event
Test: answer + –
+ aa ab
– ba bb
TP FP
sens spec
Youden index: sens + spec – 1
13. Vickers & Elkin, MDM 2006
22-Jun-22
13 Insert > Header & footer
Benefit: a – c
Harm: d – b
Odds of threshold:
Harm / Benefit
14. Net Benefit
Net Benefit = (TP – w FP)/N
w = harm / benefit ratio = threshold/ (1 – threshold)
• e.g.: threshold 50%: w = .5/.5=1;
threshold 20%: w=.2/.8=1/4
“Fraction of true-positive classifications,
penalized for false-positive classifications”
BMJ 2016;352:i6 doi: 10.1136/bmj.i6.
18. Key question: how to improve prediction models?
1. Better development + validation
a) Sample size
b) Methods
2. Updating of existing models
a) Local settings
b) Continuous learning
3. Extension with markers
22-Jun-22
18 Insert > Header & footer
19. Subquestion 2: How to balance global vs local models?
Prediction models need updating to local settings;
can we entertain the idea of a ‘global model’?
1. Global: baseline risk + predictor effects
2. Recalibrated: local baseline risk + global predictor effects
3. Refitted: local baseline risk + local predictor effects
22-Jun-22
19 Insert > Header & footer
20. Examples on updating
Single validation set
Robust approach + closed testing
22-Jun-22
20 Insert > Header & footer
22. Examples on updating
Single validation set
Classic: approach SiM 2004; closed testing
Dynamic in calendar time
Multiple validation sets
Assess heterogeneity
a) Global model?
b) Fair representation of uncertainty?
22-Jun-22
22 Insert > Header & footer
25. Key question: how to improve prediction models?
1. Better development + validation
a) Sample size
b) Methods
2. Updating of existing models
a) Local settings
b) Continuous learning
3. Extension with markers
22-Jun-22
25 Insert > Header & footer
27. Incremental value of marker
• Define a reference model, add marker to evaluate incremental value
• Regression coefficient problematic (scaling); p-value assumed to be low
• Increase in AUC / c statistic usually small (typically: +0.01)
Push to look beyond AUC: reclassification
22-Jun-22
27 Insert > Header & footer
34. Marker evaluation
NRI was a historical mistake?
Net benefit to the rescue?
22-Jun-22
34 Insert > Header & footer
35. Summary 20 June 2022
1. Prediction modeling research challenging
2. Performance assessment: calibration and Net Benefit
3. Improving performance:
a) Updating
b) Markers
c) Machine learning
22-Jun-22
35 Insert > Header & footer
