Talk at StatIstanbul 2020 about prediction models. Apologies for the pretentious title.

1. Make
clinical prediction models
Great Again
Ben Van Calster
Department of Development and Regeneration, KU Leuven (B)
Department of Biomedical Data Sciences, LUMC (NL)
Research Ethics Committee, University Hospitals Leuven (B)
Epi-Centre, KU Leuven (B)
Istanbul, February 29th 2020

2. Contents
• What are we talking about?
• Developing models:
1. What do you want, when, and how?
2. Do not ignore information
3. Mind overfitting
• Externally validating models:
1. Calibration is essential
2. Expect heterogeneity
• What about machine learning?
2

3. What are we talking about?
3

4. 4
What are we talking about Development Validation Machine learning

5. To explain
• Study (strength of) independent associations with the outcome, e.g. to
find risk factors
5
What are we talking about Development Validation Machine learning
Kempenaers et al. Injury 2018;49;2269-75.

6. To predict
• Obtain a system that gives risk estimates of the outcome
• Aim is the use in NEW patients: it should work ‘tomorrow’, not now
6
What are we talking about Development Validation Machine learning
Edlinger et al. BMJ Open 2017;7;e014467.

7. To predict
7
What are we talking about Development Validation Machine learning
cvriskcalculator.com

8. Developing clinical risk
prediction models
8

9. 1. What do you want?
• What specific outcome should be predicted? Is there a clinical need?
• When during the clinical workflow should the prediction be made?
• Which predictors are available at that time point?
• What is the purpose? E.g. which treatment decision should it support?
• What is the quality of the data?
9Cronin & Vickers. Urology 2010;76:1298-1301
What are we talking about Development Validation Machine learning

10. Mistaking the objective…
10
Riley. Nature 2019;572:27-9.
What are we talking about Development Validation Machine learning

11. Example
11Hernandez-Suarez et al. JACC Cardiovasc Interv 2019;12:1328-38.
What are we talking about Development Validation Machine learning

12. Example (contd)
12
The model also uses postoperative information.
David J Cohen, MD: “The model can’t be run properly until you know about both the presence and the absence of
those complications, but you don’t know about the absence of a complication until the patient has left the hospital.”
https://www.tctmd.com/news/machine-learning-helps-predict-hospital-mortality-post-tavr-skepticism-abounds
What are we talking about Development Validation Machine learning

13. 2. Do not ignore information
a. Continuous variables should not be dichotomized, only decisions based
on a prediction model should!
13Butts & Ng. In Lance & Vandenberg. Routledge 2009.
What are we talking about Development Validation Machine learning

14. 2. Do not ignore information
b. Use available knowledge, do not always ask the data!
14Good & Hardin. Wiley 2006.
“Perhaps the most serious
source of error lies in letting
statistical procedures make
decisions for you”
“Don’t be too quick to turn on
the computer. Bypassing the
brain to compute by reflex is a
sure recipe for disaster”
What are we talking about Development Validation Machine learning

15. 15Rajkomar et al. Npj Digit Med 2018;1:18.
Will the hype of “machine learning” make us bypass
our brain once more?
What are we talking about Development Validation Machine learning

16. But what if you don’t know at all?
16Good & Hardin. Wiley 2006.
If you have no knowledge on what variables could be good
predictors (and what variables not), are you ready to make a
good prediction model?
What are we talking about Development Validation Machine learning

17. 3. Mind overfitting
You think of buying a Porsche.
But if you do not want to pay for it,
you may get this.
The same applies for developing risk models.
17
What are we talking about Development Validation Machine learning

18. Our currency is sample size
The more complicated (or ‘fancy’) the modeling strategy,
the more you have to pay with sample size.
Preferably good data (no counterfeit money!)
Match sample size to a sensible modeling strategy, or vice versa
Further recommendations to avoid overfitting:
- Avoid data driven variable selection where you can: you have to pay!
- Be careful with interactions: you have to pay (and often get little back)!
- Do not use train-test split: you’re burning your money!
18
What are we talking about Development Validation Machine learning

19. Flexible algorithms are data hungry
19http://www.portlandsports.com/hot-dog-eating-champ-kobayashi-hits-psu/
What are we talking about Development Validation Machine learning

20. Externally validating clinical
risk prediction models
20

21. 1. Calibration is essential
Key elements:
discrimination between patients with and without the event
calibration (correctness) of risk estimates
21
DISCRIMINATION
When it rained, was the
estimated chance of rain
higher (on average)?
CALIBRATION
For days with 80% estimated
chance of rain, did it rain on
8 out of 10 days?
What are we talking about Development Validation Machine learning

22. Assess calibration!
Management decisions are influenced by the magnitude of the estimated
risk of an outcome of interest. If this estimation is systematically off,
decisions are ill-informed.
22
What are we talking about Development Validation Machine learning

23. 2. Expect heterogeneity
23Cronin & Vickers. Urology 2010;76:1298-1301
What are we talking about Development Validation Machine learning

24. Performance will depend on location
Expect heterogeneity across hospitals, regions, countries
One external validation study does not tell you much about the model!
24Pennells et al. Am J Epidemiol 2014;179:621-632. Van Calster et al, submitted.
What are we talking about Development Validation Machine learning

25. Performance will depend on time
Care changes, populations change, so will model performance
25Davis et al. JAMIA 2017;24:1052-61.
What are we talking about Development Validation Machine learning

26. Model updating?
26Riley et al. BMJ 2016;353:i3140. Snell et al. J Clin Epidemiol 2016;69:40-50.
Every hospital its
own model that is
kept up-to-date:
Realistic or utopic?
What are we talking about Development Validation Machine learning
Before
After

27. What about
machine learning?
27

28. Reason for popularity
28
Claim:
“Typical machine learning algorithms are highly flexible,
so will uncover associations we could not find before,
And hence lead to better predictions and management decisions”
→ One of the master keys, with guaranteed success!
What are we talking about Development Validation Machine learning

29. Machine Learning: success guaranteed?
29Christodoulou et al. J Clin Epidemiol 2019;110:12-22.
What are we talking about Development Validation Machine learning

30. Traditional Statistics vs Machine Learning
30Christodoulou et al. J Clin Epidemiol 2019;110:12-22.
What are we talking about Development Validation Machine learning

31. Poor modeling and unclear reporting
31
What was done about missing data? 45% fully unclear, 100% poor or unclear
How were continuous predictors modeled? 20% unclear, 25% categorized
How were hyperparameters tuned? 66% unclear, 19% tuned with information
How was performance validated? 68% unclear or biased approach
Was calibration of risk estimates studied? 79% not at all, HL test common
Prognosis: time horizon often ignored completely
Christodoulou et al. J Clin Epidemiol 2019;110:12-22.
What are we talking about Development Validation Machine learning

32. Concerns for predictive analytics
32
 Poor study design and modeling strategy
 Do we need machine learning? Get design and methodology right first.
 Flexible algorithms and complicated modeling strategies are data hungry
 Large datasets often have poor quality
 There is large heterogeneity between settings and studies
 Populations change over time, using a model further changes it
 Reporting is often problematic
What are we talking about Development Validation Machine learning