This presentation was a Plenary Talk given during the PMED Program Transition Workshop

1. Some Recent Advances in Machine Learning,
Artificial Intelligence,
and Precision Medicine
May 21, 2019
Michael R. Kosorok, PhD
Example image
Time course plot
of glucose,
Insulin, physical
activity, and
food intake for a
single patient
(data from
Maahs et al,
2012)

2. Outline
Overview of precision medicine
Sketch of machine learning
Recent developments on the interface
Overview of artificial intelligence (AI)
Example: T1D safe exercise strategies
Looking to the future
Concluding thoughts

3. Precision Medicine and Precision Health
Precision Medicine
Data-driven decision support for treating patients in the presence of
heterogeneity (dynamic treatment regimes or DTRs)
Treatment can include drug choice, administrative actions, dosing,
timing, potentially modifiable risk factors, and/or other potentially
beneficial actions to the patients
Must be reproducible and generalizable (empirically and inferentially
valid)
Precision Public Health
Data-driven decision support for families, communities,
clinics, social networks, and other entities
Potential treatments can also be policies
Precision Health
Union of precision medicine and precision health
We always consider the consequences of our actions on the
populations involved not just narrow subgroups or individuals

4. Operating Principles
Observable Constituents:
Tailoring variables (X)
Choice of treatments and/or potentially modifiable risk factors (A)
Vector of outcomes or utilities (Y)
Could be multiple (X,A,Y) triples over time for each patient
Dynamic Treatment Regime (DTR):
Single decision: make a single recommendation for treatment
Multiple decision: make a series of interdependent recommendations
Continual monitoring: for diabetes, mHealth
Role of Heterogeneity in the data:
Heterogeneity of patients is beneficial (essential) for good precision medicine
analysis so that our treatment rules are broadly applicable
Potential treatments can also be policies or risk factor modifications
Need heterogeneity of treatment assignment (either
naturally or by design) in the data so we can determine
best treatment under a variety of situations

5. Overall Pipeline Details
Dynamic Treatment Regime:
d(X) gives recommended A to maximize Y in future patients
Regression: model Y as a function of X and A (Q(X,A)=E[Y|X,A] is the “value”),
with interaction between X and A being most important
Policy estimation: directly estimate d(X) without needing Q(X,A) (e.g., outcome
weighted learning)
Prediction versus prescriptive decision support:
Suppose Y=f(X)+Ag(X)+e, where bigger Y is better and A={0 or 1}
We only care about g(x), since rule d(X)={1 if g(X)>0, 0 otherwise} yields optimal
A focus on prediction spends too much energy and focus on f(X) instead of g(X)
Propensity Score:
P(A|X) is propensity score
We estimate P(A|X) from data and make sure it is positive for all X
Causal Methods:
Potential outcome validity (Y(0) and Y(1) are well behaved)
No unmeasured confounders: Y(0) and Y(1) are independent given X
Positivity assumption (P(A|X)>0 for all X)

6. Machine Learning
Methods of statistical analysis
Possible tasks:
Unsupervised learning (e.g., clustering)
Supervised learning (e.g., classification,
regression, data-drive decision making)
Partly supervised, and more
Statistical aspects:
Will the method provide results
reflecting the real world (reproducible)?
Discovery vs. validation/confirmation

7. Machine Learning (cont.)
Additional tasks/tools:
Reinforcement learning
On-policy versus off-policy
Deep learning
Causal inference
Facilitate precision medicine:
Discover dynamic treatment regimes (DTRs)
to be described below
Evaluate and validate discoveries (statistical)
Implement in practice
AlphaGo AI becomes world champion

8. DTR Estimation
Single-decision setting:
Regression and maximization
Frequentist and Bayesian tools can be used (both
good, but some non-Bayesian is better when data is
high dimensional)
Machine learning is very helpful
Multi-decision setting:
Need reinforcement learning
Q-learning is especially useful
Q-learning involves a sequence of standard
regressions: random forests are good here
There is software for this (in R mostly)

9. Some Recent Developments
Outcome weighted learning (OWL):
Focus on estimating policy rather than entire process
OWL: convert to classification problem of predicting
A from X weighted by Y (avoids regression model)
Benefits of OWL:
Robust to regression model specification
Converts problem to a classification task which can
employ machine learning (such as SVM)
Performs very well in many situations
Many extensions

10. Recent Developments, cont.
Extensions of OWL:
Time-to-event outcomes with censoring
Continuous treatment (such as dose)
Interpretable DTRs (e.g., list based)
Multi-decision setting
Finite horizon
Infinite (indefinite) horizon (V-learning)
Other developments:
Managing multiple competing outcomes
Inverse reinforcement learning for precision medicine
Adaptive study designs for precision medicine which integrate with
machine learning (e.g. Thompson sampling)
∞

11. Recent Developments in ML, cont.
Counterfactual Gaussian processes:
Schulam P and Saria S (2017). Reliable decision support using
counterfactual models. Neural Information Processing Systems
(NIPS).
Potential outcome-based prediction of treatment effects
Outcome Weighted Deep Learning:
Liang M, Ye T, and Fu H (2018). Estimating individualized
optimal combination therapies through outcome weighted deep
learning algorithms. Statistics in Medicine 37:3869-3886.
Powerfully flexible single-decision setting discovery
Off-Policy Deep Reinforcement Learning:
Fujimoto S, Meger D, and Precup D (2018). Off-policy
deep reinforcement learning without exploration.
arXiv:1812.02900v2

12. Before answering this question, we should agree on what real intelligence is
• To think or act like people?
– The Turing test (1950)
– Spelling errors
– Taking time to solve math
• To perform tasks people
can perform?
• To reason logically?
• To be able to make on
optimal decision?
• Learning how to act
optimally?
• To reason morally?
• To create something
surprising?
What is intelligence?
• Make rational decision to
achieve pre-defined goals
• Goals are usually related to
optimizing a predefined utility
• Can a computer decide what
utility it should use?
• What can we learn about this
by studying the human brain?
– Hard to reverse engineer the
brain
– Memory and inference are
key to good decision making
What do we want
computers to do?
What is Artificial Intelligence
Relative to ML?

13. How did we get to where we are now?
• 1940-1950
– McCulloch & Pitts: Boolean circuit
(1943)
– The Turing test (1950)
• 1950-1970
– 1950s: early successes (playing
checkers), Logic Theorist
– Dartmouth meeting “Artificial
Intelligence” coined (1956)
– Robinson’s algorithm for logical
reasoning (rule based)
• 1970-1990
– Early knowledge-based systems
(1969-1979)
– Expert systems booms (1980s)
– Expert systems busts (1993)
• 1990-2000
– Rise of statistical approaches
(Machine Learning)
1940-1990
• 2000-2010
– Interactive robopets (2000)
– Robots expressing emotions (Cynthia Breazeal, MIT,
2000)
– Autonomous navigation (NASA, 2003)
– Recommendation technology (2005)
– Checkers game is solved (2007)
• 2010-Present Day
– Rise of deep learning
– IBM’s Watson (Jeopardy!, 2010)
– Google DeepMind’s AlphaGo world Go chamption (2017)
– Logic machine solves math conjecture (2017)
– Alibaba language processing AI beets humans in reading
and comprehension tests (2018)
1990-Present Day
A Brief History of AI

14. Sometimes AI is overhyped
The dark side of AI history
Typically, AI advances in
stages. Winters can be
mitigated if we stay realistic
and scientifically
responsible.
Beware of Winter

15. Some of the Hype is Justified
Natural Language Processing
• Automatic speech recognition
• Text-to-speech synthesis
• Web search
• Text classification
• Machine translation
• Causal text analysis
Alibaba beats humans in reading
Comprehension (January, 2018)
Computer Vision/Perception
• Object and face recognition
• Scene and image segmentation
• Image classification
• Deep learning for images
Segmentation of cell nuclei in Zikry, Kedziora, Kosorok, and
Purvis (2019, accepted for PEARC19)
What Can AI Do?

16. Robotics
• Robots
• Self-driving cars
• Control systems
• mHealth automated
real time interventions
Game playing
• IBM Deep Blue and chess
• AlphaGo and goCNN (Convolutional Neural Net) AI 95% accurate skin cancer
diagnosis versus 87% by dermatologists (USNews, 2018)
Google self-driving car
Logical systems
• Theorem provers
• Question answering
• Deduction systems
(natural reasoning)
• Satisfiability solvers
Decision making
• Scheduling, airline routing
• Rout planning, Google maps
• Web search engines
• Medical diagnosis
• Precision medicine/health
• Machine learning
What Else?

17. Brain-computer interface
• Non-speech communication
• Being able to walk again
Intelligent social media
• Strategic messaging to allay
vaccine resistance
• Identifying and mitigating
false health content
• Early warning of emerging
health threats
AI-guided administration
• Improved care in
developing nations
through automation
• Intelligent clinical trial
recruitment
• Better strategies for
emergency visits
Next generation radiology
• Imaging guided diagnostics
• Precision radiotherapy
Other AI Advances in Health

18. The Dark Side of AI
Some caution is needed
The democratization of AI
• Easy to use software is widely
available
• Pros: this can lead to greater
awareness of AI and citizen
science progress
• Cons: insufficient
understanding of statistical
principles (reproducibility), too
many false positives
The risk of bad actors
• Evil people could use AI for harm
• Automated AI tools could have
unintended adverse consequences:
• Unintended disparities
• Provoking discord/conspiracy
theories
Good research takes care
• Easy implementation leads to
impatience in research
• This leads to inadequate
evaluation (peer review)
• Promising advances may be
dismissed early due to poor
implementation
• Study design is crucial
The Dark Side of AI/ML

19. Example: T1D and Safe Exercise
• Goal: precision medicine
for minute-by-minute
control of blood glucose
level and overall control
of weight gain
• Framework: data-driven
decision science at
multiple time scales
using new hybrid SMART
designs, mHealth, micro-
randomization, system
dynamics, and statistical
control theory (robotics)
Time course plot of glucose, insulin, physical
activity, and food intake for a single pediatric
patient (data from Maahs et al, 2012)

20. Example, cont.
• We use a Markov
decision process
mathematical and
statistical framework
• We measure the state
of the patient and
actions which have
been taken
• V-learning is a new
kind of reinforcement
learning which
incorporates outcome
weighted learning
Graphical depiction of a Markov decision
process indexed by discrete time points
We assume the data consists of a sample of n
i.i.d. trajectories (𝑆1
, 𝐴1
, 𝑆2
, … , 𝑆 𝑇
, 𝐴 𝑇
, 𝑆 𝑇+1
)
where S is state, A is action (treatment), and
U is a specified utility (outcome)

21. Results From V-Learning
We applied V-learning to the mobile health data of
Maahs et al (2018)
State included insulin use, food intake, physical
activity, and blood glucose
Possible actions included take insulin, eat, exercise,
combinations (such as take insulin and eat)
Outcome: weighted average of time glucose out of
range
Simulations verify that V-learning leads to significantly
better performance than Q-learning (≈ twice as good)
For this patient, the DTR estimated from V-learning
reduced the time out of range by 64%

22. Two Modes- planning and standard
22

23. The best way to predict the future is to create it (Abraham Lincoln)
Do good AI/ML science
• Statistical foundation essential
• Reproducibility
• Generalizability
• Study design
• Computing foundation essential
• Domain expertise essential
• Rigorous team science with depth
AI/ML ethics, etc.
• Ethics of AI are highly non-trivial
• Need to preserve privacy
• Need to break barriers to data use
• Need to study unintended
consequences
• Need to develop strategies (e.g.,
game theory) to anticipate and
reduce threats from bad actors
Training the next generation
• With increased
democratization, teach
fundamental statistical
concepts
• All domain areas need
knowledge of AI and data
science
• Team science training
Communication about AI/ML
• Data science and AI experts need
specialized communication training
• Everyone needs training in data
science communication
• A new communications expertise is
needed which uses AI and data
science
• Role of public opinion is crucial
Looking to the Future

24. Concluding Thoughts
There are many useful new methods in AI and
machine learning for precision medicine discovery
We are (probably) entering a golden age of precision
medicine and precision health
Statistical inference principals are key to
reproducibility and generalizability
Although we have made much progress, much work
remains
We need to embrace trans-disciplinarity
Computational scientists need to become more
statistical and statisticians need to become more
computational

25. Thank You