In the race to deliver a COVID-19 vaccine, technology can be used to automate patient safety monitoring and assure that patients and physicians have valid data in order to make good decisions regarding risks and benefits.

1. Danny Lieberman, Founder and CEO
Clinical Trial Monitoring in a COVID-19 era
The language of automation for
clinical trials
June 2020
flaskdata.io

2. • To understand how automation can be used to speed delivery of valid data to decision
makers in clinical trials, we must first understand where we are today and how we got
there.
• Today, virtual trials are popular because of COVID-19.
• We’ll show that virtual trials are complex distributed systems with a new set of
problems.
• We’ll review the history of clinical trials and show that little has changed since the big
streptomycin trial in 1948
• We’ll devote the last part of the talk to how monitoring automation works.
This talk

3. The language of automation
• Collect - collect data from patients, investigators and devices. The study
designers should be able to choose the right model for their therapeutic and
patient population. Whether fully virtual or site-centric or a hybrid - the study
designers should have reliable tools to collect data. Data collection can be
from a phone, a desktop, a Web browser or an API. It should not matter.
• Detect - automated detection of protocol violations uses real-time data and
user behavior to detect anomalies and validate data streams.
• Act - automated detection should be able to trigger an automated response;
whether a push notification to a site coordinator phone or a Web hook callback.

4. 1
Act
Detect
Collect
2
3
Flask API

5. 3
Virtual trials
A brief
history
We're not in Kansas
anymore
Monitoring
distributed
systems
4
Outline
1 2 3 4

6. Virtual trials
Virtual Clinical Trials (VCTs), also called remote or
decentralized trials, are a relatively new and yet
underutilized method of
conducting clinical research taking full advantage
of technologies such as apps, electronic monitoring
devices, and online social engagement platforms.
https://leoinnovationlab.com/2020/04/02/virtual-clinical-trials-create-new-possibilities-for-patients-

7. Virtual trials: Complex distributed systems
Virtual clinical trials may save time and money.
John Robert Zibert - LEO Innovation Lab 4/2020
eCOA
ePRO
eSource
eConsent
EDC

8. 3
Virtual trials
A brief
history
We're not in Kansas
anymore
Monitoring
distributed
systems
41 2 3 4

9. A brief history of time.
Daniel 1
526BCE
Avicenna
Canon of
Medicine
James Lind and
the scurvy trial
1025 1747
Streptomycin
RCT
1946
FDA Guidance
on RBM
2013

10. Daniel 1
Please test your servants for ten days: Give us nothing but vegetables to
eat and water to drink. Then compare our appearance with that of the
young men who eat the royal food, and treat your servants in accordance
with what you see.”
So he agreed to this and tested them for ten days.
At the end of the ten days they looked healthier and better nourished than
any of the young men who ate the royal food.
So the guard took away their choice food and the wine they were to drink
and gave them vegetables instead.

11. Avicenna - Drug testing rules
‣ Disease without complications
‣ 2 contrary cases
‣ At time of action and reproducibility

12. James Lind and Scurvy trial
‣ Cohort 1 Gruel and mutton-broth
‣ Cohort 2 Quart of cyder a day
‣ 50 years later British Navy made lime compulsory

13. Clinical trial monitoring basics
(1948)
‣ Communicate with PI and site staff
‣ Review site processes, procedures
‣ Verify data accuracy

14. ICH E6 and ISO 14155:2011
‣ Monitoring plan
‣ Design, complexity, size, and endpoints
‣ Consistent with FDA Guidance

15. FDA Guidance for RBM (2013)
‣ Centralized monitoring can replace onsite visits (1)
‣ Focus on site, not patient oversight
‣ Response: 2-3 days to 5-7 weeks after events
‣ Mitigate safety, quality risk
‣ Too slow for virtual/hybrid trials in a COVID-19 era
(1) Bakobaki et al. The Potential for Central Monitoring Techniques to Replace On-Site Monitoring:(2012)

16. Over-monitoring
‣ Monitoring is non-invasive and seems harmless (1)
‣ 72–99% of ICU alarms clinically insignificant
‣ Cry wolf syndrome
(1) Feder and Funk, Over-monitoring and alarm fatigue: For home the bells toll?
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4926996/

17. Onsite monitoring with CRAs
‣ Onsite monitoring 20-35% of study cost
‣ Log deviations 7-13 weeks after data event
‣ SDV updates < 3% of data
‣ Creates data cleaning bottleneck at study close

18. Monitoring with Dropbox (2020)
‣ Remote monitoring with PDF
‣ More paper
‣ Less site visits. Not good.

19. 3
A brief
history
We're not in Kansas
anymore
Monitoring
distributed
systems
41 2 3 4
Virtual trials

20. Regulatory and reimbursement
conflicts
Generalizable data from
near-real-life usage
Well-designed controlled
clinical studies
Security and privacy
vulnerabilities

21. Vintage monitoring of virtual
trials does not work
Sites Study
monitors
Doing
what they
feel like
Things
happening
fast
Patients are
anywhere
10x

22. Submission by December 17
Need to revisit clinical trial design and operations
‣ Wearables, devices, apps
‣ Cloud services and APIs
‣ Real-time data & response for real-life usage

23. 3
A brief
history
We're not in Kansas
anymore
41 2 3 4
Virtual trials
Monitoring
distributed
systems

24. Monitoring by alerts
‣ Alerts urgent, important, actionable and real
‣ Symptoms better than causes
‣ Visualize trends

25. What are alerts?
‣ A metric over / under a threshold
‣ GCP, safety, common sense
‣ Rate of data acquisition

26. In your monitoring plan
‣ Monitor for you
‣ Symptom-based
‣ Latency. Fast. Fast. Fast.
10x

27. Alerts require 4 things
‣ Detection
‣ Enable mitigation
‣ Share knowledge
‣ Route to decision maker

28. Writing alert rules
‣ Metric and threshold
‣ Over-monitoring harder than under-monitoring
‣ Classify: Data latency, Protocol violation, AE/SAE, Recruitment

29. Taking action
‣ Playbook is important
‣ Rule or family of rules
‣ What it means and how to mitigate

30. Alert V&V
‣ Validation: Are we calculating the right metric?
‣ Verification: Are we calculating the metric right?

31. Alerts, issues and email
‣ Don't send mail
‣ Issue-tracking systems may help
‣ Goal is simplicity and speed
10x

32. Tracking & accountability
‣ Track and periodically review alerts
‣ If WFM remove it
‣ If alert < 50% accurate its broken

33. Over-optimism
‣ Causes beneath noise
‣ Symptoms may arrive late
‣ Rules may be more complex than issues

34. Automated detection & response is now a requirement
Wrap-up
‣ Patient-centric/virtual/hybrid clinical trials are
complex distributed systems with new challenges.
‣ The old people and process methods used by CROs
are too slow in a COVID-19 era
‣ Lessons can be learned from complex system
monitoring

35. Thank you’s.
‣ Jenya, Rivka, Eugene, Alex, Oleg, Anat, Batya and Dan
for joining me on the flaskdata.io journey
‣ Sergey and his DevOps soldiers for playing in the
mud with us
‣ Tim and Gene for allowing me to join their adventure
at Fidelis Security
‣ Rob Ewaschuk’s observations while a Site Reliability
engineer at Google
‣ Lana’s insight on conflicts

36. To learn more
dannyl@flaskdata.io
twitter.com/flaskdata
flaskdata.io