Cardiology as a field has seen phenomenal technological advances over the past few decades. Existing tools however require sensors and/or electrodes on the human body to capture physiological signals. In this talk, I will show how we can use smartphones and smart speakers to contactlessly monitor human physiological signals from a distance. I will first demonstrate how we can continuously track motion and minute breathing signals by transforming these mobile devices into contactless sensors that can monitor sleep quality and detect sleep apnea. I will then show how we can use smart speakers (e.g., Alexa) to contactlessly monitor individual heart beats and detect irregular heart rhythm, without the need for any on-body sensors or electrodes. Finally, I will present our work on using machine learning on smart speakers to detect agonal breathing, an audible biomarker and brainstem reflex that arises in the setting of severe hypoxia, which is an under-appreciated diagnostic element of cardiac arrest.

1. AI for contactless cardiology
Shyam Gollakota

2. Mobile devices that are ubiquitous
2

3. Creating contactless physiological monitors using software
Contactless breathing for
sleep apnea screening
Heart rhythm monitoring
using smart speakers
Cardiac arrest detection
using agonal breathing

4. Contactless breathing and motion
tracking using smartphones [MobiSys’15]
4

5. Challenge: Breathing Motion is Minute
5

6. Breathing Motion
Δd
Our Idea: Transform Phone into Active Sonar
6

7. Transmit inaudible linear chirps
Frequency
18 KHz
22 KHz
Δf
Δt
Transmitted
signal
Reflected
Signal
Time
10ms
7

8. Transmit inaudible linear chirps
Time
t1 tn
f1 + Δfi
Frequency
f1 + Δfj
8

9. Frequency
Speaker and multiple
microphones
Time
18 KHz
22 KHz
10ms
Transmitted
signal
Person 1 Person 2
Detecting Breathing Motion of Two People
9

10. Different smartphones: iPhone, Pixel, Samsung, Nexus
1-2 m Distances, phone positions and orientations
Different clothing (sweatshirts, shirts)
Blankets of different thickness
Sleep positions – Supine, prone, left, right
Two subjects next to each other separated by 20 cm
10

11. Sleep Apnea Diagnoses using Polysomnography
11

12. Makes Polysomnography expensive ($4000)
labor intensive and cumbersome
Sleep Apnea Diagnoses using Polysomnography
12

13. First contactless system that can diagnose sleep
apnea on a smartphone
ApneaApp [MobiSys’15, Best Paper Nominee]
13

14. Harborview sleep center over one month
37 patients over 296 hours
- 17 female and 20 male
- ages of 23 – 93
Polysomnography as baseline
Apnea Events Correlation
Central apnea 0.99
Hypopnea 0.95
Obstructive apnea 0.98 14

15. Sleep Time Accuracy
0
100
200
300
400
500
600
0 5 10 15 20 25 30 35
Sleep
time
(mins)
Patient #
Mean error = 36 mins
Median error = 27 mins
EEG data
ApneaApp
15

16. 0.0
10.0
20.0
30.0
40.0
50.0
60.0
0.0 10.0 20.0 30.0 40.0 50.0 60.0
AHI
(ApneaApp)
AHI (Polysomnography)
Correlation = 0.9816
Sleep Apnea Diagnosis Accuracy
16

17. Other applications

19. Ultrasound sensing is adopted on Amazon & Google smart speakers

20. Smart speakers are powerful active sonar platforms
7-microphone
array
Speaker

21. Creating contactless physiological monitors using software
Contactless breathing for
sleep apnea screening
Heart rhythm monitoring
using smart speakers
Cardiac arrest detection
using agonal breathing

23. Heart and major arteries pulsate with every heartbeat

24. Video Credit – Emmnuel Bhaskar
https://www.youtube.com/watch?v=uDP16MklOMY
Heart movements can be perceived on the chest wall
Video credit - https://www.nejm.org/doi/full/10.1056/NEJMicm1614250
Typical chest movements are much less
pronounced than these examples

25. RR interval capture beat-to-beat variability
Normal Rhythm
R waves

26. Atrial Fibrillation – Chaotic irregular rhythm
RR interval capture beat-to-beat variability

27. Patient going in and out of normal
rhythm and atrial fibrillation
RR interval capture beat-to-beat variability

28. So…
–Irregular = abnormal
–Regular = normal
Can we still do a frequency domain analysis
- and if there is a dominant frequency à can we call it a normal rhythm?
- And a lack of a dominant frequency à automatically interpret as an abnormal rhythm?
Unfortunately not…

29. Patient with normal rhythm with
a healthy heart rate variability

30. Extracting individual heart beats is hard
• Breathing motion is not perfect
sinusoidal motion
• overwhelms the heart motion
Time domain
Harmonics of breathing motion can
hide the heart signal
Frequency domain

31. • Signal processing technique
• Focuses a wireless signal
• Can be performed both during reception and
transmission
(cellular phones, home theater speaker systems)
Beamforming

32. Self-supervised learning-based beamforming
• Use multiple microphones to perform
beamforming on the reflected signals
• Combine the acoustic signals from microphone
using complex weights
• Learn the weights using gradient descent
• Does not require training data

33. Clinical testing
•26 healthy participants
•24 cardiac patients (patients in the cardiac inpatient floors)
•Arrhythmias
•heart failure
•Valve disorders
•Pacemakers

34. Results - Healthy Participants
Scatter plot of average heart rate Scatter plot of R-R intervals
compared with ground truth

35. Results - Hospitalized cardiac patients
Scatter plot of average heart rate
(BPM) compared with ground truth
Scatter plot of R-R intervals
compared with ground truth

36. Smartspeakers can identify irregular heart rhythm
Atrial Fibrillation Atrial Fibrillation
Respiratory arrhythmia Sinus Rhythm

37. Creating contactless physiological monitors using software
Contactless breathing for
sleep apnea screening
Heart rhythm monitoring
using smart speakers
Cardiac arrest detection
using agonal breathing

39. Cardiac arrest are a leading cause of death

40. Many victims die alone at home, specifically bedroom

41. Agonal breathing as an audible biomarker for cardiac arrest
41

42. Contributions
• First smart speaker platform to passively detect cardiac arrests
• Showed accurate performance on real-world recordings of agonal
breathing using machine learning
• Analysis across sleep lab (n=12 patients, 82 hours) and different
bedrooms (n=35, 164 hours) showed low false positive rate

43. Reducing false positives

44. 9-1-1 calls as training data

45. 9-1-1 calls as training data
162 calls (19 hours) from 2009 - 2017
236 agonal breathing instances
83 hours of negative data
Sensitivity: 97.24% (95% CI: 96.86–97.61%)
Specificity: 99.51% (95% CI: 99.35–99.67%)

46. 12 patients recorded for 82 hours in total
False positive rate before frequency filter: 0.14409%
(170/117,895)
False positive rate after frequency filter: 0%
(0/117,895)
Performance across sleep lab data

47. Performance across different bedrooms
35 subjects recorded for 164 hours in total
False positive rate before frequency filter: 0.2%
(515/236,666)
False positive rate after frequency filter (2x):
0% (0/236,666)

48. Smart speakers as the
next frontier for mhealth

49. Conclusions
• Can enable contactless physiological sensing on billions of
devices using AI/software
• Smart speakers are the next frontier for mobile health