Cardiology as a field has seen phenomenal technological advances over the past few decades. Existing tools however require sensors and/or electrodes on the human body to capture physiological signals. In this talk, I will show how we can use smartphones and smart speakers to contactlessly monitor human physiological signals from a distance. I will first demonstrate how we can continuously track motion and minute breathing signals by transforming these mobile devices into contactless sensors that can monitor sleep quality and detect sleep apnea. I will then show how we can use smart speakers (e.g., Alexa) to contactlessly monitor individual heart beats and detect irregular heart rhythm, without the need for any on-body sensors or electrodes. Finally, I will present our work on using machine learning on smart speakers to detect agonal breathing, an audible biomarker and brainstem reflex that arises in the setting of severe hypoxia, which is an under-appreciated diagnostic element of cardiac arrest.
10. Different smartphones: iPhone, Pixel, Samsung, Nexus
1-2 m Distances, phone positions and orientations
Different clothing (sweatshirts, shirts)
Blankets of different thickness
Sleep positions – Supine, prone, left, right
Two subjects next to each other separated by 20 cm
10
13. First contactless system that can diagnose sleep
apnea on a smartphone
ApneaApp [MobiSys’15, Best Paper Nominee]
13
14. Harborview sleep center over one month
37 patients over 296 hours
- 17 female and 20 male
- ages of 23 – 93
Polysomnography as baseline
Apnea Events Correlation
Central apnea 0.99
Hypopnea 0.95
Obstructive apnea 0.98 14
24. Video Credit – Emmnuel Bhaskar
https://www.youtube.com/watch?v=uDP16MklOMY
Heart movements can be perceived on the chest wall
Video credit - https://www.nejm.org/doi/full/10.1056/NEJMicm1614250
Typical chest movements are much less
pronounced than these examples
27. Patient going in and out of normal
rhythm and atrial fibrillation
RR interval capture beat-to-beat variability
28. So…
–Irregular = abnormal
–Regular = normal
Can we still do a frequency domain analysis
- and if there is a dominant frequency à can we call it a normal rhythm?
- And a lack of a dominant frequency à automatically interpret as an abnormal rhythm?
Unfortunately not…
30. Extracting individual heart beats is hard
• Breathing motion is not perfect
sinusoidal motion
• overwhelms the heart motion
Time domain
Harmonics of breathing motion can
hide the heart signal
Frequency domain
31. • Signal processing technique
• Focuses a wireless signal
• Can be performed both during reception and
transmission
(cellular phones, home theater speaker systems)
Beamforming
32. Self-supervised learning-based beamforming
• Use multiple microphones to perform
beamforming on the reflected signals
• Combine the acoustic signals from microphone
using complex weights
• Learn the weights using gradient descent
• Does not require training data
34. Results - Healthy Participants
Scatter plot of average heart rate Scatter plot of R-R intervals
compared with ground truth
35. Results - Hospitalized cardiac patients
Scatter plot of average heart rate
(BPM) compared with ground truth
Scatter plot of R-R intervals
compared with ground truth
42. Contributions
• First smart speaker platform to passively detect cardiac arrests
• Showed accurate performance on real-world recordings of agonal
breathing using machine learning
• Analysis across sleep lab (n=12 patients, 82 hours) and different
bedrooms (n=35, 164 hours) showed low false positive rate
45. 9-1-1 calls as training data
162 calls (19 hours) from 2009 - 2017
236 agonal breathing instances
83 hours of negative data
Sensitivity: 97.24% (95% CI: 96.86–97.61%)
Specificity: 99.51% (95% CI: 99.35–99.67%)
46. 12 patients recorded for 82 hours in total
False positive rate before frequency filter: 0.14409%
(170/117,895)
False positive rate after frequency filter: 0%
(0/117,895)
Performance across sleep lab data
47. Performance across different bedrooms
35 subjects recorded for 164 hours in total
False positive rate before frequency filter: 0.2%
(515/236,666)
False positive rate after frequency filter (2x):
0% (0/236,666)
49. Conclusions
• Can enable contactless physiological sensing on billions of
devices using AI/software
• Smart speakers are the next frontier for mobile health
