Presented at the International Clinical Librarians' Conference in October 2019.

1. ‘Alexa, are vaccines safe?’
Evaluating voice assistant responses to
consumer health questions
Emily Alagha, MSIS, AHIP - Clinical Information Specialist & Data
Management Librarian - Georgetown University Medical Center
Rachel Helbing, MLIS, MS, AHIP - Director of Library Services for
Health Sciences - University of Houston

2. Objectives
1. Background
2. Purpose
3. Data collection
4. Results
5. Clinical implications

3. Voice
Assistants

4. Siri
Google Assistant
Alexa

8. Online Misinformation

9. Vaccination rates

10. Outbreaks

11. Online Misinformation
Vaccination rates
Outbreaks

12. Quality & accuracy
Siri
Google Assistant
Alexa
vaccine safety & use

14. Smart speaker brand owned
©2019MintelGroupLtd.AllRightsReserved.
ConfidentialtoMintel.
Read on mintel.com
Source:
Base:
Fieldwork:
554 internet users aged 18+ who own a smart speaker
March 2019 Lightspeed/Mintel
What brands of smart speakers do you own? Please select all
that apply.
72%
30%
14%
6%
5%
1%
1%
0% 25% 50% 75% 100%
Amazon
Google
Apple
JBL
Sonos
I don't know
Other, please specify
14

15. Selecting Test Questions
The Centers for Disease
Control (CDC) immunization
FAQ page and the CDC infant
immunization FAQ page

16. AnswerThePublic.com
search for ‘vaccines’

17. Test Questions
54 total

18. Data Collection
1. Set up iPads
2. Asked & recorded answers
3. Scored with rubric
4. Calculated results

20. Siri
5.16 5.10
0.98
Google Assistant Alexa
Mean
Scores

21. Voice Assistant Alexa Google Assistant Siri
VA provided same
response to both
researchers
62.96% 70.37% 68.52%
VA understood question
and provided answer
25% 94% 94%
Average spoken words
per answer
29 21 13
Most cited source Wikipedia.org CDC.gov CDC.gov

22. Limitations
• Data collected in 2018; may not reflect recent algorithm
updates
• Scoring rubric has not been assessed for reliability and
may need future refinements
• May not reflect real use cases that incorporate tailored
search results
• Overall scores for all assistants may have been lower if
prioritizing audio only output

23. Implications: Web Search

24. Implications: Third Party Apps

25. Implications: Third Party Apps

26. Conclusions
• Siri and Google provided evidence-based links; little
audio output
• Alexa was worst at natural language comprehension;
high audio output
• Many information providers are entering voice
information provision via traditional websites and
third-party apps
• Information professionals should push for better
usability and increased transparency about health
information partnerships

28. References
1. https://about.ads.microsoft.com/en-us/insights/2019-voice-report
2. https://pixabay.com/photos/voice-control-voice-commands-2598422/
3. Brand icons from https://www.flaticon.com
• https://www.flaticon.com/authors/simpleicon (Google)
• https://www.freepik.com (Apple & Amazon)
4. https://www.pewresearch.org/fact-tank/2013/02/01/majority-of-adults-look-online-for-health-
information/
5. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0194811
6. https://www.ncbi.nlm.nih.gov/pubmed/26974260
7. https://www.nature.com/articles/d41586-018-07034-4
8. https://www.washingtonpost.com/national/health-science/percentage-of-young-us-children-
who-dont-receive-any-vaccines-has-quadrupled-since-2001/2018/10/11/4a9cca98-cd0d-11e8-
920f-dd52e1ae4570_story.html?utm_term=.f3d142106107

29. References
9. https://www.cdc.gov/measles/cases-outbreaks.html
10. https://9to5google.com/2019/04/29/google-assistant-siri-market-share/
11. https://www.healthcareitnews.com/news/uchealth-launches-alexa-based-virtual-assistant-
voice-enabled-patient-engagement
12. https://www.theguardian.com/media/2019/aug/27/bbc-to-launch-alexa-rival-that-will-grasp-
regional-accents
13. https://www.theguardian.com/society/2019/jul/10/nhs-teams-up-with-amazon-to-bring-alexa-
to-patients
14. https://www.wired.com/story/amazon-alexa-search-for-the-one-perfect-answer/
15. https://earthtoplanet.com/the-future-of-medicine/