Presentation at the NLP COVID-19 Workshop at ACL'2020. Paper: https://openreview.net/forum?id=HxIZzQZy_0F Abstract: Just as SARS-CoV-2, a new form of coronavirus continues to infect a growing number of people around the world, harmful misinformation about the outbreak also continues to spread. With the goal of combating misinfor-mation, we designed and builtJennifer–achatbot maintained by a global group of volunteers. With Jennifer, we hope to learn whether public information from reputable sources could be more effectively organized and shared in the wake of a crisis as well as to understand issues that the public were most immediately curious about. In this paper, we introduce Jennifer and describe the design of this proof-of-principle system. We also present lessons learned and discuss open challenges. Finally, to facilitate future research, we release COQB-19 (COVID-19 QuestionBank)1, a dataset of 3,924 COVID-19-related questions in 944 groups, gathered from our users and volunteers. Jennifer is available at http://bit.ly/jenniferai

1. Jennifer for
COVID-19:
An NLP-Powered
Chatbot Built for the
People and by the
People to Combat
Misinformation
Yunyao Li, Tyrone Grandison, Patricia
Silveyra, Ali Douraghy, Xinyu Guan, Thomas
Kieselbach, Chengkai Li, Haiqi Zhang

2. Pandemic + Infodemic = ?

3. Background
create a platform of evidence-based
information from reliable sources, curated
by scientists, that the public would find
easy to interact with.

4. Design Consideration Design Choices
Rapid Development Using an existing platform
Ease of Access
Chatbot available on multiple
ways
Ease of Maintenance
Maintainable without
programming by crowd
Quality Assurance
Rigorous process with clear
separation of tasks with
different levels of oversights
Extensibility
Extensible without
programming

5. Dialog
Manager
Conversation
Agenda
Juji Base System
Conversation Topics
IF
THEN
Extensions
Curator Helper Tester External Data
Source
Question-Answer Pairs
Admin QA Generator
If
relevance(input) > …
then …
Chat UI
Overall Architecture

6. Juji Base System
• Expressive visual dialog flow design
• Maintainable directly via UI
• Deployable as Web and Facebook bots
• Extensible via QA pairs in spreadsheet
• IR-style QA to match a user question against an
existing question
[Xiao, et al CHI’2020]
Design
Develop
TesteDeployment
Launch
< 24 hours
March 7
March 8

7. Main Capabilities: QA Pairs
• Crowdsourced: Majority of the efforts
• Auto-Generation: With manually-curated templates + CDC/WHO data
Current focus: statistics on case and death #s.

8. Example Process: Answer Curation
Deploy
Accurate and open?
Easy to understand?
Empathetic?

9. Chat Design
Support mixed Initiatives Allow two-way adaptations
1. User  System
2. System  User

10. Multilingual support
Plans to expand Jennifer to other languages are currently under development.
Sofía (Spanish chatbot)
- QA pairs manually translated
from the Jennifer QA pairs.
- Maintained and manually
curated by a group of bilingual
Spanish-English certified medical
interpreters.
- Uses information from Spanish
language verified sources

11. Preliminary Results (as of June 18,2020)
• 1056 sessions
• 1,480 questions(excluding questions selected via menus)
• Answered 1,059 of them (response rate = 71%)
• Average engagement duration = 3 min 15 sec
• COVID-19 Question Bank (COQB)
https://www.newvoicesnasem.org/data-downloads
• 3,924 COVID-19-related questions in 944 groups

12. Lessons Learned
• People are eager to help.
• Process and communication are Important.
• Effective and dedicated management is critical.
• Human-machine conversation requires a
proactive design

13. Open Challenges
• Scalable Crowdsourced Fact Checking Platform
• Minimize human efforts w/o sacrificing quality
• Zero-Shot Empathetic Natural Language Generation
• Identify resources and compose answers
• Competing Information Sources and Public Trust
• Require than technical solutions

14. Next Steps
• Formal evaluation
• More automation
• Fact-checking database
 Auto-generation + manual validation
• Automate process management
• Language Expansion
• Partnership

15. Thank You!
160+
Volunteers
141
institutions

16. Backup

17. QA Pairs
• The main capabilities of Jennifer come from the Question-Answer(QA) pairs.
• These are generated by two extension modes: Crowdsourcing and
Automated
• Crowdsourced QA pairs are managed by 4 volunteer groups: Curators,
Helpers, Testers, and Admins.
• To be included in the chatbot, each answer needs to be:
• Easy to understand
• Accurate and Open
• Demonstrate Empathy

18. Background
• Just as the novel coronavirus continues to infect people around the world,
harmful misinformation about it also continues to spread.
• Due to the pandemic, more people are consuming information available on
the internet, making them more vulnerable to access misleading or fake
information. The WHO has called this a “massive infodemic”.
• While scientists are well placed and willing to help fight COVID-19
misinformation, getting involved often means participating in time-
consuming efforts at the expense of their research time.
• We envisioned using AI to create a platform of evidence-based information
from reliable sources, curated by scientists, that the public would find easy to
interact with.

19. Design Considerations
• We designed and built "Jennifer" and recruited a global group of
volunteer scientists to help test and scale Jennifer’s performance.
• The goals of this proof-of-principle system is to demonstrate the feasibility to
directly crowd-source the global scientific community’s expertise for public
benefit without the need for intermediaries, thus helping improve public trust
in science.
• Our core design considerations are:
• Rapid Development
• Ease of Access
• Ease of Maintenance
• Quality Assurance
• Extensibility