Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
ON EXPLOITING MULTIMODAL INFORMATION
FOR MACHINE INTELLIGENCE AND NATURAL INTERACTIONS
WITH EXAMPLES FROM HEALTH CHATBOTS
...
▰ “But most
importantly, by
freeing physicians
from the tasks that
interfere with
human connections,
AI will give doctors
...
3
Outline
❖ Humans benefit from consuming data in the form of various modalities
(text, speech, and visual).
❖ Multimodal ...
4
Source: https://www.businessinsider.com/amazon-reveals-alexa-sales-2019-1
Voice Assistants on the Rise
5
Machine-centric to
Human-centric Computing
Artificial
Intelligence
Ambient
Intelligence
Augmenting
Human Intellect
Human...
Eric Topol:
The World of Shallow Medicine
▰ “(Doctors) have not had access to
patients in their real-world, on the
go, at ...
8
Using Smart Chatbots to Go Help Escape
Shallow Medicine
Socio-
economic
Demo-
graphic
Family &
social
Psychological
Envi...
Figure source: https://www.aarp.org/health/conditions-treatments/info-2017/
bronchitis-and-pneumonia-symptoms.html
A machi...
AUGMENTED PERSONALIZED HEALTH
EXPLOITING MULTIMODAL INFORMATION FOR:
SELF-MONITORING
SELF-APPRAISAL
SELF-MANAGEMENT
INTERV...
11
This not only prevent the disease, but also enhances the patient’s health.
BariatricsAsthma
Use Cases: APH for Asthma a...
“
12
The Holy Grail of machine intelligence is the ability to
mimic the human brain. However, the human brain’s
cognitive ...
What is Modality
GENERAL
A particular mode in
which something exists
or is experienced or
expressed.
A particular form of
...
14
Machine Intelligence for Chatbot:
Incorporating Diverse Streams
& Modalities
Figure: Chatbot exploiting
multimodal info...
USE CASES & PROTOTYPES
Examples and early progress on ongoing collaborative
healthcare (chatbot) projects
@ KNO.E.SIS
Thre...
16
Health Related Studies at KNO.E.SIS
[Overview]
HealthChallenges
(Also Dementia,
Obesity,
Parkinson’s, Liver
Cirrhosis, ...
3 Chatbots (Alpha/Beta Stage)
1. NOURICH: A Google Assistant based
Conversational Nutrition Management
System
2. kBOT: Kno...
18
Physical-Cyber-Social (PCS) Data
Mobile app Q/A (tablet), forced exhaled volume in 1 sec (FEV1),
peak expiratory flow (...
19
Modalities in Select mApps
20
Chatbots for
Healthcare KNO.E.SIS
Overview
21
Use Case 1: ASTHMA
Many Sources of Highly Diverse Data
(& collection methods: Active + Passive):
Up to 1852 data points...
Data Collection So Far
110
patients
30
parameters
1852
data points per
patient per day
63%
kit compliance
● Data Collectio...
23
Utkarshani Jaimini, Krishnaprasad
Thirunarayan, Maninder Kalra, Revathy
Venkataramanan, Dipesh Kadariya, Amit
Sheth, “H...
24
Use Case 2: NOURICH
(diet management chatbot)
25
Use Case 3: Elder Care Intelligent Assistant to support elderly with
Heart Failure (HF),
Chronic Obstructive Pulmonary ...
“To support the (chatbots’) data analysis and reasoning
needs, we use a pedagogical framework consisting of
Semantic compu...
SEMANTIC-COGNITIVE-PERCEPTUAL
COMPUTING
Knowledge-Infused AI with Contextualization
(Knowledge Graphs), Personalization & ...
28
Semantic Browsing
Extraction
Data Integration and Interlinking
Entity
Complex Extraction
Aberrant
Drug-related
Behaviou...
29
SOCIAL -MEDIA TEXT
(July 12,2016)
EVENT-SPECIFIC
SCHEMA-BASED
KNOWLEDGE
30
Application: Evolving Patient Health Knowledge Graph (PHKG)
Figure: A healthcare intelligent assistant interacts with t...
31
ONE SLIDE TO SHOW HOW
PHKG EVOLVES OVER TIME
Knoesis Alchemy API
KHealth Project (IoT) datasets (e.g., asthma, obesity,...
32
Contextualization and
Personalization
kBOT initiates greeting
conversation.
Understands the patient’s health
condition ...
GENERIC CHATBOT VS
INTELLIGENT CHATBOT
With Examples of Contextualization, Personalization, and
Abstraction
34
Contextualization
refers to data interpretation in terms of knowledge (context).
Without Domain Knowledge With Domain K...
35
Personalization
refers to future course of action by taking into account the contextual factors such as
user’s health h...
36
Abstraction
A computational technique that maps and associates raw data to action-related
information.
With Abstraction...
37
Smarter Chatbot with
Semantically-Abstracted Information
Smarterdata
Data Sophistication
Smart (semantically-abstracted...
38
Semantic, Cognitive, Perceptual Computing:
Paradigms That Shape Human Experience
http://bit.ly/SCPComputing
Humans are ...
Knowledge-Infused
Learning with
Semantic,
Cognitive,
Perceptual
Computing
Framework
39
THE BABY STEPS:
MACHINE / DEEP LEAR...
40
In short,
❖ Multimodal information are essential and can
be exploited for machine intelligence and
natural interactions...
41
Special Thanks
Hong Yung (Joey) Yip
(Graduate Student)
Prochain SlideShare
Chargement dans…5
×

ON EXPLOITING MULTIMODAL INFORMATION FOR MACHINE INTELLIGENCE AND NATURAL INTERACTIONS WITH EXAMPLES FROM HEALTH CHATBOTS

395 vues

Publié le

Keynote: SECOND INTERNATIONAL WORKSHOP IN MULTIMEDIA PRAGMATICS MMPrag 2019, San Jose, California, 28-30 March 2019
http://mipr.sigappfr.org/19/keynote-speakers/

The Holy Grail of machine intelligence is the ability to mimic the human brain. In computing, we have created silos in dealing with each modality (text/language processing, speech processing,image processing, video processing, etc.). However, the human brain’s cognitive and perceptual capability to seamlessly consume (listen and see) and communicate (writing/typing, voice, gesture) multimodal (text, image, video, etc.) information challenges the machine intelligence research. Emerging chatbots for demanding health applications present the requirements for these capabilities. To support the corresponding data analysis and reasoning needs, we have to explore a pedagogical framework consisting of semantic computing, cognitive computing, and perceptual computing (http://bit.ly/w-SCP). In particular, we have been motivated by the brain’s amazing perceptive power that abstracts massive amounts of multimodal data by filtering and processing them into a few concepts (representable by a few bits) to act upon. From the information processing perspective, this requires moving from syntactic and semantic big data processing to actionable information that can be weaved naturally into human activities and experience (http://bit.ly/w-CHE). Exploration of the above research agenda, including powerful use cases, is afforded in a growing number of emerging technologies and their applications - such as chatbots and robotics. In this talk, I will provide these examples and share the early progress we have made towards building health chatbots (http://bit.ly/H-Chatbot) that consume contextually relevant multimodal data and support different forms/modalities of interactions to achieve various alternatives for digital health (http://bit.ly/k-APH). I will also discuss the indispensable role of domain knowledge and personalization using domain and personalized knowledge graphs as part of various reasoning and learning techniques.

Publié dans : Technologie
  • Login to see the comments

  • Soyez le premier à aimer ceci

ON EXPLOITING MULTIMODAL INFORMATION FOR MACHINE INTELLIGENCE AND NATURAL INTERACTIONS WITH EXAMPLES FROM HEALTH CHATBOTS

  1. 1. ON EXPLOITING MULTIMODAL INFORMATION FOR MACHINE INTELLIGENCE AND NATURAL INTERACTIONS WITH EXAMPLES FROM HEALTH CHATBOTS Keynote: SECOND INTERNATIONAL WORKSHOP IN MULTIMEDIA PRAGMATICS MMPrag 2019, San Jose, California, 28-30 March 2019 Amit Sheth LexisNexis Ohio Eminent Scholar The Ohio Center of Excellence in Knowledge-enabled Computing & BioHealth Innovations (Kno.e.sis) Wright State, USA Icon source used in the entire presentation - https://thenounproject.com Presentationtemplateby SlidesCarnival Photographsby Unsplash
  2. 2. ▰ “But most importantly, by freeing physicians from the tasks that interfere with human connections, AI will give doctors the gift of time—to restore care in healthcare.” 2 ▰ “…virtual assistants, powered by personalized AI, can provide us with coaching to promote our health, shape our diet, and even prevent illness.”
  3. 3. 3 Outline ❖ Humans benefit from consuming data in the form of various modalities (text, speech, and visual). ❖ Multimodal information are essential and together, they provide nuances that a single modality can’t. ❖ For a machine to attain intelligence, it requires comprehensive understanding of the environment that it is in. ❖ And to develop natural interactions with human, a machine needs to develop understanding of the data it consumes. ❖ This talk will focus on different data modalities and examples on how a machine (chatbot) can use such information to provide intelligent assistant and natural communication in the health domain.
  4. 4. 4 Source: https://www.businessinsider.com/amazon-reveals-alexa-sales-2019-1 Voice Assistants on the Rise
  5. 5. 5 Machine-centric to Human-centric Computing Artificial Intelligence Ambient Intelligence Augmenting Human Intellect Human-Computer Symbiosis Computing for Human Experience Machine-centric Human-centric John McCarthy Mark Weiser Douglas Engelbart Joseph C.R. Licklider Figure: Views along the spectrum of machine-centric to human-centric computing. At the far right is our work on Computing for Human Experience, which explores paradigms such as Semantic, Cognitive, and Perceptual Computing. http://bit.ly/SCP-Magazine Kno.e.sis Center http://bit.ly/k-Che, http://slidesha.re/k-che
  6. 6. Eric Topol: The World of Shallow Medicine ▰ “(Doctors) have not had access to patients in their real-world, on the go, at work, while asleep. The data doctors access is from the contrived setting of the medical office, constrained by the temporal lists of the visit itself….(EHR info) is remarkably incomplete and inaccurate.” ▰ “Patients exist in the world of insufficient data, insufficient time, insufficient context, and insufficient presence.” 7
  7. 7. 8 Using Smart Chatbots to Go Help Escape Shallow Medicine Socio- economic Demo- graphic Family & social Psychological Environment Genetic Susceptibility Source: Why do people consult the doctor? - Stephen M Campbell and Martin O Roland Decision Making Can voice assistant (chatbot) technology substantially improve monitoring of patient’s conditions and needs? Simple Tasks ● Appointment scheduling ● Information retrieval ● Scripted-automation Complex & Demanding Tasks ● Multimodal input and output ● Natural communication ● Augmented Personalized Health (serving different levels of health needs) Contextualization Personalization Abstraction Different modality of data ImagesText Speech Videos IoTs
  8. 8. Figure source: https://www.aarp.org/health/conditions-treatments/info-2017/ bronchitis-and-pneumonia-symptoms.html A machine may recognize the picture as “a woman is coughing”. As human, we immediately conjecture and relate to many phenomena with different contexts. Semantic Association (Label picture as coughing) Cognitive (Look at additional background information & interpret in different context, ie: cough vs wheezing cough Perception (Has the patient condition worsen? How well is the patient doing?) Paradigms that Shape Human Experience
  9. 9. AUGMENTED PERSONALIZED HEALTH EXPLOITING MULTIMODAL INFORMATION FOR: SELF-MONITORING SELF-APPRAISAL SELF-MANAGEMENT INTERVENTION DISEASE PROGRESSION AND TRACKING
  10. 10. 11 This not only prevent the disease, but also enhances the patient’s health. BariatricsAsthma Use Cases: APH for Asthma and Bariatrics: Patient-centric drivers
  11. 11. “ 12 The Holy Grail of machine intelligence is the ability to mimic the human brain. However, the human brain’s cognitive and perceptual capability to seamlessly consume, abstracts massive amounts of multimodal data, and communicate information challenges the machine intelligence research. Growing number of emerging technologies such as chatbots & robotics present the requirements for these capabilities.
  12. 12. What is Modality GENERAL A particular mode in which something exists or is experienced or expressed. A particular form of sensory perception: ‘the visual and auditory modalities’. HEALTHCARE MODALITY Modality (medical imaging), a type of equipment used to acquire structural or functional images of the body, such as radiography, ultrasound, nuclear medicine, computed tomography, magnetic resonance imaging and visible light. IN HCI A modality is the classification of a single independent channel of sensory input/output between a computer and a human. Multiple modalities can be used in combination to provide complementary methods that may be redundant (or complementary) but convey information more effectively. 13
  13. 13. 14 Machine Intelligence for Chatbot: Incorporating Diverse Streams & Modalities Figure: Chatbot exploiting multimodal information for machine intelligence and natural interactions From simple informational interface (text, speech) to intelligent assistant
  14. 14. USE CASES & PROTOTYPES Examples and early progress on ongoing collaborative healthcare (chatbot) projects @ KNO.E.SIS Three varieties:  Data for patient’s real world  Virtual Medical Coach  Smart Nutrition
  15. 15. 16 Health Related Studies at KNO.E.SIS [Overview] HealthChallenges (Also Dementia, Obesity, Parkinson’s, Liver Cirrhosis, ADHF) Public Policy/ Population Epidemiology Personalized Health PCS + EMR + Multimodal (Speech + Image) kHealth Asthma in Children Bariatric Surgery Nutrition Physical(IoT)/Cyber/ Social (PCS)+ EMR Marijuana Social Drug Abuse Social Mental Health Depression & Suicide Social + Public + EMR Health Knowledge Graph Services Social + Clinical Data ...and infrastructure technologies: Context-aware KR (SP), KG Development, Smart Data from PCS Big Data, Twitris
  16. 16. 3 Chatbots (Alpha/Beta Stage) 1. NOURICH: A Google Assistant based Conversational Nutrition Management System 2. kBOT: Knowledge-enabled (kHealth) Personalized ChatBot for Asthma: Contextualized & Personalized Conversations involving Multimodal data (IoT & Devices) 3. ReaCTrack: Personalized Adverse Reaction Conversation-based Tracker for Clinical Depression 17 HCI: Mobile Applications & Chatbots @ KNO.E.SIS kHealth Asthma kHealth Bariatrics Depression Active (Subset) Healthcare Projects @ KNO.E.SIS with mApps/chatbot kHealth Framework: a knowledge-enhance AI learning platform that captures the data and analyzes it to produce actionable information.
  17. 17. 18 Physical-Cyber-Social (PCS) Data Mobile app Q/A (tablet), forced exhaled volume in 1 sec (FEV1), peak expiratory flow (PEF), indoor temperature, indoor humidity, particulate matter, volatile organic compound, carbon dioxide, air quality index, pollen level, outdoor temperature, outdoor humidity, number of steps, heart rate and number of hours of sleep. Also clinical notes. kHealth Asthma Nutrition Depression Active Healthcare Projects in Kno.e.sis (Subset) Modality of Data kHealth Bariatrics For monitoring asthma control and predict vulnerability Pre and Post Surgery monitoring and self adherence Mobile app Q/A (tablet), weighing scale, pill bottle sensor, water bottle sensor for reminder to drink water, number of steps, heart rate and number of hours of sleep. Also clinical notes. Q/A, diet, food profile, food images, nutrition knowledge bases, user knowledge graph. For nutrition tracking and diet monitoring Modeling Social Behavior for Healthcare Utilization in Depression Q/A, social media profile (Twitter, Reddit).
  18. 18. 19 Modalities in Select mApps
  19. 19. 20 Chatbots for Healthcare KNO.E.SIS Overview
  20. 20. 21 Use Case 1: ASTHMA Many Sources of Highly Diverse Data (& collection methods: Active + Passive): Up to 1852 data points/ patient /day kBot with screen interface for conversation Images Text Speech *(Asthma-Obesity) ★ Episodic to Continuous Monitoring ★ Clinician-centric to Patient-centric ★ Clinician controlled to Patient-empowered ★ Disease Focused to Wellness-focused ★ Sparse data to Multimodal Big Data http://bit.ly/kHealth-Asthma
  21. 21. Data Collection So Far 110 patients 30 parameters 1852 data points per patient per day 63% kit compliance ● Data Collection: Since Dec 2016 ● Active sensing: 18 data points/day (Peak flow meter and Tablet) ● Passive sensing: 1834 data points/ day (Foobot, Fitbit, Outdoor environmental data) 5-17 years of age 1 or 3 months of monitoring 22
  22. 22. 23 Utkarshani Jaimini, Krishnaprasad Thirunarayan, Maninder Kalra, Revathy Venkataramanan, Dipesh Kadariya, Amit Sheth, “How Is My Child’s Asthma?” Digital Phenotype and Actionable Insights for Pediatric Asthma”, JMIR Pediatr Parent 2018;1(2):e11988, DOI: 10.2196/11988.
  23. 23. 24 Use Case 2: NOURICH (diet management chatbot)
  24. 24. 25 Use Case 3: Elder Care Intelligent Assistant to support elderly with Heart Failure (HF), Chronic Obstructive Pulmonary Disease (COPD) or Type 2 Diabetes Mellitus (T2DM).
  25. 25. “To support the (chatbots’) data analysis and reasoning needs, we use a pedagogical framework consisting of Semantic computing, Cognitive computing, and Perceptual computing This requires moving from syntactic and semantic big data processing to actionable information that can be weaved naturally into human activities and experience. 26
  26. 26. SEMANTIC-COGNITIVE-PERCEPTUAL COMPUTING Knowledge-Infused AI with Contextualization (Knowledge Graphs), Personalization & Abstraction
  27. 27. 28 Semantic Browsing Extraction Data Integration and Interlinking Entity Complex Extraction Aberrant Drug-related Behaviour Neuro-Cognitive Symptoms Adverse Drug Reaction Relation Event Severity Personal Sensor Data De-identified EMR Blog Post Context Representation Relevant Subgraph Selection Semantic Search Disease-specific Chatbot Visualization Health Knowledge Graph Intent Open Health Knowledge Graph
  28. 28. 29 SOCIAL -MEDIA TEXT (July 12,2016) EVENT-SPECIFIC SCHEMA-BASED KNOWLEDGE
  29. 29. 30 Application: Evolving Patient Health Knowledge Graph (PHKG) Figure: A healthcare intelligent assistant interacts with the patient via various conversational interfaces (voice, text, and visual) to acquire and disseminate information, and provide recommendation (validated by physician). The core functionalities of the chatbot (Component C boxed in blue) are extended with a background HKG (Component A boxed in green) and a evolving PKG (Component B boxed in orange). ★ Smarter & engaging agent ★ Minimize active sensing (Questions to be asked) ★ Ask only informed & intelligent questions ★ Relevant & Contextualized conversations ★ Personalized & Human-Like
  30. 30. 31 ONE SLIDE TO SHOW HOW PHKG EVOLVES OVER TIME Knoesis Alchemy API KHealth Project (IoT) datasets (e.g., asthma, obesity, Parkinson) Reasoning mechanisms Enriching KG Enriching KG In-built rule-based inference engine Machine Learning Updating the KG with more triples Analyzing datasets Executing reasoning Ontology Catalogs: ● BioPortal ● Linked Open Vocabularies (LOV) ● Linked Open Vocabularies for Internet of Things (LOV4IoT) Linked Open Data (LOD): ● UMLS ● SNOMED-CT ● ICD-10 ● Clinical Trials ● Sider Personalized Health Knowledge Graph (PHKG) Personal Sensor Data Electronic Medical Records (EMR) Figure: How a PHKG evolves with multimodal information
  31. 31. 32 Contextualization and Personalization kBOT initiates greeting conversation. Understands the patient’s health condition (allergic reaction to high ragweed pollen level) via the personalized patient’s knowledge graph generated from EMR, PGHD, and prior interactions with the kBot. Generates predictions or recommended course of actions. Inference based on patient’s historical records and background health knowledge graph containing contextualized (domain-specific) knowledge. Figure: Example kBot conversation which utilizes background health knowledge graph and patient’s knowledge graph to support intelligent interactions including individualized recommendation. ★ Conversing only information relevant to the patient Context enabled by relevant healthcare knowledge including clinical protocols.
  32. 32. GENERIC CHATBOT VS INTELLIGENT CHATBOT With Examples of Contextualization, Personalization, and Abstraction
  33. 33. 34 Contextualization refers to data interpretation in terms of knowledge (context). Without Domain Knowledge With Domain Knowledge Chatbot with domain (drug) knowledge is potentially more natural and able to deal with variations.
  34. 34. 35 Personalization refers to future course of action by taking into account the contextual factors such as user’s health history, physical characteristics, environmental factors, activity, and lifestyle. Without Contextualized Personalization With Contextualized Personalization Chatbot with contextualized (asthma) knowledge is potentially more personalized and engaging.
  35. 35. 36 Abstraction A computational technique that maps and associates raw data to action-related information. With AbstractionWithout Abstraction .
  36. 36. 37 Smarter Chatbot with Semantically-Abstracted Information Smarterdata Data Sophistication Smart (semantically-abstracted) data should answer: ★ What causes my disease severity? ★ How well am I doing with respect to prescribed care plan? ★ Am I deviating from the care plan? I am following the care plan but my disease is not well controlled. ★ Do I need treatment adjustments? ★ How well controlled is my disease over time? Example of Abstraction
  37. 37. 38 Semantic, Cognitive, Perceptual Computing: Paradigms That Shape Human Experience http://bit.ly/SCPComputing Humans are interested in high-level concepts (phenotypic characteristics). Semantic Computing: Assign labels and associate meanings (representation & contextualization). Cognitive Computing: Interpretation of data with respect to perspectives, constraints, domain knowledge, and personal context. Perceptual Computing: A cyclical process of semantic-cognitive computing for higher level of perception and reasoning (abstraction & action).
  38. 38. Knowledge-Infused Learning with Semantic, Cognitive, Perceptual Computing Framework 39 THE BABY STEPS: MACHINE / DEEP LEARNING INFUSED WITH PERSONALIZED HEALTH KNOWLEDGE GRAPH Knowledge Domain (Ontology) Personalized HKG Multisensory Sensing & Multimodal Data Interactions ImagesText Speech Videos IoTs Natural Language Processing, Machine with Deep Learning AUGMENTED PERSONALIZED HEALTH (APH) Modeling broader disease context, and personalized user behavior Reasoning & decision- making framework To achieve ABSTRACTION and minimize data overload, assist in making choices, appraisal, recommendations
  39. 39. 40 In short, ❖ Multimodal information are essential and can be exploited for machine intelligence and natural interactions. ❖ Knowledge-infused learning could give us the power need to match complex requirements. ❖ Semantic-Cognitive-Perceptual Computing enables contextualization, personalization, and abstraction for Augmented Personalized Health.
  40. 40. 41 Special Thanks Hong Yung (Joey) Yip (Graduate Student)

×