Keynote at LD4IE 2017, Linked Data for Information Extraction, co-located with ISWC2017.

1. IBM Research
IE in Action:
Capturing Social and
Clinical Knowledge for
Personalised Care
Vanessa Lopez
Health & Person-centered Systems
IBM Research Ireland

2. IBM Research
Cognitive AI -> Broad AI
Smart Cities
Integrated Care

3. IBM Research
Community curated KGs
(e.g. Wikidata)
Global-view
(proprietary) KGs
Web of Data
- Heterogeneous
- Size vs. Quality
- Multilingual
- Varying levels of trust
NLP, IE ..
Into the Gap
Providing answers to
questions without
engineering our own data
Language
Knowledge
Semantic Technology
• QALD [2011-2017]: DBpedia, interlinked KBs
• WebQuestions, Free917 (2013): Freebase.

4. IBM Research
QA over Linked Data
Core Techniques of Question Answering Systems
over Knowledge Bases: a Survey. KAIS 2017

5. IBM Research
But with most information
still unstructured..

6. IBM Research
What is the impact of
semantic technologies in
business and society?

7. IBM Research
Examples in QA: Watson Jeopardy
“WAR MOVIES: A 1902 Joseph Conrad work set in Africa inspired
this director to create a controversial 1979 war film.”
Answer (deepQA): “the heart of darkness” (a book inspired on “Apocalypse now” a
movie directed by FF Coppola)
“Structured analytics are a natural complement to unstructured in that they
cover a narrower range of questions but are more precise within that range”
Structured data and inference in DeepQA. A. Kalyanpur, B. Boguraev, S.
Patwardhan, et al. IBM Journal of Research and Development, 56(3):10, 2012.

8. IBM Research
Successful QA systems
• Key#1 Strong KA to mine the web for facts
• Key#2. Do not assume a completely and precisely translated
representation of a question to find and combine pieces of knowledge in
a way that is meaningful for the task at hand
How to approach Data Integration?

9. IBM Research
Knowledge, not just Data
• How to approach Data Integration?
– Just don’t do it ! Too hard

10. IBM Research
Knowledge, not just Data
• How to approach Data Integration?
– Just don’t do it ! Too hard
– Use a common unique model “Any truth is better
than indefinite doubt”

11. IBM Research
Knowledge, not just Data
• How to approach Data Integration?
– Just don’t do it ! Too hard
– Use a common unique model “Any truth is better
than indefinite doubt”
– Be willing to accept noise "I'd rather walk with a
doubt than with a bad axiom"
In Watson: Not a single component does all the job. A ML algorithm
learns how to combine multiple methods that do similar jobs in
unpredictable ways to provide inexact solutions that are
meaningful for the task at hand.

12. IBM Research
Cognitive technologies promise to have significant
societal impact in domains where there is a need to
transform multidisciplinary information across
systems into actionable services.
Knowledge is not the destination:
“The level of advancement of a society
is often measured in terms of
protection of the less able”
Use Case: Integrated Care

13. IBM Research
Cognitive AI=
Semantics + NLP + Learning
Care Manager Doctor
Docs/Notes
Clinical
DBs
Social
DBs
Vanessa Lopez LD4IE– ISWC 2017
DATA / INFORMATION
LAYER
MODELS:
Descriptive/ Predictive
To expand / augment
human cognition
What is the role of technology?

14. IBM Research
Integrated Care: Business Value
• Containing costs while improving outcomes from coordinated social and
health care services has been identified as a 21st century societal gran
challenge
• Patient-centered (high-need, high-cost)
• Team-based approach that rewards quality and outcomes
Socio-economic factors drive health
and disease:
"It cost us one million dollars
not to do something about
Murray,"

15. IBM Research
Meet the users
• Task: Captures information about the needs of her
patients, creates personalized care plans and
coordinates a care team.
Susan Brown – Care Manager• Pain points:
Pain
Points
Short
amount
time
Know-
ledge
sharing
Overload
with data
Relevancy
personali-
sation
Digging
info/
insights
Filling in
gaps
Laurie – elderly patient
Vanessa Lopez LD4IE– ISWC 2017
• Goal:
• Support a care team to make better informed decisions

16. IBM Research
Data, data, data: a 360◦ person view
Time
Care Network
Care Team Social Network
Symptoms
Diagnoses
Medication
Labs
..
Clinical
ADL
Social
ADL
Behavi
oral
Mental
health

17. IBM Research
Capture knowledge and learn best practices:
Decrease the cost of information seeking
How to to support care workers gain a
comprehensive social and clinical picture of a
patient?
How to learn from the actual practice of care
professionals to suggest actionable insights?
Hill#1
Note Highlights: Surfacing Relevant Concepts from
Unstructured Data for Health Professionals. ICHI 2017
health
food
safety
shelter education
income

18. IBM Research
Right info at the right time:
Present comprehensive information with enough evidence
Hill#2
How to make this information available for
a care professionals in a natural way?
QuerioDALI: Question Answering over Dynamic
And LInked knowledge graphs. ISWC’16

19. IBM Research
Data ingestion & lifting
19
Enterprise Data
Open Data /
Linked Data Models
NL Query
Highlights &
suggestions
Semantic QA
[Hill 2]
CASE
NOTES
Capture knowledge and
best practices [Hill 1]
Answer generation
Family
(Social Care)
Records
Social Care
System
Patient
Social
Care
Record
Social
Vocabulary
Patient
EMR
Clinical
Vocabulary
Healthcare
System
DBpedia
(Places, Things)
W3C
Vocabulary &
metadata
Safety
Net
Building Blocks
Speech to Text
Knowledge Graphs
NL
Understanding
Context
• LOD to ingest and organize
knowledge across tabular data:
proving common vocabularies,
generalize specialize terms and
acting as anchors
Incremental
linkage without
a unique model
but exploiting
heterogeneous
models

20. IBM Research
KA to build social context
• What are the social determinants of health for vulnerable populations?
• What are the resources available and the connections between them?
(Safety Net of providers and services)
Hospi-
tal_y
Belle-
vue
Nursing
home
Hospi-
tal_x
Belle-
view
Read. Rate
cardiology
NYC Hospitals Medicare USA
sameAs
Which hospitals with elderly care have the
lowest readmission rates for cardiology in NYC?
Data Access Linking and Integration with DALI: Building a
Safety Net for an Ocean of City Data. ISWC 2015

21. IBM Research
Susan
Parenting Skills
ProviderChild 1
Early Intervention
Specialist
Medical
Provider
Employment
Counselor
Child Care
Provider
Addictions
Counselor
Provider
Payment
Provider
Payment
Provider
Payment
Provider
Payment
Provider
PaymentProvider
Payment
Provider
Payment
Provider
Payment
TANF Food
Stamps
Foster Care
Provider
Child Welfare
Caseworker
Provider
Payment
Child 2
Boyfriend
Food
Stamps
UI
Payment
Child 3
Many roles,
information needs not
known in advance
1000’s of
sources,
impossible to
fully integrate
School
Vast amounts of information,
privacy restrictions
Domain
knowledge
is broad
(social) and
deep
(clinical)
Interdisciplinary elite team of care professionals working together can
reduce hospital readmission rates from 18% to 5%
The challenge is to scale the right practices to the whole organization
A Cognitive Care Mentor to capture knowledge and
best practices

22. IBM Research
A patient receiving multiple services accumulates a lot of case notes. It’s
easy to miss something. Notes Highlights builds a personalized list of
important concepts.

23. IBM Research
Notes Highlights Enables care professionals to quickly access key facts
(highlights) from pages of notes
• Curation to ensure the team gets an accurate picture.
• Collaboration by selecting surfacing most relevant facts.
Vanessa Lopez LD4IE– ISWC 2017
An entity-based
temporal view of a
patient organized by
semantic type

24. IBM Research
• Are we asking the right questions? What is the missing info?
• Learn from experienced care team (past history) and existent
knowledge to suggest relevant actions for a given patient
A Cognitive Care Mentor: Suggestions
• Prediction based on historical data:
• Frequent Pattern Mining
• Collaborative filtering
• Prediction based on literature:
• Word2Vec
• Semantic Recommender

25. IBM Research
Phone
IBM Watson Care Manager
Laurie
Thompson
Female 72 Years
Actions
Address
22 Chesnut Ave,
Boston, MA
02130
Phone
541 754-3010
Questionnaires
General
0 of 2
0 of 2
Suggestions
Cornerstone
ProgramsSummary Data History TeamPlan
Back
Hi Susan
Programs
Cornerstone Program
Insights DetailsRepeat
Complete Save
Do you feel that because of the time you spend with your relative that you don’t have
enough time for yourself?
Do you feel stressed between caring for your relative and trying to meet other
responsibilities (work/family)?
Notes Highlights
These highlights are based on relevant
information and evidence associated with
Laurie’s case.
Missed PCP Appointment
Difficulty Walking
Transport Problems
Demographic Summary
Options for Laurie’s journey
to her PCP
Transportation Suggestions
Options for Laurie’s journey
to her PCP
Do you feel strained when you are around your relative?
Do you feel uncertain about what to do about your relative?
Caregiver Burden Screening
Depression Screening (PHQ-9)
Caregiver Burden Screening
Suggests:
• Use suggestions to prioritize tasks / assessments / follow-ups
“Its all about the patient. Personalization is all about giving the
individual the power to choose - we don’t want to limit what they
choose to meet their goals”

26. IBM Research
Annotation Scoring
Feedback
Ranked
Entities
Notes
Notes
Annotators
Semantic
Reconciliat.
Notes
NotesTerminolo-
gies &
Ontologies
Prediction
Predictive
Engines
Patient
Profile
Ranking
loop
Suggestion
loop
Search
loop
Predictive
Engines
Predictive
Engines
Predictive
Engines
Underlying Innovation

27. IBM Research
Underlying Innovation: the value of semantics
The value of semantics:
• Organize and select relevant entities (semantic view)
• Abstract from annotators terminology and lexical differences (eye-drops
= ocular lubricant)
• Provides an integrated view (actionable types) for analytics insights
• To semantically maps entities to assessments / questionnaires
Db:Depression
Mood_disorders
Mental_behavioural_disorders
Mental_health Social_problems Human_diseases_disorders
Type Reasoning – DBpedia:

28. IBM Research
https://www.ibm.com/watson/developercloud/retrieve-rank.html
Vanessa Lopez LD4IE– ISWC 2017

29. IBM Research
Domain experts validation (gold standard)
• Scenario: finds all relevant and only relevant information
• Extract entities from notes as to what domain experts
would choose.
• Datasets: 20 clinical and social cases
• Judgements (ground-truth): 22 evaluators (4 per case)
• Manually highlighted all relevant annotations
• assigned a category to each
• top-10 for each case
• Metrics: 80% agreement, 64% P, 85% R, 73% F1
Criteria User annotation Gold-standard annotation
Remove function words She does the shopping shopping
Split different entities 23 yo female 23 yo, female
Temporal modifiers No past psychiatric illness; (No Past) medical illness ;
Negations Denies ear abnormalities (Denies) ear abnormalities
Measures A1c dropped from 13.0 to 10.4 A1c dropped (from 13.0 to 10.4)

30. IBM Research
Measure, Measure, Measure
Cognitive technologies aren’t mean to be 100% accurate,
how do we measure the real value to the user?
Key finding from this study:
Experts agree on what concepts are important
For important concepts, coverage is high (91%)
Next Step: Health field study
Can it improve productivity?: Care
Managers spend a large amount of time
reviewing notes prior to their interaction with
patients.
Can it reduce care gaps?: Ineffective
team communication cause a large percent
of all medical errors by missing key facts
What is the impact of learning (training)?

31. IBM Research
What information are you looking for in your notes?
“What we talked about last time, their goals, interventions, concerns, labs”
Are notes from other team members of interest to you?
“I’m interested if someone has added a note after I spoke to them”
How long does it take you to review notes?
“Brand new patients maybe 30 minutes, otherwise maybe 10 minutes”
Interviews with Domain Experts
Do you want to see what other team members mark important?
“If a physician marked this as important then yes, that’s very important
for me to see”
“I would like to flag (’push to the top’) those things that are the most
relevant about Laurie just now.”
“I’d want to know more about it - why its there”
“I want the patient to feel I know about
them. They expect you to know about them.”
-- Care Transition Navigator,
“This will save us so much time”
-- Care Coordinator
CM would follow words that the system may
have mistakenly pulled. Time costly.
If it’s not an easy 1-click we won’t get feedback

32. IBM Research
Validation: Observations & Inhibitors
• P/R trade-off: affected by noise (21%), non relevant entities (17%
of entities had no agreement) and lack of models’ coverage
– Ambiguous acronyms (e.g., PCP), partial annotations (e.g. normal):
– Keywords are not facts: annotations typically missed:
• Factual changes and actions: “(lose) one-half pound of weight” , “lost
his job”, “stop taking the insulin”, “eats too much in the evening”, “left
side of her face is dropping”
• Feelings, progress and emotional status: “doing OK”, "achieve that
goal”, “(really)overwhelmed”, “did not mind dying”, “inflated self worth”
• Some (social) entities and complex entities: "ball of both feet" ,
“running”, “lives with mother”, daughter assists with meds”.
• Not enough context:
– “He quit smoking several years ago but he picked up the habit recently”
– “Personalizing context requires lots of domain knowledge (and reason
with negation and temporality)

33. IBM Research Hill#2: Right info at the right time
• User’s needs not known in advance. Explore natural ways to answer
complex information needs across KGs, even without training data.

34. IBM Research
QA pipeline
Semantic
Entity Search
Is Eplerenone having side effects for
Teresa’s conditions?
Dependency tree
Be
(verb)
Eplerenone
(noun)
side effects
(noun)
subj
for (prep)
pred
have
(verb)
conditions
(noun)
Teresa
(noun)
objprep
mod
modmod
Deep Parsing
NE / NLP
pipeline
Pattern
engine
Graph Pattern
(GP) Search
Merge & Rank
candidate GPs
PAS:
<side effects, Eplerenone> <side effects, condition> <condition, Teresa>
Inspra
(Eplerenone)
Type_2_
Diabetes
(?sjoin)
sideEffect (?p1)
Side_effects
rdf:type
pre-diabetes
(?s)
Skos:
closeMatch
Condition
treatmentFor(?p2)
10334
(?ent)
rdf:type
Teresa
activity
(?p3)
dbp:Diabete
s_mellitus
Owl:sameAs
Graph#Sider
Graph#WCMPersona
Graph#DBpedia
Answer: yes!

35. IBM Research
What is really novel ?

36. IBM Research
What’s next for cognitive technologies?
• Validating: understanding notion of value - it can fail but still be useful …
– Metrics based on explicit and implicit user behavior (clicks, logs)
– Does it requires lots of training to get it up to speed?
– Novelty: “tell me something I don't know” (with evidence)
• Cognitive = non-definitive non-deterministic results
– Weigh information from multiple sources and past actions
– “Knowledge” is enhanced as new data arrives or humans interact with
the system
Medication: metformin
Conditions: chronic kidney disease,
diabetes type 2
Patient: Laurie contraindicated !
Cognitive AI

37. IBM Research
• Explainable and trustable AI
– Intelligible systems: systems that explain themselves
(clinician and computer science barrier)
– Present enough evidence to built trust in the AI
– Advice to experts vs. patients
• Active Learning: in response to users interactions or
actions to evolve knowledge
– Leverage user explicit and implicit feedback
– KA with open and domain independent dialog systems
has been a longstanding goal of AI
– without a fixed ontology or domain model that
predetermines what users can say
What’s next?
Vanessa Lopez LD4IE– ISWC 2017
Which patients
have a thyroid
disorder and have
not had their TSH
tested in the past
1 year?
Is TSH use to
detect thyroid
disorder?

38. IBM Research BlueMix services

39. IBM Research
Other AI research projects
• Deep analysis of behavioral literature and policies in to
extract relevant information: Entities in context (relations)
Human Behavior Change Project: aims to build an AI system to scan the literature
on behavior change, extract key info, and build a model of human behavior to
answer : ‘What interventions work, how well, for whom, in what setting, for what
behaviors and why?’ http://www.ucl.ac.uk/human-behaviour-change
Program Integrity: to understand unstructured policies and built rules to help a policy
investigator detect uncompliant claims by providers (Fraud Waste and Abuse)

40. IBM Research
Thank you!
The truth is Rarely Pure and
Never Simple
Oscar Wilde