It has been five years since IBM’s Watson soundly defeated the all time human champions of the television game show Jeopardy! Watson has evolved since then, and other modern AI/cognitive computing technologies and platforms have emerged, but we still find that most people who watched the Jeopardy! tournament have no idea how Watson actually won. Some think Watson was almost magical - as in generalized artificial intelligence - while others see it as a clever parlor trick that simply exploited high speed search of massive data sets. In this webinar, participants will learn about the strategies, algorithms, and some of the subsystems that were developed to generate and test hundreds of hypotheses and rank confidence in potential answers within three seconds. We will also look at some of the challenges inherent in representing terabytes of data to optimize analysis, and discuss opportunities to leverage deep QA in consumer and enterprise applications.

1. Copyright (c) 2016 by STORM Insights Inc. All Rights reserved. 7/17/2015
Deep QA (Question/Answer)
Lessons From Watson and Jeopardy!
October 13, 2016
Adrian Bowles, PhD
Founder, STORM Insights, Inc.
info@storminsights.com

2. Copyright (c) 2016 by STORM Insights Inc. All Rights reserved.
Deep Question/Answering - Lessons from Watson & Jeopardy!
The Game
The Challenge
Scope of the problem
DeepQA Architecture & Processes
Software, Hardware & Resources
Next Steps

3. Copyright (c) 2016 by STORM Insights Inc. All Rights reserved.
Answers must be given in the form of a question
Last contestant to answer correctly chooses the next question
Correct responses must satisfy the demands of both the clue and the
category
Jeopardy
Six categories, 5 Questions for each category, $100-500 based on
difficulty
Double Jeopardy
Six categories, 5 Questions for each category, $200-1,000 based on
difficulty, and 3 hidden questions allow the person who chooses them to
bet everything they have at that point in the game
Final Jeopardy
Player must have a positive balance from the previous round to play
Players see the category and then decide - secretly - how much to wager
The question is presented
30 seconds to answer
Playing the Game:
Wikipedia, The Free Encyclopedia. October 12, 2016, 02:40 UTC.
Available at: https://en.wikipedia.org/w/index.php?title=Jeopardy!&oldid=743931483. Accessed October 12, 2016.

4. Copyright (c) 2016 by STORM Insights Inc. All Rights reserved.
Open Domain, broad use of language - Jeopardy! questions often involve puns, ambiguity…
IBM reviewed a sample of 20,000 questions, and found 2,500 distinct lexical answer types (LA
No single LAT accounted for more than 3% of the total
For each category, there could be thousands of questions
Best players provide correct answers ~85% of the time
Best players know what they don’t know - base their bets on their confidence
~3 seconds to answer questions
Challenges of Jeopardy! for Machines:
Players may only use the data/knowledge they have on arrival - no lifelines, resources…
Constraint
Winning Jeopardy! requires a contestant to answer
~70% of the questions, with 80%+ precision.

5. Copyright (c) 2016 by STORM Insights Inc. All Rights reserved.
Predicting lexical answer types in open domain question and answering (qa) systems
US 20130035931 A1 2013, Ferrucci, Gliozzo, Kalyanpur

6. Copyright (c) 2016 by STORM Insights Inc. All Rights reserved.
Precisio
n
Speed
Confide
nce
Quality
SpeedCost
Business Constraints Jeopardy! Constraints

7. Copyright (c) 2016 by STORM Insights Inc. All Rights reserved.
Look for Similar
Solved
Problems
Accept or Create
Problem Statement
Generate
Hypotheses
Identify Evidence
in Corpus
Score
Evidence
Score
Hypotheses
Present
Results
Get
Feedback
Train
ModelOrientAct
Observe
Decide
World
Model
Formalizing the Decision-Making Process
Boyd’s Loop
John Boyd (1927-
1997)
Continuous Learning

8. Copyright (c) 2016 by STORM Insights Inc. All Rights reserved.
Machine
Learning
NLU NLG
Informatio
n
Retrieval
Reasoning
Knowledge
Represent
ation
Evidence
Gather Decide
Evaluate Weigh
Generate
Hypothese
s
Automating QA

9. Copyright (c) 2016 by STORM Insights Inc. All Rights reserved.
* Building Watson: An Overview of the DeepQA Project, AI Magazine, Fall 2010 Issue,
Ferrucci, Brown, Chu-Carroll, Fan, Gondek, Kalyanpur, Lally, Murdock, Nyberg, Prager, Schlaefer, Welty.
Build a database of question/answer pairs
Build a formal model of the world
Build a search engine
What they didn’t do:
What they did:
DeepQA - “a massively parallel probabalistic evidence-based architecture.”*
Develop reusable NLU tech to analyze text
Analyze sources - structured and unstructured - to capture background knowledge
Apply knowledge representation and Reasoning (KRR) to the resulting structured knowledge
Use machine learning to generate and score hypotheses

10. Copyright (c) 2016 by STORM Insights Inc. All Rights reserved.
Massively Parallel Probabalistic Evidence-based Architecture

11. Copyright (c) 2016 by STORM Insights Inc. All Rights reserved.
Content Acquisition
Building the corpus
For Jeopardy! this had to be
completed before the game
commenced.
Ingested encyclopedias,
dictionaries, thesauri,
newswire articles, literary
works, databases,
taxonomies, ontologies…
IRL, we can identify and use new resources
based on the problem at hand.

12. Copyright (c) 2016 by STORM Insights Inc. All Rights reserved.
Question Analysis
What is being asked?
Question classification:
any words with double
meanings?
Puzzle question, factoid…?
Detect
focus
LAT
relations

13. Copyright (c) 2016 by STORM Insights Inc. All Rights reserved.
Relation-detection
“They’re the two states you could be reentering if
you’re crossing Florida’s norther border.”
Category: Head North
borders(Florida, ?,x,north)

14. Copyright (c) 2016 by STORM Insights Inc. All Rights reserved.

15. Copyright (c) 2016 by STORM Insights Inc. All Rights reserved.

16. Copyright (c) 2016 by STORM Insights Inc. All Rights reserved.
Hypothesis Generation
& Scoring
Use a candidate answer with the
question, try to prove correct with
a degree of confidence supported
by the evidence.
Scoring may use a variety of
relationships:
temporal
spatial
geospatial
taxonomic classification
correlation between candidate
and question…

17. Copyright (c) 2016 by STORM Insights Inc. All Rights reserved.
Evaluating Potential Answers
Watson scores evidence in
multiple dimensions
What works for a factoid question
may not work for a puzzle question.
“Chile shares its longest
land border with this country.”

18. Copyright (c) 2016 by STORM Insights Inc. All Rights reserved.

19. Copyright (c) 2016 by STORM Insights Inc. All Rights reserved.

20. Copyright (c) 2016 by STORM Insights Inc. All Rights reserved.
Merging & Ranking
Identifying the most likely
answer based on confidence
scores.
Answer scores are merged
before ranking and
confidence estimation.
Uses ML approach to
compare with training set
data when confidence scores
in different categories result
in “too close to call” results.

21. Copyright (c) 2016 by STORM Insights Inc. All Rights reserved.
Wikipedia, The Free Encyclopedia. October 12, 2016, 17:06 UTC.
Available at: https://en.wikipedia.org/w/index.php?title=Watson_(computer)&oldid=744021754. Accessed October 12, 2016.

22. Copyright (c) 2016 by STORM Insights Inc. All Rights reserved.
Software
Apache Hadoop
http://hadoop.apache.org
Apache UIMA - Unstructured
Information Management
Architecture
http://uima.apache.org
IBM DB2
Linux (Suse Enterprise Server 11)

23. Copyright (c) 2016 by STORM Insights Inc. All Rights reserved.
Resources
Wordnet(R) Princeton University "About WordNet." WordNet.
Princeton University. 2010. <http://wordnet.princeton.edu>
Wordnet(R)

24. Copyright (c) 2016 by STORM Insights Inc. All Rights reserved.
Resources
Wordnet(R) Princeton University "About WordNet."
WordNet. Princeton University. 2010.
<http://wordnet.princeton.edu>
Wordnet(R)

25. Copyright (c) 2016 by STORM Insights Inc. All Rights Reserved. 9/28/2011
IBM Power 750
90 servers, 32 cores/server,
2880 Cores in 10 racks
16Tb RAM
~80TeraFLOPS
80,000,000,000,000FLOPS
Hardware

26. Copyright (c) 2016 by STORM Insights Inc. All Rights reserved.
Next Steps…

27. For more information:
Copyright (c) 2016 by STORM Insights Inc. All Rights reserved.
adrian@storminsights.com
Twitter @ajbowles
Skype ajbowles
Upcoming Webinar Dates & Topics
November 10 Emerging Hardware Choices for Modern AI Data Management
December 8 Leverage the IOT to Build a Smart Data Ecosystem
2017 Webinar Themes
Technology Trends
Market Trends
Communicating
Learning
Understanding
Reasoning
Planning