Spotlight talk given by Eric Brown and Rob High at Watson DevCon 2016. Since the beginning of the computer age, scientists have been working to improve the ways computers understand us, reason through problems, and essentially ‘think’— ideas that represent the heart of artificial intelligence. In 2011, IBM took a giant leap forward with Watson, a cognitive computing technology that could answer Jeopardy! questions with the same level of skill as a human grand champion. Since then, Watson’s capacity to understand, interact, reason, and learn has grown exponentially. Here’s an insider's look at the origin of Watson— and its future.

1. From Jeopardy! to the Future
Eric Brown, PhD
Director, Watson Algorithms
IBM Watson Health
Rob High, Jr.
IBM Fellow, Vice President, CTO
IBM Watson

2. Please note IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice and
at IBM’s sole discretion.
Information regarding potential future products is intended to outline our general productdirection and it should
not be relied on in making a purchasing decision.
The information mentionedregarding potential future products is not a commitment, promise, or legal obligation to
deliver any material, code or functionality. Information about potential futureproducts may not be incorporated into
any contract.
The development, release, and timing of any future features or functionality described for our products remains at
our sole discretion.
Performance is based on measurements and projections using standard IBM benchmarks in a controlled
environment. The actual throughput or performance that any user will experience will vary depending upon many
factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O
configuration, the storage configuration, and theworkload processed. Therefore, no assurance can be given that
an individual user will achieve results similar to those stated here.

3. The History of Watson
Eric Brown, PhD
Director, Watson Algorithms
IBM Watson Health

4. A Brief History of AI
1945
Memex:
Vannevar
Bush
1970 1980 1990 2000 2010
Natural Language Processing
Information Retrieval
Machine Learning
Knowledge Representation and Reasoning
Question Answering
Dartmouth
Conference of
1956
State of the Art in
Question Answering
1st AI Winter
Reasoning as Search
Natural Language Understanding
Logic
Neural Networks
2nd AI Winter
Expert Systems
Knowledge and Reasoning
Text REtrieval
Conference
Big Data
Computational Power

5. Real Language is
Real Hard
Chess
– A finite, mathematically well-­defined search space
– Limited number of moves and states
– Grounded in explicit, unambiguous mathematical rules
Human Language
– Ambiguous, contextual and implicit
– Grounded only in human cognition
– Seemingly infinite number of ways to express the same meaning

6. $200
The juice of this bog fruit is
sometimes used to treat
urinary tract infections
$400
An early name for Google
was this type of massage
The Jeopardy! Challenge
Broad/Open
Domain
Complex
Language
High Precision
Accurate
Confidence
High
Speed
$600
In cell division, mitosis splits
the nucleus & cytokinesis
splits this liquid cushioning
the nucleus
$800
Grace Murray Hopper is credited
with applying this 3-­letter term to
a mysterious computer problem
$1000
IBM is known informally as Big
this
A palpable, compelling and notable way to drive the technology of Question Answering
along Key Dimensions

8. What It Takes to compete against Top Human Jeopardy! Players
Our Analysis Reveals the Winner’s Cloud
Winning Human
Performance
2007 QA Computer System
Grand Champion
Human Performance
Top human
players are
remarkably
good.
Each dot – actual historical human Jeopardy! games
More Confident Less Confident
In 2007, we committed to
making a Huge Leap!
Computers?
Not So Good.

9. Large Hand-­Crafted Models won’t cut it
– Too Slow, Too Narrow, Too brittle, Too Biased
– Need to acquire and analyze information from As-­Is Knowledge sources
Intelligence from the combination of many
– Consider Many Hypotheses. Reduce early biases.
– Consider Many diverse algorithms. No single one is perfect or complete.
– Analyze evidence form different perspectives
– Balanced combination is continually learned, tested and refined
Architecture to Support Integration and Learning
– Facilitate integration of many components
– Automatically learn how to combine
– Enable scale-­out and parallel deployment
Drive Performance with Measurement and Analysis
– Define appropriate metrics
– Measure, analyze, improve, repeat
Early Philosophical Commitments

10. DeepQA: The architecture underlying Watson
Generates many hypotheses, collects a wide range of evidence and balances the combined confidences of over 100 different
analytics that analyze the evidence from different dimensions
Answer
Scoring
Models
Answer &
Confidence
Question
Evidence
Sources
Models
Models
Models
Models
ModelsPrimary
Search
Candidate
Answer
Generation
Hypothesis
Generation
Hypothesis and
Evidence Scoring
Final Confidence
Merging & Ranking
Synthesis
Answer
Sources
Question &
Topic
Analysis
Evidence
Retrieval
Deep
Evidence
Scoring
Learned Models
help combine and
weigh the Evidence
Hypothesis
Generation
Hypothesis and Evidence
Scoring
Question
Decomposition

11. 11
Goal-­Oriented Metrics and Incremental Investments
– Identify a Target and Technical Approach
– Headroom Analysis: Estimate idea’s potential impact on key metrics
– Balance long-­term & short-­term investments. Have the next priority ready. Be Agile.
Extreme Collaboration
– Implemented “One Room” to optimize team work and communication
– Immediate access to the right “expert”, spontaneous discussions, no good idea lost
Disciplined Engineering and Evaluation (Regular Blind Data Experiments)
– Bi-­weekly End-­to-­End Integration Runs & Evaluations (Large Compute Resources)
– >10 GBs of error analysis output made accessible via Web-­Based Tool
– Positive impact on last run required to get into the next bi-­weekly run
Rapid Innovation Methodology Emerged
>8000 Documented experiments performed in 4 years

12. DeepQA: Incremental Progress in Answering Precision on the
Jeopardy Challenge: 6/2007-­11/2010
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Precision
% Answered
Baseline 12/06
v0.1 12/07
v0.3 08/08
v0.5 05/09
v0.6 10/09
v0.8 11/10
v0.4 12/08
v0.2 05/08
IBM Watson
Playing in the Winners Cloud
V0.7 04/10

13. With Precision, Accurate Confidence and Speed, the rest was History

14. From battling humans at Jeopardy! to transforming how business thinks,
acts, and operates
Contact Center
Healthcare Financial Services
Government
Diagnostic/treatment
assistance, evidenced-based
insights, collaborative
medicine
Investment and retirement
planning, institutional
trading and decision support
Call center and tech support,
enterprise knowledge
management, consumer insight
Public safety, improved
information sharing,
security

15. The Era of Cognitive
Computing
Rob High, Jr.
IBM Fellow, Vice President
Chief Technology Officer
IBM Watson

16. Cognitive systems amplify human cognition.

17. Cognitive systems understand human expressions – textual, verbal, visual
By reasoning about the actual intention or problem being addressed
They learn how to recognize patterns of meaning through examples and feedback
And they interact with humans on their own terms, and in a way that inspires people.
What is Cognitive Computing?
… and do so at enormous scale.

18. Watson Reference Model
Developer
Tooling
Platforms as a
Service
Watson Services
Data as a
Service
Watson Content
Application
Tooling
Maker
Tooling
Content
Tooling
Cloud Infrastructure
Public Private
Crowd
Sourced
Knowledge
Organization
Skills
Foundational
Cognitive
Skills
Higher
Reasoning
Skills
IBMWatsonMarketplace Skills as a
Service
Watson
Built Apps
& Bots
Watson
Built Skills
IBM Built
Apps & Bots
IBM Built
Skills
3rd
Party
Apps &
Bots
3rd
Party
Skills
Software as a
Service
Watson Apps &
Bots
Hybrid Client
Platform
Watson Explorer

19. Watson is available as a set of services delivered as APIs in the Cloud
Higher Reasoning Skills
• Conversation
Higher
Reasoning
Skills
Ibm.com/bluemix
Can be combinedwith the 100s ofother available services on Bluemix

20. • Add a natural language interface to your
application to automate interactions with
your end users.
• Common applications include virtual
agents and chat bots that can integrate
and communicate on any messaging
platform.
• Model is trained on user-­defined intents,
entities and dialogs
• Expanded to recognize the emotion of
the user and to respond accordingly
Conversation

21. • Intents classify the kinds
of actions or questions
that the conversation will
respond to
• You can begin with as few
as just one intent, and
then expand as new
functionality is added to
the virtual agent
• It only takes a few
examples to train Watson
to recognize a wide
variety of ways to express
an intent
• The tooling allows you to
test as you train
Conversational Intents

22. • Conversation can
detect entities in
an utterance and
identify them to
clarify the intent
• Typically used
with Dialogs to
condition the
response
Conversational Entities

23. • Dialogs can be
created to control
the flow of the
conversation
around specific
intents
Conversational Dialogs

24. Watson is available as a set of services delivered as APIs in the Cloud
Watson Foundational Skills are grouped into four categories
Ibm.com/bluemix
Foundational
Cognitive
Skills
Language
• Author
• Concepts
• Dates
• Entities
Speech
• Speech to Text
• Telephony Speech to Text
• Keyword Spotting
Vision
• Image Classification
• Face Detection and Attribution
• Celebrity Recognition
Empathy
• Personality Insights
• Tone Analyzer
• Emotion Analysis
• Text to Speech
• Expressive Text to Speech
• Relations
• Typed Relations
• Sentiment
• Taxonomy
• Text Extraction
• Feeds
• Keywords
• Language
• Microformats
• Publication Date
• Visual Text Recognition
• Similarity Searching
Can be combinedwith the 100s ofother available services on Bluemix

25. Tone Analyzer understands and helps fine tune your message
Uses psycholinguistics, emotion analysis and language analysis to assess Tone
Online Dating Profile
I'm a hard working adventurous, very talented man who's been caring
and helpful throughout my life, I like to travel, play my guitar, dance, and
cook, I love the beach, sailing my boat, and the outdoors.
I raised two great kids and now I'm starting a new chapter in my life.
Thanks.
What I’m doing with my life
Working toward a new goal, keeping fit, helping others, and traveling
whenever i get a chance.
I’m really good at Listening, enjoying the moment, and many other
things.
The six things I could never do without
Family, the ocean, intimacy, friends, adventure, music, love.
On a typical Friday night I am
Meeting with friends, listening to a band or playing my guitar, dancing or
just staying home with someone special and enjoying each other.
You should message me if
You're looking for a relationship with someone that likes to sail his boat,
ride bicycles, travel, swim, go to the beach, listen to music and enjoy
everyday pleasures together.

26. Emotional Analysis helps build empathetic systems
Uses state-­of-­the-­art machine learning models and feature engineering techniques to predict emotion labels

27. Personality Insights

28. Watson Visual Recognition Service

29. Watson Visual Recognition leveraged to
enhance the game of Pokemon Go

30. Enables speech that
reflects intended tone
– Expressive SSML
– Voice Transformation
SSML
Expressive Text to Speech

31. Watson is available as a set of services delivered as APIs in the Cloud
KnowledgeOrganizationSkills
• Watson Knowledge Studio
• Document Conversion
• Retrieve and Rank
Knowledge
Organization
Skills
Ibm.com/bluemix
Can be combinedwith the 100s ofother available services on Bluemix

32. Enable subject matter experts
and developers to teach
Watson the linguistic nuances
of industries and knowledge
domains
Watson Knowledge Studio

33. Uses machine learning to
improve search across
documents using natural
language questions
Retrieve and Rank

34. Amplifying Human Cognition
Requires that we affect people in a very
human way
Human can be influenced by cognitive
experiences
– Something that creates a sense of
presence with people
– Something that inspires people
– Something that embodies intellectual
cognition with behavioral cognition
Project Intu is a Platform for creating
great cognitive experiences
avatars In-the-walls
devices
spaces wearables
robotics
Go to the Project Intu workshop to get access to the experimental platform and to
learn how to program Intu to create great cognitive experiences

35. • Amplify human creativity
- Inspiring us to new alternatives to decision options
- Bringing the breadth of all human knowledge to the tip of our tongue
• Learn their behavior through formal and informal training processes
• Interact with humans on our terms – in the language of humans
• Demonstrate their expertise through trust and depth of character
• Evolve strategies of success – adapting to ever changing knowledge and
understanding
• Establish transformative relationships between humans and machines
In 5 years, cognitive systems will be to computing what transaction
processing is today

36. ibmwatson.com facebook.com/ibmwatson @ibmwatson