- Learn how to "usability test" AI interactions with humans and measure success - Understand the two distinct ways that humans construct commands to AI systems and how, using physiological measurements, you can measure the human response to the AI system responses Description John Whalen explores the concept of cognitive design, describing how humans structure their commands to AI systems (syntax, word usage, prosody) and how to measure human reactions to AI responses using biometrics (facial emotion recognition, heart rate, GSR). Along the way, John shares insights into how to optimally architect the customer experience. John offers an overview of the results of an evaluation of four major AI systems (Siri, Cortana, Alexa, and Google Assistant), tested by the young and old, those new to AI systems and those that use these tools every day, native and non-native speakers, and techies and non-techies. Each were asked to interact with the systems to request facts, complex information, jokes, commands, and calendar information while the evaluators recorded their commands, the AI response, and the human’s physiological response to the AI response (facial emotion, heart rate, and GSR). There were several intriguing findings: - There were two distinct ways humans constructed commands for the AI systems. - The testers’ favorite AI systems were not always the ones that performed the best in terms of giving correct answers. - There was a distinct physiological signature associated with a positive experience. John explains how these findings can help you determine how you should measure the success of your AI system or chatbot and suggests new ways to predict market success that go beyond AI answer accuracy.

1. Next-generation
intelligent
applications require
cognitive design
John Whalen, Ph.D.
Chief Experience Officer
10Pearls
Presented at O’Reilly AI Conference, San Francisco, 2017

2. Brief Introductions

3. PhD Cognitive Science
Johns Hopkins University
Cognitive Neuroscience
Linguistics
Neural Networks/ML
Vision Science
Post Doc. at UCLA
during dot.com
boom
Professor in
Psychology
Univ. Delaware
Biometrics
Numerical Cognition
Founder, UX Lead
Brilliant Experience
User Insights
Digital Strategy
UX / CX
CXO
10Pearls
User Insights
Digital Strategy
AI + UX
John Whalen

4. According to Google Images, I am User Experience!

5. “Intelligent Experiences”
Business reimagined
- Significant development
- Data driven / AI-based
Experience reimagined
- Research, psychology
- Innovative UX/CX
+

6. An end-to-end digital experience and enterprise
software application partner.
Digital Experience &
Enterprise Mobile
Partner
Supplier of the Year:
Building Digital
Marketplace
50 on Fire – Hottest
Companies Award

7. Special thanks:
Max Kalicka, Researcher
Imran Aftab, CEO
Jen Foster, Director

8. Agenda
1. Why AI UX is so different
2. Designing for how people think
3. Introducing studies
4. Examples & results breakdown
5. Summary & implications

9. 1. Why AI UX is so different

10. Challenge 1:
What is a usability success?

11. “Okay Google,
show me a
waterfall.”
@johnwhalen #TheAIConf

12. “Okay Google,
order me a pizza.”
“Here are some
nearby pizza
restaurants…”
@johnwhalen #TheAIConf

13. Challenge 2:
Humanized experiences

14. “Ask Alexa to play some music.”
“Okay, I’ll ask her.”

15. Pronouns given
It It It
Her Him Her
@johnwhalen #TheAIConf

16. “Hey Siri,
do you love me?”
[ouch!]

17. 2. Design for how people think

18. Language Wayfinding
Vision /
Attention
MemoryEmotion
Decision
Making
Design for how people think
@johnwhalen #TheAIConf

19. Language Wayfinding
Vision /
Attention
MemoryEmotion
Decision
Making
@johnwhalen #TheAIConf
Use brain science to build intelligent experiences
What words and
word order were in the
command?
Could user
understand their choices and
where they are in the
system?
Did the AI match
expectations?
Did the AI generate
a positive emotional
response?
Did the answer help
to solve a problem?
Did AI focus the user
on the answer?

20. 3. Case in point: AI experiences

21. Prompt:
Chicago, second
tallest building
Participant:
“Alexa, what is the
second tallest
building in
Chicago?”
@johnwhalen #TheAIConf

22. Task types and AI assistants studied
Siri
Study1
Personal
Cortana Google
Assistant
Alexa
Google
Home
Alexa SiriHound
Study2
Personal+
Business

23. Ages
Younger Older

24. Experience with AI
AI experience No AI experience

25. Careers
Techie Non-techie

26. English fluency
Native English speaker
Non-native English
speaker

27. Types of requests
1. Simple facts
2. Complex facts
3. Commands
4. Games/Jokes
5. Business needs

28. Captured:
1. Request uttered
2. AI response
3. Facial expression
4. Heart rate
5. GSR (sweat)

29. 4a. Study 1 Results: Personal Tasks

30. Hotel near MOMA in NYC

31. Population of country with Eiffel Tower

32. 24hr grocery store near me

33. Do aliens exist?

34. Do you like me?

35. Answer accuracy by age
63% 68% 66% 65%
@johnwhalen #TheAIConf
Note: Command style variance

36. Answer accuracy by assistant
76% 68% 61% 61%
@johnwhalen #TheAIConf
Siri CortanaGoogle
Assistant
Alexa

37. Note: The most accurate AI tools
are not the most preferred.
Emotion matters!
@johnwhalen #TheAIConf

38. Accuracy and preference by assistant
76% 61% 68% 61%
@johnwhalen #TheAIConf
PREFERREDACCURACY
38% 33% 5% 5%
Siri CortanaGoogle
Assistant
Alexa

39. Preferred Google Assistant:
“Annoying when they are
being human. I don't feel
anything with them.”
“It is weird that they
have a personality and
can already tell a joke.”
“Just give me the answer”
“I want general information
and answers fast.”
@johnwhalen #TheAIConf

40. Preferred Alexa:
“Alexa feels like it is addressing
me back. Feels like I am
interacting with a person.”
“Prefer Alexa because
it tells me its not sure.”
“I like when it answers the
question the way I worded it.”
“Responses were direct.
Fun to banter back and
forth.”
@johnwhalen #TheAIConf

41. 4b. Study 2 Results:
Personal + Business Tasks

42. Task types & AI assistants
Google
Home
Alexa SiriHound
Study2
Personal+
Business

43. News on Johnny Depp

44. Flight status United 753, tomorrow

45. Find 24-hour grocery store

46. Microsoft’s last closing stock price

47. When to leave, arrive Dulles airport 10pm

48. May housing starts, up or down?

49. Text Max Kalicka “Meet me on July 10”

50. Answer accuracy by assistant
77% 74% 59% 50%
@johnwhalen #TheAIConf
Hound
Google
Assistant
Siri Alexa

51. Answer accuracy by task type
72% 58%
@johnwhalen #TheAIConf
Personal
Tasks
Business
Tasks

52. Accuracy and preference by assistant
50% 74% 59% 77%
@johnwhalen #TheAIConf
PREFERREDACCURACY
35% 21% 21% 14%
Google
Assistant
SiriAlexa Hound

53. Preferred Hound:
“I do love that when you ask
it a question it provides you
an answer in two options.”
“It remembered what I
was talking about.”
“I liked the way it asked more
than one question to get to
the answer.”
“It was better at the
back and forth.”
@johnwhalen #TheAIConf

54. Biometric signature of preferred tool
Heart
Rate GSR Positive
Affect
@johnwhalen #TheAIConf

55. Product designers:
We can detect which AI assistant
is preferred and if your assistant is
improving over time.
@johnwhalen #TheAIConf

56. 5. Summary & implications
@johnwhalen #TheAIConf

57. 1. The most accurate tools are not the most preferred.
50% 74% 59% 77%
@johnwhalen #TheAIConf
FAVORITEACCURACY
35% 21% 21% 14%
Google
Assistant
SiriAlexa Hound

58. 2. Biometrics can detect emotional preference.
Heart
Rate GSR Positive
Affect
@johnwhalen #TheAIConf

59. 3. There is an opportunity for business assistants.
72% 58%
@johnwhalen #TheAIConf
Personal
Tasks
Business
Tasks

60. 4. Design for how
people think.
1. Conversational
cadence
2. Context relevance
3. Emotional response
@johnwhalen #TheAIConf
Language Wayfinding
Vision /
Attention
MemoryEmotion
Decision
Making

61. Thank you!
John Whalen
@johnwhalen
linkedin.com/in/johnwhalen
John Whalen
Designing
for How
People Think
USING BRAIN SCIENCE TO BUILD BETTER PRODUCTS

62. Summary of Findings
1. The most accurate tools are not the most preferred.
Emotion matters!
2. Biometrics can detect emotional preference.
3. There is an opportunity for business assistants.
4. Humanize the experience. We need to design for how
people think.
@johnwhalen #TheAIConf