The document discusses the key technologies needed to build an enterprise virtual assistant, including automatic speech recognition, natural language understanding, dialog management, support for multiple languages, and analytics. It notes that an enterprise virtual assistant must be tailored specifically for business tasks and data within an organization, unlike a general-purpose virtual assistant. The document also provides examples of user interface designs for integrating voice capabilities into mobile business apps.

1. Copyright © 2017. All rights reserved.
Pardon My French
And Other Adventures on the Road to
Enterprise Virtual Assistants
Editt Gonen-Friedman
Oracle Voice & Emerging Technologies
Editt.gonen-friedman@oracle.com

2. Copyright © 2017. All rights reserved.
Voice Interaction
“Voice-based technologies are the most important area of growth
for mobile user interfaces… hands-free use and always-on
interfaces will drive increased use of speech recognition…
enterprise application developers will need to accommodate
new ways of accepting input”. - Intelligence report, May 2015,
Tractica
“Enterprises are going to be affected by a worker’s need to do
more than type, click and swipe” - ITWC
“2016 will be the year of Conversational Commerce” – Chris
Messina on Medium
2

3. Copyright © 2017. All rights reserved.
What Does it Take to Build an Enterprise Virtual Assistant?
(ASR)
Automatic
Speech
Recognition
Voice UI
Dialog
Management
(NLU)
Natural
Language
Understanding
3
• Multiple
technologies must
come together to
build it.

4. Copyright © 2017. All rights reserved. 4
• Needs SR
• Build your own- advantages:
– Build a massive language corpus
(Google)
– Handle surround sound, priority by
proximity (Amazon’s Alexa)
– Use voice biometrics to identify
speaker (Alexa, Nuance)
• Or use 3rd party services
A Mobile Enterprise VA
Image source: itpro.co.uk

5. Copyright © 2017. All rights reserved. 5
• Speech service considerations:
– Footprint: local install (Sensory) or cloud
service
– Security: enterprise data is sensitive
– WER (word error rate)
– Device support
– Languages (global enterprise)
– Vocab customization: ability to add
recurring entity names and industry
jargon
Automatic
Speech
Recognition
Mobile Enterprise VA

6. Copyright © 2017. All rights reserved. 6
• Compared to a general purpose VA
– Supported actions are limited
– Context is limited
• Is it easier?
• As a rule there’s less ambiguity, but sometimes
need to resolve to less popular meaning
• Example:
– The user says: “Leads” or “Go to leads”
– Intent: navigate to the leads page in my speech-enabled
mobile app for sales
Speech Considerations: Vocabulary

7. Copyright © 2017. All rights reserved. 7
– Result: “Go to Leeds”. A general purpose VA might understand this to mean “bring
up the map for Leeds, England”
Speech Considerations: Vocabulary
Leeds, Northern England
– In this case we had to add “Leads” to the ASR custom vocabulary with an increased
‘weight’ of 50% instead of 10%
• Could also be solved at the NLP step, with full NLP that resolves ambiguity

8. Copyright © 2017. All rights reserved.
A Mobile Enterprise VA
8
Automatic
Speech
Recognition
Voice UI
• Needs voice interaction
design
– How to make it look like a
speech app?
– How to deal with command
discoverability?
– Can you ‘wake it’ with a key
word?
– In what case do you allow
touch and voice combo?
– How to indicate ‘listening’?

9. Copyright © 2017. All rights reserved.
A Mobile Enterprise VA
9
– Here are some attempts to answer those questions in a dedicated speech app,
Oracle Voice

10. Copyright © 2017. All rights reserved.
A Mobile Enterprise VA
10
– And this is a UI change to a regular app, Oracle Sales Cloud Mobile, where speech
capabilities have been added

11. Copyright © 2017. All rights reserved.
A Mobile Enterprise VA
11
• Needs dialog management
– One-step response: gives you
a simple answer or link, or
navigates you to another page
– Multi-step dialog: manages a
back-and-forth dialog in which
context is retained
– Perhaps add more useful
interactions
• Such as business content reading
(news, emails, app listings)
Dialog
Management
Automatic
Speech
Recognition
VUI
Dialog
Management

12. Copyright © 2017. All rights reserved.
A Mobile Enterprise VA
12
• Needs NLU
• Many 3rd party solutions
available
– It’s possible to start with a
basic solution that
understands a number of
meanings and intents and can
follow up with specific actions
and taskflows
– Soon you’ll run into the need
for full language and context
intelligence
Dialog
Management
Automatic
Speech
Recognition
VUI
Dialog
Management
Natural
Language
Understanding

13. Copyright © 2017. All rights reserved.
A Mobile Enterprise VA
13
• Robust NLU needs to resolve
ambiguity in context
– Leads vs. Leeds is a simple example
– ‘Diversification’ means ‘investment
variety’ in Finance, but ‘getting rid of
assets’ in Marketing
Image source: Oracle Intelligent UX

14. Copyright © 2017. All rights reserved.
A Mobile Enterprise VA
14
• Adding an NLU solution to
the mobile app is no simple
task
– Test performance
– Word error rate
– Intent error rate
Image source: Right Now Intent Guide

15. Copyright © 2017. All rights reserved.
Are We Done Yet?
15
• Users want language
support
Dialog
Management
Automatic
Speech
Recognition
VUI
Dialog
Management
Natural
Language
Understanding
Languages

16. Copyright © 2017. All rights reserved.
Languages
16
Source: Technology Review
– Your speech service recognizes in 40 languages – why
doesn’t your app?
The user
asked how
I’m doing
Respond
that I’m
doing well
How are
you doing?
Speech
Engine
How are
you doing?

17. Copyright © 2017. All rights reserved.
Languages
17
Source: Technology Review
I have no
idea what
that means
Error
handling
Comment
allez-vous?
Speech
Engine
Comment
allez-vous?
– A user speaks French. SR output is French text.

18. Copyright © 2017. All rights reserved.
Languages
18
• A middle step is missing, a translation, or a mapping
• You could translate the text to English before further processing, or-
• You could add NLP in other languages
– When adding NLP in other languages you also essentially add a mapping between key
words in English that associate intent with actions, and the corresponding words in
the other supported languages.
Source: Technology Review

19. Copyright © 2017. All rights reserved.
Languages
19
Source: Technology Review
• Translation services work differently, using statistics on many translated
examples
• In a late 2016 blog post Googlers’ implied that Google’s AI translation tool
seems to have invented its own secret internal language, an internal
representation, a machine initiated mapping
– The tool was trained to translate between English and Korean, and between English and
Japanese
– The team found that the tool has spontaneously acquired the ability to translate
between Korean and Japanese
– Science fiction? Read here:
https://techcrunch.com/2016/11/22/googles-ai-translation-tool-seems-to-have-invented-its-own-secret-internal-language/

20. Copyright © 2017. All rights reserved.
Languages
20
Source: Technology Review
A visualization of the translation system’s memory when translating a
single sentence in multiple directions.

21. Copyright © 2017. All rights reserved.
Are We Done Yet?
21
• Users want AI
Source: Technology Review

22. Copyright © 2017. All rights reserved.
Analytics=AI
Automatic
Speech
Recognition
VUI Design
Dialog
Management
Natural
Language
Understanding
22
Languages Analytics
• Users want AI
• What they are really asking
for is analytics
• Simple analytics gives you
hindsight about what
happened

23. Copyright © 2017. All rights reserved.
What Does It Take to Build an Enterprise Virtual Assistant?
Automatic
Speech
Recognition
VUI
Dialog
Management
Natural
Language
Understanding
Languages
Descriptive
Analytics
Predictive
Analytics
Internet of
Things
Prescriptive
Analytics
Machine
Learning
23
• Descriptive analytics allows answering
more complex questions and gives you
insight about what’s happening
• Predictive analytics gives you foresight
about what will happen. It should also
pull data from the real world
• Prescriptive analytics tells you what to
do to get specific outcomes
• Machine learning makes sure the
system gets better and smarter with
every interaction

24. Copyright © 2017. All rights reserved.
That’s What It Takes to Build an Enterprise Virtual Assistant
Automatic
Speech
Recognition
VUI
Dialog
Management
Natural
Language
Understanding
Languages
Descriptive
Analytics
Predictive
Analytics
Internet of
Things
Prescriptive
Analytics
Machine
Learning
24
When will you be done?
Source: http://theegeek.com/artificial-intelligence/

25. Copyright © 2017. All rights reserved.
Editt Gonen-Friedman
Editt.gonenfr@gmail.com
Editt.gonen-friedman@oracle.com
https://www.linkedin.com/in/editt
25