This document discusses speech application tuning, which is the process of analyzing recorded caller interactions to identify areas for improving speech recognition accuracy and the caller experience. It involves transcribing utterances and categorizing them as correct or incorrect recognitions. Focus areas are identified, such as points where callers exceed maximum tries or opt out. Recommendations are made and implemented, such as adjusting prompts, grammars, or confidence thresholds. Tuning is an ongoing process to adapt the system to callers and ensure success. Guidelines recommend addressing low-hanging fruit through objective, methodical testing of targeted changes.

1. interactivenw.comCUSTOMER EXPERIENCE SOLUTIONS
Interactive Northwest, Inc.

2. interactivenw.comCUSTOMER EXPERIENCE SOLUTIONS
Demystifying Speech Application Tuning:
What It Is, and Why It’s Important

3. interactivenw.comCUSTOMER EXPERIENCE SOLUTIONS
Learning Objectives
• Understand speech application tuning and its
benefits
• Learn the mechanics of the tuning process
• Learn how to use tuning results to improve
overall caller satisfaction and success
• Apply these learnings to any application use
case/vertical.

4. interactivenw.comCUSTOMER EXPERIENCE SOLUTIONS
We bring over 25
years of experience in
the IVR industry
developing products
of excellence tailored
to our customers’
specific and diverse
needs.
Experience
Our attention to
detail and high
quality products
stand out from our
competitors.
We are thorough in
development, testing
and delivery.
Quality
We stay abreast of
new technologies and
are in a constant state
of refining our
products to meet the
demands of an ever-
changing IVR
landscape.
Innovation
Who We Are
LumenVox technology is used worldwide by companies like Verizon, GM, Toyota, Kmart, the Aussie
Post, British Telecom, government agencies, over 300 banks, telecom service providers, 511
operations, and nearly 2,000 other resellers and developers.
LumenVox speech automation solutions have won
numerous awards for innovation and technical
excellence. Based on industry standards, LumenVox
Speech Software is certified as one of the most
accurate, natural sounding, and reliable solutions in
the industry.

5. interactivenw.comCUSTOMER EXPERIENCE SOLUTIONS
Agenda
TUNING
What is
it?
How is it
done?
Examples
What
not to
do
What to
look for

6. interactivenw.comCUSTOMER EXPERIENCE SOLUTIONS
Introduction: What is Tuning?
What is speech
application
tuning?
Tuning is the process of analyzing
recorded caller interactions and
using that data to inform
application changes that will
improve speech recognition
accuracy and caller experience.

7. interactivenw.comCUSTOMER EXPERIENCE SOLUTIONS
Introduction: What is Tuning?

8. interactivenw.comCUSTOMER EXPERIENCE SOLUTIONS
Introduction: Why Tune?
True or False?
The goal of tuning is to
make users interact with
the application in a way
that guarantees success.
TRUE
FALSE
The goal of tuning is to adapt
the SYSTEM to the caller!
Note: the caller dictates the conversation. If the system fails to meet the caller's
needs, it's not the caller who has failed; it's the speech application.

9. interactivenw.comCUSTOMER EXPERIENCE SOLUTIONS
Introduction: Why Tune?

10. interactivenw.comCUSTOMER EXPERIENCE SOLUTIONS
Introduction: Definitions
1.In–Grammar (IG) and Out–of–Grammar (OOG) are labels that look
at whether the speech engine matches a path in the grammar with
what the caller actually said. If it can, then the spoken words are
considered In–Grammar, if not, the spoken words are considered.
2.Confidence Scores indicate the speech engine's certainty about the
answer it returns.
3.Confirmations are dialog techniques to help the speech
application avoid making a mistake, in cases where the results are
ambiguous.

11. interactivenw.comCUSTOMER EXPERIENCE SOLUTIONS
Tuning: Understanding the Process
Identify Focus Areas
Enable Audio Capture
Transcribe Utterances
Analyze Results
Implement Recommendations

12. interactivenw.comCUSTOMER EXPERIENCE SOLUTIONS
Step One: Identify Focus Areas
Where are callers
exceeding max
tries?
Where are callers
opting out?
What feedback are
agents hearing from
callers?
IVR Data

13. interactivenw.comCUSTOMER EXPERIENCE SOLUTIONS
Step Two: Enable Audio Capture
Always verify that utterance capture is working soon after enabling it!
“Your call may be
monitored or
recorded.”
For How Long?
Depends upon:
- Call Volume
- Seasonality
- Menu Frequency

14. interactivenw.comCUSTOMER EXPERIENCE SOLUTIONS
Step Three: Transcribe Utterances
How Many?
More is never too much!
(We recommend minimum 500 per menu)

15. interactivenw.comCUSTOMER EXPERIENCE SOLUTIONS
Step Three: Transcribe Utterances
…but, how?
1.Listen
2.Transcribe
3.Categorize
Correct-Accept
Correct-Reject
False-Accept
False-Reject

16. interactivenw.comCUSTOMER EXPERIENCE SOLUTIONS
Step Three: Transcribe Utterances

17. interactivenw.comCUSTOMER EXPERIENCE SOLUTIONS
Step Three: Analyze Results
Prompt Wording
Issues
Grammar Under-
and Over-Coverage
Pronunciation
Difficulty
Confidence
Thresholds
User Experience
Problems
Do users
understand the
options?
Are they saying
unexpected
things?
Are we rejecting
valid utterances?Are the keywords
difficult to say?

18. interactivenw.comCUSTOMER EXPERIENCE SOLUTIONS
Step Four: Make and Implement Recommendations
Provide good cues and guidelines,
so callers choose the pathway you
designed for the application.
Prompt Verbiage
Eliminate items that are unused or
causing ambiguity and consider
adding items that don’t exist.
Grammar Items
In extreme cases, modify the
application flow by adding or
removing dialogs.
Code Changes
Adjust confidence thresholds and
input timeout intervals to optimize
application/recognizer performance.
Configuration Updates

19. interactivenw.comCUSTOMER EXPERIENCE SOLUTIONS
Tuning Example: Path of Least Resistance
The Issue: 62% of callers say “I don’t know
it” or “I don’t have one”. Callers are
failing at this prompt and sound frustrated.
The Solution: Reduce the perception of failure
by giving callers an easy way to choose their
preferred method of identification.
The Recommendation
And Why It Works: “Do you know
your contract
number?”
“To get started, I’ll
need to get your 8-
digit contract number,
which is printed on
your contract. Please
say or enter it now.”

20. interactivenw.comCUSTOMER EXPERIENCE SOLUTIONS
Tuning Example: Elegant Error Recovery
62%
The Issue: Callers are saying Yes/No
instead of selecting an option
The Solution: Treat Yes responses as if caller
selected the first item in the list
The Recommendation
And Why It Works:
<Okay, ship an item.>
A vehicle, personal
household goods, or
something else?
Do you want to
ship an item, or
hear delivery
status?
Alternate Solution: Coach the voice talent to
record the options with more emphasis

21. interactivenw.comCUSTOMER EXPERIENCE SOLUTIONS
Tuning Example: Prompt Efficiency
The Issue: The second most popular option
is “Other”, which results in a transfer—not
self-service. This is known as “The
Recency Effect”.
The Solution: Reduce the prompt length and
style it in a list format which encourages
barge-in.
The Recommendation
And Why It Works:
“What type of claim are you calling
about? You can say:
Appliances. Heating. Air
Conditioning. Plumbing. Electrical.
Water Heater. Garage Door Opener.
Pools and Spas. <pause>
Or, ‘Something Else’.
“Now, I'll need to know the
type of claim you're calling
about. Please say “Air
conditioning” or press 1,
“Heating” or press 2,
“Appliances” or press 3,
“Water Heater” or press 4,
“Plumbing” or press 5,
“Electrical” or press 6,
“Garage Door Opener” or
press 7, “Pools and Spas” or
press 8, or say “Other” or
press 9. [pause] To hear these
choices again, say “repeat”.

22. interactivenw.comCUSTOMER EXPERIENCE SOLUTIONS
Tuning Example: Prompt Clarity
The Issue: 25% of utterances are Out-Of-
Grammar. This was due to callers saying
the date or time INSTEAD of the option #.
The Solution: Provide better instructions to
users, increasing their understanding and
chance of success.
The Recommendation
And Why It Works:
When you hear the
appointment you want, say
the NUMBER of its place in
the list. For example, for
the second appointment,
say TWO.
For <date/time>, say one.
For <date/time>, say two.
For <date/time>, say three.
To hear that again, say
Repeat. To hear more
appointments, say More
Options.

23. interactivenw.comCUSTOMER EXPERIENCE SOLUTIONS
Prompt Design: Best Practices
1. Sensory memory (2 seconds) vs. short-term memory (30 seconds)
2. Magic number: 7 (plus or minus 2) is the general rule of information
retention
3. Chunking: User memory is best exploited when data is grouped based on
meaning, sound, or timing
4. Primary effect: Users remember the first item in a list
5. Recency effect: Users remember the last item in a list (guess which options
are usually last in a prompt?)
6. Vicarious learning depends on the Observe > Imitate cycle. Callers often
learn quickly how to behave after a single transaction
7. Distractions impede learning/memory. Avoid distracting accents or jargon
by voice talent
IMPORTANT: No Match/No Input prompts can erase sensory memory.
“I’m sorry, I didn’t catch that. Let’s try again…”
A better no match prompt would be:
“Can you say that again?”

24. interactivenw.comCUSTOMER EXPERIENCE SOLUTIONS
Prompt Design: Best Practices
If your newspaper is
damaged, say one.
Not delivered, two.
Wet, three.
If your newspaper is wet,
say one.
All other damage, two.
If it was never delivered,
three.
Creates a specific-
to general conflict
Better!

25. interactivenw.comCUSTOMER EXPERIENCE SOLUTIONS
Prompt Design: Best Practices
What’s the approximate length of
the box in inches? Please round
off to the nearest inch.
What’s the approximate width in
inches?
And, what’s the height?

26. interactivenw.comCUSTOMER EXPERIENCE SOLUTIONS
Guidelines
 Be Objective
 Be Methodical
 Utilize a large dataset
 Address low-hanging
fruit
 Test changes thoroughly
WHAT TO DO
Χ Make changes based
on a single (or few)
instances
Χ Get caught up in huge
redesign efforts
Χ Overly-complicate the
user experience
WHAT NOT TO DO
*Assume that 40–50% of total development and
deployment time will be spent on the tuning process

27. interactivenw.comCUSTOMER EXPERIENCE SOLUTIONS
Summary
1. Enable utterance capture on speech server
2. Download logs and recorded .wav files
3. Transcribe and categorize
4. Analyze results
5. Recommend changes
6. Follow best practices
7. Repeat!
30-90 Days After
Deployment
Are things working as
expected?
Do users understand the
options?
Are the prompts
soliciting the right
responses?
Has the popularity of
options changed?
Have new business
needs arisen?
Are there recurring
problem areas?
Once A
Year

28. interactivenw.comCUSTOMER EXPERIENCE SOLUTIONS
Visit our website at
www.interactivenw.com
or give us a call at 1-800-732-3236
Whether it’s a custom application or
one of our award-winning products, we
have a solution that is right for you.
Learn More About INI’s Innovative Solutions