In this webinar, Jeff Sauro and Alfonso de la Nuez discuss how to: - Combine both quant plus qual data and benchmark UX - Be leaner and cost-effective in your research - Tell a complete story about UX and convince your stakeholders with both stats and video recordings

1. UserZoom Session Replay
An Automotive Case Study
Jeff Sauro | Measuring Usability LLC & UserZoom
2/26/13
#uzwebinar
Measuring Usability LLC

2. Quick intro to UserZoom & UZ Session Replay
• All-in-one Enterprise software solution that helps
Businesses cost-effectively test, measure and improve
UX over websites & mobile apps.
• Specialists in online or remote UX research & testing
• Scale your UX research, save time, money, effort
• While still obtaining very rich insights
• 45 employees, offices in Sunnyvale (CA) Manchester
(UK), Munich (DE) and Barcelona (Spain)
• Work for 50% of Fortune’s Most Admired Brands
Measuring Usability LLC

3. About Remote Unmoderated User Testing
• Like online surveys… ‘on steroids’
• Test large samples, from all over the place
• Natural context
• Ask users to complete tasks, sort
items, click here…
• Track usability metrics (success rates, time
on task), satisfaction, and also behavior
• Lots of automation, lots of time saving, $
saving too
• Lots of metrics, charts, user feedback, etc.
Measuring Usability LLC

4. About Remote Unmoderated User Testing
Measuring Usability LLC

5. About Remote Unmoderated User Testing
How about VIDEO?
Measuring Usability LLC

6. Introducing UZ Session Replay
• You can collect UX metrics such as task success ratios, time on
task, number of clicks and number of unique pages viewed…
• …and now, with UZ Session Replay, in the same study you can
also get video recording of each user session.
• UX Storytelling just got a whole lot better!
• Combine both QUANT + QUAL
• The Lab, at times, just doesn’t seem to add that much
value
• Filter videos by task, questionnaire answers or success/fail per task,
bookmark important events or actions, create highlight clips for your
reports, record particular URLs or block particular URLs from
recording (https for example), Share the videos with others
Measuring Usability LLC

7. Introducing UZ Session Replay
Measuring Usability LLC

8. Introducing UZ Session Replay
Measuring Usability LLC

9. Measuring Usability LLC
9

10. Measuring Usability LLC
10

11. 1981 Toyota Camry
Measuring Usability LLC
11

12. 2013 Toyota Camry
Measuring Usability LLC
12

13. Background and Purpose
• Finding the monthly payment for a car
is hard.
– Task Completion rate < 40%
– Harder than ~80% of all tasks
• Why? What causes that?
– Is it site specific?
– Is this just a difficult task?
• Multi-step of finding the right car with the right features
• Then calculating the payments
– Is it both or something else?
Measuring Usability LLC 13

14. Find the Monthly Payment for a 2013 Toyota Camry
$485
$477
$458
$492
$458
102 Users Each Randomly Assigned 3 Websites
How much would your monthly payment be for a 2013 Toyota Camry
Hybrid LE Series in zip code 90210 with a $2000 down payment, a
3.89% interest rate over 5 years with a 8.75% sales tax.
Measuring Usability LLC 14

15. Monthly Payment Completion Rates
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
35% 38% 17% 19% 33%
0%
Edmunds KBB Cars.com Autotrader Toyota
The 2 part task was made harder on Cars.com and Autotrader because finding the price of the basic model was more difficult
Measuring Usability LLC 15

16. Task Level Ease (SEQ 1-7 Standardized Question)
Extremely Easy
100%
90%
80%
70%
60%
50%
40%
30%
Extremely Difficult
20%
10%
22% 29% 31% 10% 22%
0%
Edmunds KBB Car.com Autotrader Toyota
Users rated Cars.com higher than the others in terms of ease, even though Cars.com had the lowest completion rates. Cars.com gave users a
false sense of confidence
Measuring Usability LLC 16

17. Disasters [High Confidence & Task Failure]
100%
9% 9%
13% 13%
90%
24%
80%
70%
52% 49% 57%
60%
72% Disaster
50%
Error
59%
Success
40%
30%
20% 38%
35% 33%
10% 19%
17%
0%
Edmunds KBB Cars.com Autotrader Toyota
Cars.com has the highest disaster rate, users felt a false sense of confidence on Cars.com
Measuring Usability LLC 17

18. Edmunds Videos
Measuring Usability LLC 18

19. Edmunds (phase 1) 33% Completion Rate
7%
11%
8%
8%
4%
40%
Measuring Usability LLC
19

20. Why was this task hard? – Edmunds
Categorized open ended responses
“couldn’t find a way
to calculate a
monthly payment
without entering
contact information
for a call-back or
email”
Measuring Usability LLC 20

21. To The Other Videos
Measuring Usability LLC 21

22. KBB (phase 1) 38% Completion Rate
10%
17%
13%
Measuring Usability LLC
22

23. Why was this task hard – KBB
Categorized open ended responses
90%
“KBB didn’t let me
80% choose a specific
model/make on the
payments estimator
70% page. I had to first
search for the Toyota
60%
Camry Hybrid LE”
50%
40%
30%
20%
10%
69% 19% 13%
0%
Hard me finding car/price of car Non-Intui ve website Could not find calculator
Measuring Usability LLC 23

24. Why was this task hard? – Cars.com
Categorized open ended responses
80%
“Website didn’t let
me choose the car I
70% wanted to finance (or
I couldn’t find that)
and I had to guess at
60% the purchase price”
50%
40%
30%
20%
10%
50% 25% 15% 10%
0%
Difficulty finding price of the Hard Task/Did not know where Slow Loading Time Could not find the calculator
car to start
Measuring Usability LLC 24

25. Why was this task difficult? – Autotrader
Categorized open ended responses
60%
finding the hybrid model was
difficult
50%
40%
30%
20%
10%
41% 32% 14% 8% 5%
0%
Hard to find the price of Could not find the Hard website to Navigate A lot of steps Did not want to enter
the car calculator personal informa on
Measuring Usability LLC 25

26. Why was this task difficult? – Toyota
Categorized open ended responses
60% The Toyota website seemed
to slow my computer down
considerably.
50%
40%
30%
20%
10%
30% 26% 22% 13% 9%
0%
Slow website/hard to Calculator was hard to Could not find price Could not find Could not find price of
navigate use without dealers quote calculator car
Measuring Usability LLC 26

27. Phase 2 : Eliminate the Car Finding
57 Users Each Attempted the Same Task on Both Websites
(Randomized)
Phase 1
How much would your monthly payments be for a 2013 Toyota Camry Hybrid LE Series in
zip code 90210 with a $2000 down payment, a 3.89% interest rate over 5 years with a
8.75% sales tax.
Phase 2
How much would your monthly payments be for a 2013 Toyota Camry Hybrid LE
Series priced at $28,500 in zip code 90210 with a $2000 down payment, a
3.89% interest rate over 5 years with a 8.75% sales tax.
Measuring Usability LLC 27

28. Completion Rates (Phase 2)
80%
70%
60%
50%
40% Original Task
Given Price
30%
20%
10%
35% 57% 38% 53%
0%
Edmunds KBB
The completion rate for Edmunds went up 22% points when the participant was given the price of the car. It went up 15% for KBB
Measuring Usability LLC 28

29. Task Level Ease (Phase 2)
Extremely Easy
70%
60%
50%
40%
Original Task
30% Given Price
20%
Extremely Difficult
10%
22% 20% 29% 47%
0%
Edmunds KBB
KBB is perceived to be much easier when the price is given because they do not give you the option to select your vehicle from the calculator
like Edmunds. Instead, if you navigate to the calculator first, you then have to search for your car price, therefore eliminating that step
increases the task ease.
Measuring Usability LLC 29

30. Why was this task Difficult? – Edmunds (Phase 2)
Categorized open ended responses
90%
“I wasn’t shown any
tools to help me
80% calculate my monthly
payment. I had to do
it on my own.”
70%
60%
50%
40%
30%
20%
10%
68% 23% 9%
0%
Hard to Find Calculator Site was Confusing Popups
Finding the calculator and getting coaxed by new car quotes is what continues to make this a hard task
Measuring Usability LLC 30

31. Time on Task Success Only (Phase 2)
360
300
240
180 Original Task
Given Price
120
60
252 253 227 108
0
Edmunds * KBB
In Phase 2, the time it took to complete the task on KBB dropped significantly and statistically
Measuring Usability LLC 31

32. Edmunds (phase 2) 57% Completion Rate
2%
11%
9%
8%
51%
Measuring Usability LLC
32

33. KBB (phase 2) 52% Completion Rate
31%
13%
4%
Measuring Usability LLC
33

34. Summary
Benefits of having Video
1. Having a Record of the Interaction
2. Verify Task Completion
By relaxing the success criteria the completion rate went from
47% - 57%
3. Understand Causes of Failure
Where and why?
How much of the difficulty is finding the right car?
Between 15 and 22 percentage points: Still only ~50% completing
How much is it the problem with the task design?
Slight changes to the calculator swing monthly payments
Changing Zip Codes and a specific car users aren’t interested in are
artificial
Measuring Usability LLC 34

35. About Jeff Sauro & Measuring Usability LLC
Measuring Usability LLC is a usability and quantitative research firm based in
Denver, Colorado USA focusing on the statistical analysis of human
behavior and quantifying the user experience for Fortune 1000
companies.
DenverUX.com
Sept 18-20th
www.MeasuringUsability.com
@MeasuringU
Measuring Usability LLC 35

36. Appendix
Measuring Usability LLC 36

37. Website’s Capabilities
The website’s capabilities meet my requirements.
Kbb 3.44
Cars.com 3.33
Edmunds 3.23
Toyota 3.07
Autotrader 2.56
1 2 3 4 5
Completely Disagree Completely Agree
Measuring Usability LLC 37

38. Time on Task – Success Only
420
360
300
240
180
120
60
252 227 219 144 156.5
0
Edmunds KBB Cars Autotrader Toyota
Autotrader was the fastest, though not statistically
Measuring Usability LLC 38

39. Experimental Study
• How difficult is this task across a number of websites?
– Across all the websites we looked at, there were low completion rates, proving
that across platforms, it is a difficult task
• Edmunds everyone gets distracted by the Price Quote pop-ups and pages
• KBB, Cars.com and Autotrader do not let you select the car from the calculator
• Toyota has a lot of graphics, but is hard to navigate through
• How much of the difficulty is finding the right car?
– The two step process of finding the right car and then finding the calculator adds to the difficulty of
the task. On the websites that are more geared towards car inventory instead of research, finding
the basic price of the car to input into the calculator is difficult
• How much is it the interaction with the payment calculator?
– It is less the interaction with inputting figures into the calculator that is difficult, instead it is when
you cannot choose a car FROM the calculator that makes this step of the task difficult.
– Due to the artificiality of the task, uses have ore trouble entering in the correct figures than if they
were doing this on their own. A small typo or mistake would look like a task failure
• How much is it the problem with the task design?
– More just the artificiality of if you were going to search for a car, you would not be changing things
like your zip code or your sales tax
• Is there a mismatch with the users mental model?
Measuring Usability LLC 39

40. SUPR-Q Items & Factors
Usability
This website is easy to use.
It is easy to navigate within the website.
Credibility (Trust, Value & Comfort)
This website keeps the promises it makes to me.
I feel confident conducting business with this website.
Loyalty
How likely are you to recommend this website to a friend or colleague? NPS
I will likely visit this website in the future.
Appearance
I found the website to be attractive.
The website has a clean and simple presentation.
Measuring Usability LLC 40

41. SUPRQ & NPS (Phase 1)
60%
40%
20%
19%
6% 10% 1% 10%
0%
Edmunds KBB Cars.com Autotrader Toyota
-5%
-20% -38% SUPRQ
-41%
-52% NPS
-63%
-40%
-60%
-80%
-100%
Across all sites, users have low NPS and low SUPR-Q scores which illustrates that this is a difficult task on any website.
Measuring Usability LLC 41