In this webinar we covered the results of a recent Retail UX Benchmark Study that included Nordstrom, Neiman Marcus, Saks Fifth Avenue and Lord & Taylor. Nate also gave tips on how to create your own successful competitive benchmarking study.

1. Webinar:
Retail UX Benchmarking: Which Brand
Provides the Best Website Experience?
Nate Colker| UserZoom
12/13/2012

2. Intros
Alfonso de la Nuez Nate Colker
Co-Founder & Co-CEO UX Researcher
Twitter Hashtag #uzwebinar
Introduction Study Design Results Tips

3. Overview
• Defining UX Benchmarking
• Luxury department store Competitive Industry Study
• Tips for building your own competitive benchmarking study
Introduction Study Design Results Tips

4. Define Benchmarking
UX Benchmarking is the process
“Benchmarking is the
of comparing a website’s (or any
process of comparing digital product)
one's business performance, specifically in
processes and relation to UX metrics, including
performance metrics to among others
industry bests or best effectiveness, efficiency, overall
practices from other satisfaction and quality
industries.” perception, to another that is
widely considered to be an
industry standard benchmark or
best practice
Introduction Study Design Results Tips
http://en.wikipedia.org/wiki/Benchmarking

5. Definining UX Benchmarking & Metrics
Task Completion
Task Confidence
Number of Clicks Task Level - Single Ease Question
Number of Page Views Test Level - System Usability Scale
Net Promoter Score
Task Time Satisfaction
Introduction Study Design Results Tips

6. Define measurable & comparable UX-related KPIs
 Jeff Sauro,
Measuringusability.com
Introduction Study Design Results Tips

7. Why do we Care about Measuring User Experience?
Ordering a Pizza Online (1994)
Introduction Study Design Results Tips
http://articles.chicagotribune.com/1994-08-22/news/9408230031_1_pizza-hut-santa-cruz-operation-internet

8. Why do we Care about Measuring User Experience?
Ordering a Pizza Online (2012)
Introduction Study Design Results Tips

9. The World of User Experience Testing
Introduction Study Design Results Tips
http://www.useit.com/alertbox/user-research-methods.html Chart by Christian Rohrer

10. Unmoderated Usability Testing
Introduction Study Design Results Tips

11. Unmoderated Usability Testing
Introduction Study Design Results Tips

12. Case Study: 4 Brands
Introduction Study Design Results Tips

13. Study Summary
• Between subjects design
• 50 Participants per brand (200 total)
• 2 Tasks + survey questions
• Randomized task order
• Recruitment criteria
– Must shop online
– Shopped at luxury department stores
• Total study length : ~14 minutes
• Used a panel vendor
• 3-4 days to field the study (11/20-11/23)
Introduction Study Design Results Tips

14. Task Descriptions
Task 1 Objective: measure findability of a common apparel item (“scavenger hunt task”)
Task Description
Please find the price of a grey wool dress by Eileen Fisher. Please note down the price, as
you will be asked for it later.
Task 2 Objective: Access desirability and perceived quality of the sales section (“open task”)
Task Description
Please take your time to explore the sale section and look for things that are of interest to
you. Once you feel you have spent an adequate amount of time exploring this section, click
on the “I’m Finished” button. We will then ask you some questions about your experience
with the sale section of Nordstrom.com.
Introduction Study Design Results Tips

15. Case Study Preview
Introduction Study Design Results Tips

16. Task 1 Completion Rates: Find a dress
Task Success Rate
100%
90% 84%
80% 74%
72%
70%
60% 54%
50%
40%
30%
20%
10%
0%
Nordstrom Neiman Marcus Saks Lord & Taylor
The completion rates on Nordstrom, Saks and Lord & Taylor were significantly better than on Neiman Marcus (90% Confidence Level)
Introduction Study Design Results Tips

17. Disaster Rates (Highest Confidence but Failure)
100%
8% 4%
14% 16%
90% 12%
80% 20%
10%
70% 32%
60%
50% Disaster
Failure
40% 84%
74% Success
72%
30%
54%
20%
10%
0%
Nordstrom Neiman Marcus Saks Lord & Taylor
Introduction Study Design Results Tips

18. Task-Level Ease
Overall, how difficult or easy was it to find the price of a grey wool dress?
Very Easy
7
6 5.9
6
5 4.8
3.9
4
3
2
Very Difficult
1
Nordstrom Neiman Marcus Saks Lord & Taylor
Introduction Study Design Results Tips

19. Open-ended responses from participants who
thought the task was difficult (1, 2 or 3)
Briefly describe why you chose the number you did.
Nordstrom
 Too many choices. Small print beneath the pictures.
 I couldn’t find it and wasted a lot of time trying.
 I never found the grey wool dress therefore I couldn't know the cost. This is obviously a failure
at the task asked of the survey. I gave myself the lowest score possible, not for lack of trying
thought!
Neiman Marcus
 I could not find the dress. I searched under eileen as the designer. I typed in grey wool dress in
the search box and only two childrens dresses came up. I looked under sale items too and found
three dresses by this designer non of which were wool.
 I put Eileen Fisher grey wool dress in search bar and search query came up with "can not find
Eileen Fisher grey wool dress" so I put in search Eileen Fisher wool dress and I found dresses
none were grey.... I then looked through other pages and did not see a grey dress.
 Search didn’t work - came back with not found, didn’t have intuitive drop-downs, under
designers didn’t have dresses as option. No description said was a grey wool dress, closest was
gray 3/4 length dress. Took to long to weed through options looking for dresses by Eileen.
Introduction Study Design Results Tips

20. Open-ended responses from participants who
thought the task was difficult (1, 2 or 3)
Briefly describe why you chose the number you did.
Saks Fifth Avenue
 The Saks website kept sending me to Saks Direct - it took a number of attempts (7? 8?) to get to
dresses, then find the red wool dress assigned. It was very frustrating. Had the site worked
well, it should have been easy.
 Number of steps to complete task
 It was not a difficult site to navigate, but I would rather narrow down a category rather than to
search for a specific item using the search box
Lord & Taylor
 Its the only grey dress that said it was wool. it kept bringing up all different thing unrelated.
 Because i could not figure out which brand of grey wool dress it was.
 It took time to find the right dress, and I’m not completely sure it was the right one
Introduction Study Design Results Tips

21. Task Time: Completed tasks only
350
300 293
250 236
200
Seconds
146
150
126
100
50
0
Nordstrom Neiman Marcus Saks Lord & Taylor
Completion time on Lord & Taylor was significantly lower than it was on Nordstrom & Neiman Marcus (90% Confidence Level)
Introduction Study Design Results Tips

22. Post task questions
Strongly Agree
7
6.3 6.2 6.2 6.2
6 5.9
6
5.5
5.2 5.1
5 4.7
4.4 It was easy to navigate the
4.1 website
4 It was clear how to start finding
the price of the wool dress
3 I am satisfied with the amount of
time it took to complete the task
Strongly Disagree
2
1
Nordstrom Neiman Marcus Saks Lord & Taylor
Introduction Study Design Results Tips

23. Task 2
Evaluate the sales section
Please take your time to explore the sale section and look for things that
are of interest to you. Once you feel you have spent an adequate amount
of time exploring this section, click on the “I’m Finished” button. We will
then ask you some questions about your experience with the sale section
of Nordstrom.com
Introduction Study Design Results Tips

24. Sale Section Evaluation
How satisfied are you with the number of items that were on sale?
Very Satisfied
7
6
5.5
5.2 5.3
4.9
5
4
3
Not at all Satisfied
2
1
Nordstrom Neiman Marcus Saks Lord & Taylor
Introduction Study Design Results Tips

25. Sale Section Evaluation
How satisfied are you with the sale prices?
Very Satisfied
7
6
5
3.9
4 3.7
3.3
3
3
Not at all Satisfied
2
1
Nordstrom Neiman Marcus Saks Lord & Taylor
Introduction Study Design Results Tips

26. Sale Section Evaluation
Strongly Agree
7
6.2 6.2
6
6 5.7 5.7
5.4 5.5
5.1
It was easy to navigate the sale
5 4.8
4.5 section
4 3.9
4 I found some items that I liked
3 I would consider checking out
the sale section again in the
future
Strongly Disagree
2
1
Nordstrom Neiman Marcus Saks Lord & Taylor
Introduction Study Design Results Tips

27. Final Questionnaire:
What’s your attitude towards the brand?
7
6
5 4.8 4.9 4.8 4.8
4.6
4.3 4.4
4.1
4 Before
After
3
2
1
Nordstrom Neiman Marcus Saks Lord & Taylor
Introduction Study Design Results Tips

28. Final Questionnaire:
Brand Perception
Nordstrom Neiman Marcus
Pre Post Pre Post
1. Trendy 30% 62% +32% 1. Expensive 60% 80% +20%
2. Good Value 8% 26% +18% 2. Trendy 34% 48% +14%
3. Friendly 12% 30% +18% 3. Trustworthy 24% 18% -6%
4. Exciting 18% 32% +14% 4. Customer 20% 8% -12%
Focused
Saks Fifth Avenue Lord & Taylor
Pre Post Pre Post
1. Trendy 50% 62% +12% 1. Expensive 48% 82% +34%
2. Expensive 80% 88% +8% 2. Trendy 22% 52% +30%
3. Exciting 16% 12% -4% 3. Exciting 8% 26% +18%
4. Trustworthy 28% 12% -16% 4. Good Quality 42% 58% +16%
Introduction Study Design Results Tips

29. Final Questionnaire
Nordstrom Neiman Saks Fifth Lord &
Marcus Avenue Taylor
The website is easy to use
5.8 5.1 6.2 6.3
I feel comfortable
purchasing from this
5.2 4.3 5.1 5.2
website
I find the website to be
attractive
5.5 5.3 5.5 5.6
The website has a clean
and simple presentation
5.7 5.3 5.1 5.9
I will likely visit this
website in the future
4.5 3.6 3.9 4.4
On a scale from 1 to 7 Study Design
Introduction Results Tips

30. Final Questionnaire:
Net Promoter Score
NPS = % of Promoters
(9s and 10s) - Detractors
(0 through 6)
0 1 2 3 4 5 6 7 8 9 10 NPS
Nordstrom
10% 0% 2% 6% 6% 16% 14% 12% 18% 10% 6% -38%
Neiman Marcus
8% 10% 10% 4% 8% 18% 10% 12% 18% 6% 6% -56%
12% 2% 10% 0% 2% 18% 16% 10% 10% 6% 14% -40%
Saks Fifth Avenue
Lord & Taylor
10% 0% 2% 6% 6% 16% 14% 12% 18% 10% 6% -38%
Detractors Promoters
Introduction Study Design Results Tips

31. Final Questionnaire:
What did you like the most about the site?
Nordstrom Neiman Marcus
“I really liked that you can change the colors of “The search function quickly found what I was
different items without being forced to open that looking for.”
particular item, taking you to another screen.”
“I like how the pictures of the apparel would show
front and back when I hovered the mouse over
“That I could click on the pictures and see the the picture.”
back of the garment.”
Saks Fifth Avenue Lord & Taylor
“The amount of items I could see on one window. “It was easy to navigate and find products I was
I hate hitting next over and over again.” looking for.”
“Garment being worn by a moving person” “Search options. By Designer By Color By Price.
“Easy to find anything. love the search/browse by Very Easy and nice to use.”
size, color etc.”
Introduction Study Design Results Tips

32. Final Questionnaire:
What did you like the least about the site?
Nordstrom Neiman Marcus
“Small print beneath the pictures.” “The filters by designer only includes size color
“Hard to search compared to other sites I have etc. not dresses, tops etc.”
been on.”
“Drill through for finding the dress was difficult.”
“The models are ultra slim, it can be discouraging “It was difficult to navigate when searching for
for someone who isn’t a size 2 or size 0.” something specific.”
Saks Fifth Avenue Lord & Taylor
“A little too expensive to buy more than one or “The prices were a little steep.”
two items.”
“They were very expensive. I also felt that for the
“The prices, but that is not the website” price you would think they would have more
trendy items of clothing. I didn’t find anything that
Interested me.”
Introduction Study Design Results Tips

33. We have an exciting
announcement about
a new feature…

34. Exciting announcement of new feature

35. Some Final Tips for Creating your own Benchmark
 Task Selection
 Standardized design
 Randomize Conditions
 Cleaning the Data
Introduction Study Design Results Tips

36. Selecting the Right Tasks
A Balance of User Data and Business Goals
What are users coming to the sites to do (core tasks) ?
Are there any tasks that are required to use your site?
What drives revenue on your site?
Introduction Study Design Results Tips

37. Selecting the Right Tasks
A Balance of User Data and Business Goals
What are users coming to your site to do (core tasks) ?
- Browsing new sale items
Are there any tasks that are required to use your site?
- Creating an account?
What drives revenue on your site?
- Completing a purchase
Introduction Study Design Results Tips

38. Standardize Tasks Across Competitors
Introduction Study Design Results Tips

39. Randomize Task Order
Task 1
Screener Pre-task questions Randomized Final Questions
Task 2
Control for ordering effects
• Participants become more proficient as they use the site etc.
There are exceptions, sometimes tasks must be completed in order
Introduction Case Study Results Analysis

40. Clean the Data
Watch out for Cheaters
• Very Low task times
• 0 Clicks
• Nonsense Answers
• Enable Quality Controls
Introduction Study Design Results Tips
http://en.wikipedia.org/wiki/File:Cheating.JPG by Hariadhi

41. Take this with you today:
• Triangulate, use multiple metrics
• Experimental design
• Clean data before analysis
• Want to learn more?

42. Thanks so much for your attention!
Time for Q&A
Follow us on Twitter @userzoom