Slides used for RUSA webinar on September 4, 2013

1. How to Conduct Usability Studies
A Librarian Primer
Tao Zhang, PhD
Digital User Experience Specialist
Purdue University Libraries
zhan1022@purdue.edu
Sponsored by ALA RUSA
#RUSAUX
Ilana Barnes
Business Information Specialist
Purdue University Libraries
ibarnes@purdue.edu

2. Learning Objectives
2
• Participants will be able to describe the usability
concept and it's applications to libraries.
• Participants will be aware of the basic workflows
and techniques needed when doing usability tests.
• Participants will be aware of resources for usability
studies and where to find out more.

3. Agenda
3
• What is a usability study?
• Usability study process
• Conducting the study
• Analyzing Results
• Communicating
• Where to find out more

4. About Us
Ilana Barnes Tao Zhang
4

5. Tell us about you!
What type of library do you work in?
A) Public
B) Academic
C) Other
How familiar are you with Usability Studies/Tests?
A) Very familiar
B) Somewhat familiar
C) Not familiar but excited to learn!
Chat box: Areas you especially want us to talk about
5

6. What is User Experience?
6

7. Definition
Usability study refers to“evaluating a
product or service by testing it with
representative users.“
- http://www.usability.gov/how-to-and-tools/methods/usability-testing.html
7

8. How UX fits into Reference Services
• Decisions that user experience research can help
you with:
• What should our libraries front page look like?
• What do our undergraduates expect from the library?
• What would we like our undergraduates to expect from the library
that they do not know or understand or need?
• Understanding your users aims and needs help you
help them and show your impact at the same time.
8

9. How to plan your usability study topic
• What are my objectives?
• How much time do I have?
• How many teammates do I have?
• Who am I presenting this to?
• Is this for an existing product?
9

10. When to Test
• Early or middle stages of development
• Types:
Exploratory | Assessment | Comparison | Validation
• Start preparing at least three weeks before actual
test
• DO pilot tests
10

11. Usability Study Process
11
Recrui'ng
Preparing
Conduc'ng
Analyzing
Presen'ng
Choosing Features
Creating Tasks
Preparing Test Script

12. Recruiting
• Determining the target audience (behavior)
• Finding representative members
– Behavioral criteria, demographics
– Avoid sampling bias
• Convincing participants to come
– Incentives
– Informed content (duration of study, data to collect)
• Scheduling
– Reminding
– Deal with no shows
12

13. Recruiting
Best people to invite:
• People who are going to need the service in the
near future
• People who have used a competing/previous
version service in the recent past
Number of participants:
• 5-8
• Also consider representativeness and tasks
13

14. Choosing Features
• Input from: Development team | Stakeholders | Users
• Content inventory & task analysis
14

15. Creating Tasks
Tasks should be:
• Reasonable and doable
• Described in terms of end goals
• Specific
Run through tasks to see how long each will take
Determine what quantitative data to collect
15

16. Discussion
You need to do a usability study of your library’s
website, because users find it difficult to find
information they want on the website.
What are some possible tasks you can use in the
usability study?
16

17. Writing A Study Script
• Introduction (5 min)
• Warm-up interview, demographics survey (5 min)
• Scenarios and tasks (20-30 min)
• Interview (10-15 min)
• Wrap-up (5 min)
17

18. Conducting the Study
The physical layout with observer
18

19. Conducting the Study
The physical layout without observer
19

20. Conducting the Study
TechSmith Morae www.techsmith.com/morae.html
20
Recorder
Observer
#1
Participant Test Moderator
Observer
Manager
Observer
#2
…

21. Conducting the Study
21
http://silverbackapp.com/

22. Conducting the Study
Testing on mobile devices
22
IPEVO Ziggi-HD High-Definition
USB Document Camera
DIY Mobile Usability Testing - SXSW
Interactive 2012
http://www.slideshare.net/beleniq/diy-
mobile-usability-testing-sxsw-
interactive-2012

23. Moderation
• Follow script but allow some flexibility
• Encourage participants to“think aloud”
• Ask about expectations
• Ask“why”a lot
• Suggest solutions when participants are stuck
• Discuss with observer right after test
23

24. Two Basic Techniques
Observation
Interview
24

25. Observation
• Watch, Understand
• Limit the time users have to perform a task
• Encourage users to think aloud with neutral
prompts
• Elicit detailed information
25

26. Interview
• Introduction
• Warm-up
• General issues
• Deep focus
• Wrap-up
26

27. Nondirected Interview
Prepare nondirected questions
• Focused on the person answering it
• Focused on experience, not extrapolation
• Concentrate on immediate experience
• Avoid judgmental language
• Focus questions on a single topic
• Keep questions open-ended
• Avoid binary questions
27

28. Nondirected Interview
Examples
“Is this a useful feature?”
“Is this feature valuable to the work you do right now?”
“Do you think this feature would be better if it was also
available on smart phones?”
28

29. Nondirected Interview
Running the interview
• Define (technical) terms
• Don’t force opinions
• Restate answers
• Never say the participant is wrong
• Work with a partner
29

30. Analyzing Results
• Collecting observations
– Notes from moderator and observer
– Quantitative information: task successfulness,
navigational path, number of steps, errors, time to
complete, subjective ratings (System Usability Scale)
– Qualitative feedback from participants
• Extracting trends
– Repeated concerns
– Multiple issues with possible common origin
– Identify effects on user experience
30

31. Analyzing Results
• A list of usability issues
– Quantitative and qualitative data as evidence
– Possible causes
– Effects on user experience
• Assign importance/severity
• Suggest solutions
31

32. Communicating Findings
Usability test report
• Project overview
• Participants
• Testing environment
• Tasks
• Procedure
• Key findings
– Issues
– Recommendations
32

33. Communicating Findings
Presentation
• Prepare the audience
• Use supporting evidence (e.g., highlight videos)
• Emphasize user perspectives
• Use numbers carefully (because of small sample)
• Leave enough time for questions
33

34. Communicating Findings
Common problems:
• “This is not statistically significant.”
• “This user is our typical user.”“This user does not
have relevant experience.”
• “User X did Y; therefore, everyone must do Y.”
• “More than half of users did not like this design;
therefore, we should change it.”
34

35. What’s Essential?
• Develop a study plan
• Find participants
• Conduct the sessions
• Debrief with observers and decide together
• Present results
35

36. Discussion
Think of an existing design that you want to evaluate
for effectiveness
• What are my objectives?
• What types of resources do I have?
• Who am I presenting this to?
36

37. Other Types of Usability Studies
• Remote usability testing http://www.userzoom.com/
• Eye tracking
37
http://www.normalmodes.com/blog/2009/09/28/eye-tracking-
heatmap-gallery-a-preview-discussion-of-ui-considerations/

38. Where to Find Out More
• 6 Must-See Usability Testing Videos
http://blog.usabilla.com/7-must-see-usability-testing-videos/
• Moderating Usability Tests
Joe Dumas & Beth Loring (2008)
• A Field Guide to Usability Testing
Smashing eBook
https://shop.smashingmagazine.com/guide-to-usability-testing-1.html
• Handbook of Usability Testing
Rubin & Chisnell (2008)
• Don’t Make me Think by Steve Krug
Chapter:“Usability Testing on 10 cents a Day”’
38

39. Learning Objectives
39
• Participants will be able to describe the
usability concept and it's applications to
libraries.
• Participants will be aware of the basic
workflows and techniques needed when
doing usability tests.
• Participants will be aware of resources for
usability studies and where to find out more.

40. Agenda
40
• What is a usability study?
• Usability study process
• Conducting the study
• Analyzing Results
• Communicating
• Where to find out more

41. Works Cited
• 6 Must-See Usability Testing Videos - The Usabilla Blog. (n.d.). Retrieved from http://
blog.usabilla.com/7-must-see-usability-testing-videos/
• Affairs, A. S. for P. (2013). Usability Testing. Retrieved from
http://www.usability.gov/how-to-and-tools/methods/usability-testing.html
• Clear, Consistent, Capable, NICE» Blog Archive » Eye Tracking Heatmap Gallery: A Preview &
Discussion of UI Considerations. (n.d.). Retrieved from http://www.normalmodes.com/blog/
2009/09/28/eye-tracking-heatmap-gallery-a-preview-discussion-of-ui-considerations/
• Dumas. Joe, & Loring, B. (2008). Moderating Usability Tests.
• DIY Mobile Usability Testing - SXSW Interactive 2012. (n.d.). Retrieved from
http://www.slideshare.net/beleniq/diy-mobile-usability-testing-sxsw-interactive-2012
• Morae usability testing software from TechSmith. (n.d.). Retrieved from
http://www.techsmith.com/morae.html
• Krug, S. (2005). Don’t Make Me Think.
• Rubin, J., & Chisnell, D. (2008). Handbook of Usability Testing.
• UserZoom | zooming in on the user experience. (n.d.). Retrieved from http://www.userzoom.com/
• Silverback — guerrilla usability testing. (n.d.). Retrieved from http://silverbackapp.com/
41

42. Final Notes
Evaluation of this webinar
Questions?
Tao Zhang zhan1022@purdue.edu
Ilana Barnes ibarnes@purdue.edu
42