A quick romp through our brief history to provide context for where we are today.

1. Usability Testing through
the Decades
Where we were, Where we are, Where we’re going
Presented by
Carol Barnum
9/23/15
Michigan Tech

2. 1970s

3. Designers’ prevailing belief systems
 Users are like designers
 Users don’t know what they need
 Designers don’t need to interact with users
 Reason will prevail—systems based on logical design will make sense to
users
 Design guidelines are sufficient
 Good design means getting it right the first time (goal of design is zero
defects)
 The power of technology will triumph—users will figure it out

4. First lab built – was it IBM?

5. 1980s

6. Lab testing was expensive, time consuming,
scientific … and mostly summative
It’s RESEARCH
Required lots of test subjects
Produced statistical validity
Produced formal reports
Cost big $$$$$$$$

7. Gould and Lewis show the way to user-centered
design
1985—Designing for Usability: key principles and what designers think
1. Early and continual focus on users and tasks
2. Empirical measurement from user involvement
3. Iterative design
THE FOUNDING PRINCIPLES

8. John Brooke develops System Usability Scale
1986 (published and made available to all in 1996)
 “Quick and dirty”
 10 Likert-scale questions
 Results easily calculated to a score 1 – 100
MOST POPULAR EVALUATION
QUESTIONNAIRE TO THIS DAY

9. Late 80’s - Change is in the wind

10. 1990s

11. Jakob Nielsen and others break with the pack
 1989—Nielsen CHI paper “Usability Engineering at a Discount”
 1994—Nielsen “Guerilla HCI—Using Discount Usability Engineering to
Penetrate the Intimidation Factor” in Cost-Justifying Usability
 Others doing similar research
 1990, 1992—Robert Virzi
 1994—James Lewis
5 USERS IS ENOUGH

12. Maximum cost benefit ratio—3 - 5 users give you
85% of the findings
Nielsen and Landauer

13. The most striking truth of the curve is that zero
gives you zero insights.
As soon as you collect data from a single test
user, your insights shoot up and you have already
learned almost a third of all there is to know
about the usability of design. THE DIFFERENCE
BETWEEN ZERO AND EVEN A LITTLE BIT OF DATA
IS ASTOUNDING.

14. Why bother testing at all?
(according to Nielsen in Usability Engineering)
 Your best guess isn’t good enough
 The user is not always right (don’t ask them what they want)
 The user is always right (experience beats opinion every time)
 Users are not designers
 Designers are not users
 Vice presidents are not users
 Less is more
 Details matter
 Help doesn’t

15. ISO 9241-11
(1998)
This part deals with the extent to which a product
can be used by specified users to achieve
specified goals with effectiveness (task
completion), efficiency (task time) and
satisfaction in a specified context of use

16. Personas – Alan Cooper
1999 The Inmates are Running the Asylum
“Personas are the
single most powerful
design tool that we
use.”

17. 21st
Century

18. CUE studies 1998 – 2011
http://www.dialogdesign.dk/CUE.html
 CUE 1—4 teams reviewed calendar software (1998)
 CUE 2—9 teams reviewed Hotmail (1999)
 CUE 3—11 teams reviewed Avis Rent a Car (2001)
 CUE 4—17 teams reviewed Hotel Pen (2003)
 CUE 5—13 teams reviewed Ikea wardrobe instructions (2005)
 CUE 6—13 teams reviewed Enterprise car rental (2006)
 CUE 7—9 teams produced recommendations from prior studies (2007)
 CUE 8—15 teams reviewed Budget.com car rental (2008)
 CUE 9—35 teams (Europe and US) reviewed Uhaul.com (2011)

19. UPA member surveys
2005 – 2011
(2014 in progress)

20. 2011 survey results - techniques

21. 58% do formal lab testing

22. 68% do informal testing

23. 50% do remote, moderated usability testing

24. Research Exploration
Testing
prototypes
Understanding
users
Answering
arguments
Reasons for testing

25. Reasons for NOT testing
Time Cost
Ignorance
Can’t get
users
Agile!!!!

26. What can you do in a day?

27. Welcome – 5 min
Task-based scenarios–
40 min
Assessment – 10 min
Thanks! – 2 min
Basics of 1-hour testing session

28.  Usability Testing of Fruit
Let’s watch a session

29. Faster, cheaper methods
offer many options

30.  Rapid Iterative Testing and Evaluation
 Developed by Microsoft’s Game Studios
 Requires full team commitment
 Observe
 Analyze findings immediately
 Change immediately
 Retest
 Repeat cycle
RITE

31.  http://fivesecondtest.com/
 5-second usability tests—how much time is enough? Jeff Sauro 2010
 Compared 5 seconds, 60 seconds, no time limit
 Measured results with SUS: user response is the same for 5 secs & no limit
 The UX Five-Second Rules—Paul Doncaster 2014
5-second test

32. First click
Bob Bailey reported on research on CDC and other govt. websites
 Users who got the first click right were 87% successful
 Users who got the first click wrong were only 47% successful
 Fast way to get lots of feedback!
 In one study each participant completed 136 scenarios in 1 hour

33.  http://www.usertesting.com US and other countries—small
 http://www.whatusersdo.com/ UK/Europe— small
 http://www.userzoom.com/ US, UK, and other countries—big
 http://www.loop11.com/ US and other countries—big and small,
subscription only
 http://www.trymyui.com/ mostly US—small
23% do web-based unmoderated testing

34. Optimal Workshop
 Treejack—information architecture
 OptimalSort—card sorting
 Chalkmark—first-click testing
Here are some other unmoderated tools

35. Here’s another technique—customer journey
mapping

36. Here’s another technique—diary studies

37. What’s ahead??
I don’t know,
but I’m
curious!
Are you??

38. Questions now?

39. Questions later?
carolbarnum@uxfirm.com
www.uxfirm.com
Read more