Usability Testing Bootcamp

How to Plan and Moderate
a Usability Test
Dr David Travis, Userfocus ltd

#uxlx #ut

I currently run a consultancy
firm called Userfocus. We work
with clients like Yahoo!, the
World Health Organisation and
the UK Tax Office. Yes, even they
are using these techniques to
make sure you pay what you owe.

Who am I?
?

You can still buy my
book on amazon!

Design and plan usability tests.
Recruit the right test participants.
Apply usability testing
checklists to make sure your test
runs without a hitch.
Develop test tasks that are
relevant to your customers and
your business.
Organise and prepare usability
testing sessions.
Moderate 'thinking aloud'
usability tests in an unbiased and
balanced manner.
Code, collect and interpret
behavioural data.

What you’ll learn

Where do you work and what
do you do?
What one thing do you
already know about usability
testing?
What do you want to be able
to do differently as a result of
the training?

Interview your partner

How to Plan and Moderate a
Usability Test
What so special about a usability test?

When people are asked
to justify their
choices, they do so
readily — even when
they never made the
choice. We need to be
aware of this
confabulation when
running any kind of
user research.

Johansson, P., Halls, L., Sikström, S & Olsson, A. (2005)
Failure to Detect Mismatches Between Intention and Outcome in a Simple Decision Task. Science, 310: 116-119.


Lichfield Dakota 3 Man Confidence Holiday 6 Lichfield Crow 2 Man Lichfield Cherokee 3
Tent Denim/SkyBlue Man 2 Room Tent Tent Denim/Navy Man Tent Red

 The first tent
presented was
Lichfield Crow 2 Man Lichfield Cherokee 3 Confidence Holiday 6 chosen 2.5 times
Lichfield Dakota 3 Man
more often than any
Tent Denim/Navy Man Tent Red Man 2 Room Tent Tent Denim/SkyBlue
other. Yet people
‘explained’ their
choices using logic


and rationality.

Lichfield Cherokee 3 Lichfield Crow 2 Man Lichfield Dakota 3 Man Confidence Holiday 6
Man Tent Red Tent Denim/Navy Tent Denim/SkyBlue Man 2 Room Tent

Felfernig, A. et al. (2007) Persuasive Recommendation: Serial Position Effects in Knowledge-Based Recommender Systems.
Lecture Notes in Computer Science, 4744: 283-294.

“The accuracy of subjective
reports is so poor as to
suggest that any introspective
access that may exist is not
sufﬁcient to produce generally
correct or reliable reports.”

- Nisbett, R.E., and Wilson, T.D. “Telling more than we can know: Verbal reports on mental processes.”
Psychological Review, 84 (1977), pp. 231-259.

Behaviour is observed —
what users do (rather than
what users say they do).
Participants perform real
tasks — tasks that are
meaningful to them and to the
business.
Involves real users — not just
someone at the next desk.
Observations and data are
recorded and the data are
analysed.

What is a usability test?

If we’re going to test for usability, we better define it first.
Listen to the definitions of usability and call out the missing words.

Quick Activity: Usability Blankety Blank

“Usability means that the people who use the
product can do so quickly and easily to
accomplish their own tasks. This deﬁnition
rests on four points: (1) Usability means
focusing on USERS; (2) people use products
to be productive; (3) users are busy people
trying to accomplish tasks; and (4) users
decide when a product is easy to use.”

- Janice (Ginny) Redish and Joseph Dumas (1999). A Practical Guide to Usability Testing.

“Usability is a term used to
denote the ease with which
people can employ a particular
tool or other human-made
object in order to achieve a
particular GOAL.”

- Wikipedia.

“The capability in human functional
terms to be used easily and
EFFECTIVELY by the specified range of
users, given specified training and user
support, to fulfil the specified range of
tasks, within the specified range of
environmental scenarios.”

- Brian Shackel (1991). ‘Usability—context, framework, definition, design and evaluation’. In B. Shackel and S. J.
Richardson (eds) Human factors for informatics usability.

“At ﬁrst, using words that all started
with “E” was a word game, but I was
also looking for a way to make the
dimensions of usability easy to
remember and the 5Es were born. I
decided on: Effective; EFFICIENT;
Engaging; Error Tolerant; Easy to Learn.”

- Whitney Quesenbery (2004) ‘Balancing the 5Es: Usability’. Cutter IT Journal, Feb, 2004.

“It is important to realize that usability
is not a single, one-dimensional property
of a user interface. Usability has multiple
components and is traditionally
associated with these ﬁve usability
attributes: learnability, efﬁciency,
memorability, errors, SATISFACTION.”

- Jakob Nielsen (1993), Usability Engineering.

“Usability is a function of the ease
of use (including learnability when
relevant) and the acceptability of
the product and will determine the
actual usage by a particular user
for a particular task in a particular
CONTEXT.”

- Nigel Bevan, Jurek Kirakowski and Jonathan Maissel (1991) ‘What is Usability?’ Proceedings of the 4th
International Conference on HCI, Stuttgart, September 1991.

So based on these
quotations, it seems that users
usability has something to
do with: goals
• USERS effectiveness efﬁciency
• GOALS
satisfaction
• EFFECTIVENESS
•
•
EFFICIENCY
SATISFACTION
context
and

• CONTEXT
tHere’s a definition we can
use that brings all of these
together…

“Extent to which a product,
system or service can be
used by specified users to
achieve specified goals with
effectiveness, efficiency and
satisfaction in a specified
context of use.”
- Effectiveness
The accuracy and completeness with which users
achieve specified goals.

- Efficiency
The accuracy and completeness of goals achieved in
relation to resources.

- Satisfaction
Freedom from discomfort, and positive attitudes
towards the use of the product, system or service.

ISO 9241 definition of usability (from ISO 9241-11 and ISO 9241-210)

S T
U
“Extent to which a product,
Jbe E
O UT S
system or service can
used by specified users to
N O
achieve specified goals with
IS T
effectiveness, efficiency and
Y SY
IT A
satisfaction in a specified

IL E
context of use.”
B- S Effectiveness

SA ING
U H -
The accuracy and completeness with which users
achieve specified goals.

T T
A G
Efficiency

H N
T I
The accuracy and completeness of goals achieved in
relation to resources.

S K
N A
- Satisfaction

EA T M
Freedom from discomfort, and positive attitudes
towards the use of the product, system or service.

M U
IS O
H B
T A
ISO 9241 definition of usability (from ISO 9241-11 and ISO 9241-210)

1
With usability, we ask if
2
people can actually use a
product.

You don’t need many people
to identify a problem that
3
4
needs to be fixed.

For example, which knob
would you use to light the
gas under the back right
hob?

Usability testing
then gives you
these kinds of
insights to improve
designs.

“Testing one user
early in the project is
better than testing
50 near the end.”
- Steve Krug

Usability tests don’t require:
- A usability lab
- Observers
- A moderator
- Screen recording
- Video recording

All these are ‘nice to have’.
Focusing on these alone puts
you at risk of running a ‘cargo
cult’ usability test.

So what deﬁnes a usability test?

Usability Test
A live test

Try to deconstruct the
usability test.
How would you describe what
happened when?
Are there different phases?

Deconstruct a usability test

Q. Would you like to take part
in a usability evaluation?
As a thank-you, you’ll get a
free copy of E-Commerce
Usability!

YES: Remain standing
NO: Sit down

Q. In the last 6 months, have
you bought any electronic
goods online, such as a
computer, a camera or a
mobile phone?

Q. Do you use a colour printer
at home?

Q. Is your printer a laser
printer or an ink-jet printer?

LASER: Sit down
INK JET: Remain standing
DON’T KNOW: Remain
standing

Q. Have you done any of
these activities in the last 2
weeks? Give yourself 1 point
for each ‘YES’ answer.
- Edited the html code on a web page
- Sold something on eBay
- Created or commented on a blog
- Installed software on your computer
- Signed into your online bank account
- Played an online game
- Downloaded and listened to podcasts
- Shared photographs or videos on web sites
like ﬂickr, Twitpic or YouTube
- Added a page to a social bookmarking site
like del.icio.us, reddit or Digg
- Subscribed to a syndicated ("RSS") web feed

7 or more: Sit down
6 or fewer: Remain standing

Oh no! Your home printer is
broken.
You can spend up to £100 on
a new one.
The printer must be wi-ﬁ
enabled and it must work with
a Mac.
Find the best printer within
your budget.

Deconstruct the usability test

Participant
Recruitment ? ? ?

? ? ?

Identify the test goals
Screen and recruit
participants
Identify the test tasks
Greet the participant
Get the video consent form /
NDA signed
Demonstrate and practice
thinking aloud
Probe with open questions

Usability Test
Planning a usability test

Why are you running the test?
Where will it take place?
When will it take place?
Who will be the test
participants?
What system (and what
functionality) will you be testing?
How will you collect and analyse
the data?

The Five W’s (and one H)

!"#$%&'()*+,-*.&/00'01*/*2(/3'&')4*)#()
5*6##%(*3#+,-#*)"#*)#()* :"#*;,-0'01*,+*)"#*)#()*
! "#$%%!&%'&!()*%+&,-%'! ! H2&!KD.'&$2+&,(.'!0($!()'%$-%$'LA!&/%!4,'&!(0!'+%.1$,('!1.5!
! "#$%%!(.!&/%!0%1&2$%!'%&!&/1&!3,44!)%!&%'&%5! 1!+(6<!(0!&/%!&%'&!641.!,.!&/%!()'%$-1&,(.!$((9!
! "#$%%!61$&,+,61.&!6$(0,4%'! ! 7/%+;!&/%!6$(&(&<6%!3($;'!
! 7$%1&%!61$&,+,61.&!'+$%%.%$! ! 7/%+;!&/%!$%+($5,.#!%=2,69%.&!R!'+$%%.!$%+($5,.#!
! 8.#1#%!$%+$2,&9%.&!0,$9! '(0&31$%!3($;'!?125,(!1.5!-,5%(B!
! :((;!(2&!1.<!%=2,69%.&!<(2>44!.%%5!?%@#@!$((9A! ! 7/%+;!&/%!-,5%(!1.5!125,(!0%%5'!,.!&/%!()'%$-1&,(.!
+(962&%$A!-,5%(!+19%$1B! $((9!
! !

7*6##%(*3#+,-#*)"#*)#()* <#+,-#*#/$"*./-)'$'./0)*/--'=#(*
! C$,&%!&1';!'+%.1$,('! ! M%S'%&!&/%!3%)!',&%!&(!,&'!5%0124&!'&1&%T!+4%1$!&/%!)$(3'%$!
! C$,&%!&%'&!641.!1.5!'%.5!&(!'&1;%/(45%$'!0($!$%-,%3! /,'&($<A!5%4%&%!+((;,%'A!4(15!1!)41.;!)$(3'%$!61#%!
The test plan toolkit contains guidance and checklists for running a
! D.-,&%!()'%$-%$'! ! O2$.!(00!&/%!'+$%%.!?)2&!;%%6!&/%!+(962&%$!$2..,.#B!
! ! F&1$&!&/%!$%+($5,.#!%=2,69%.&!
usability test.
8*6##%*3#+,-#*)"#*)#()* ! M1.5(9,'%!&/%!'+%.1$,(!+1$5'!
!
You! 7/%+;!$%+$2,&9%.&!,'!(.!'+/%524%! in Microsoft
get an editable version Word >"#0*)"#*./-)'$'./0)*/--'=#(*
when you sign up for refresher training
! E,.14,'%!1.5!,''2%!&%'&!641.!!
(free!)
! F%.5!(2&!166(,.&9%.&!&,9%A!916!1.5!-,5%(!+(.'%.&! ! U1.1#%!%I6%+&1&,(.'!G!%I641,.!&/%!'%&&,.#!
0($9!&(!61$&,+,61.&! ! "';!&/%!61$&,+,61.&!&(!+(964%&%!1!-,5%(!+(.'%.&!0($9!
! 7/%+;!&/%!6$(&(&<6%!'266($&'!&/%!'+%.1$,('!G!9(5,0<! !
'+%.1$,('!,0!.%+%''1$<! >"#0*)"#*./-)'$'./0)*'(*+'0'("#9*
! D0!1.<&/,.#!<(2>$%!2',.#!.%%5'!)1&&%$,%'A!#%&!'61$%'!
! ! "';!&/%!61$&,+,61.&!&(!+(964%&%!1!6('&S&%'&!
=2%'&,(..1,$%!
8*9/4*3#+,-#*)"#*)#()* ! 7/%+;!,0!()'%$-%$'!/1-%!1.<!0(44(3S26!=2%'&,(.'!
! H$,.&!'+%.1$,('!(.!,.5%I!+1$5'!0($!%1',%$!/1.54,.#! ! H1<!&/%!,.+%.&,-%!1.5!#%&!1!',#.%5!$%+%,6&!
! H$,.&!&%'&!'+$,6&! ! F&(6!&/%!$%+($5,.#!G!.19%!&/%!$%+($5,.#!166$(6$,1&%4<!
Usability !Test Plan Toolkit
H$,.&!61$&,+,61.&!=2%'&,(..1,$%'!?J!6%$!61$&,+,61.&B! ! H$%61$%!0($!&/%!.%I&!61$&,+,61.&!
! H$,.&!-,5%(!+(.'%.&!0($9'!?61$&,+,61.&'!91<!0($#%&!&(! !
)$,.#!&/%!(.%!<(2!'%.&B! ?)*)"#*#09*,+*#/$"*9/4@(*)#()'01*
! H$,.&!KD.'&$2+&,(.'!0($!()'%$-%$'L!

Purpose of usability test
Product under test
Participants & responsibilities
Evaluation procedure
Data collection
Appendices
- Screener, consent form, checklists, etc.

Contents of the Test Plan

Step 1: Use personas
- If you don’t have any, create assumption
personas based on low / high technological
expertise and low / high domain knowledge

Step 2: Go where your users
congregate
- Friends & family

- Customer lists

- Professional societies

- LinkedIn, Twitter

- Recruitment agencies

Step 3: Filter with an online
screener
Step 4: Follow up by phone

Recruiting participants

Portable test lab
The moderator takes the test to
the participant and sits at the
participant’s desk or cubicle. If
observers are present, they
squeeze in and sit where they
can.

A taxonomy of tests

Single room setup
The moderator and participant sit
at a desk in a small ofﬁce.
Observers sit at the back of the
room and either crane their
necks or watch the test on a
separate computer running
screen sharing software.
Observers are close enough to
hear what the moderator and
participant say to each other.

A taxonomy of tests

Classic testing lab setup
in one room. Observers can see
the moderator and participant
through a one-way mirror and
hear their conversation through
the lab’s built-in audio system.
Observers get a close-up view of
the participant’s screen via

A taxonomy of tests

Classic benchmark test
The participant sits in one room
and works alone. The moderator
and observers can see the
participant through a one-way
mirror and get a close-up view of
the participant’s screen via

A taxonomy of tests

Multi-room setup
in one room. Observers sit in
another room elsewhere in the
building. Observers watch the
participant’s screen via screen
sharing software projected on the
wall and hear the participant
through the computer’s
speakers.

A taxonomy of tests

Remote moderated test
The participant carries out the
test from his or her home or
ofﬁce desk. The moderator uses
screen sharing software to see
the participant’s screen and
presents the tasks to the
participant over the phone.
Observers use conferencing
software to view the test over the
Internet.

A taxonomy of tests

Remote unmoderated test
The participant carries out the
test from his or her home or
ofﬁce desk. The computer
presents the tasks to the
participant and the participant
signals when he or she has
completed the task. Nobody
observes or moderates the test.

A taxonomy of tests

Usability Test
Writing Test Tasks

Roads in London with red lines on them
are known as red routes: these are the key
road arteries in London. Transport for
London does everything it can to keep
these routes clear.

Your product or web site has “red routes”
too. They are the critical “user
journeys” with your product.

Red routes

What are the Red Routes for a
university web site?
- Can I study [subject]?
- What does it cost?
- What qualiﬁcation will I earn?
- How will this help me in my job or
career?
- Where do I go to take this course?
- Can I study part time, while working?

Red Route Examples (from Gerry McGovern (2010), The Stranger’s Long Neck).

government web site that
aims to help small business?
- What grants are available to my
company?
- How do I apply for funding?
- What funding is available to a start-up?
- How do I apply for funding?
- How can I reduce my tax liability?


health web site?
- Based on my symptoms, what disease/
condition do I have?
- What are the basic fact about
[condition / disease]?
- How do I book an appointment online?
- How can I get advice from a health
professional?
- Based on my symptoms, what should I
do next?


You have 300 seconds.
Think about a product,
system or service that you are
working on at the moment.
Brainstorm a handful of red
routes — the ‘headline’ tasks.

Brainstorm red routes

People approach tasks
differently based on the
context of use.
- Imagine you need to book a flight and hotel
in Paris for a 2-day trip with your partner.

- Imagine you need to book a flight and hotel
in Paris for a 2-day business trip.

The red routes — finding a
hotel, booking a flight — are
the same but the context of
use alters the way you would
carry out these tasks.
You build context into your
test tasks with scenarios.

The context of use

Red route Scenario

“You’re fed up with the long journey to
“Search for jobs by geographic work. Find all the suitable job
area” openings within a reasonable
commute of your home.”

What are the implications of this
scenario for the design?

We can offer a better experience if we
show “commuting time” as an option in
the search results. And note that this
may not be the same as distance, it
depends on transport connections.

You wouldn’t get this from the
original red route which doesn’t have
the contextual information.
Expand your red routes into scenarios

Red route Scenario

“Darren wants to buy a Sony PlayStation
Portable and a suitable game for his 12
“Buy a PSP” year-old son for his birthday this Monday.
Because of the short notice, he won’t buy
the product unless he can collect it from a
store before his son’s birthday.”

What are the implications of this scenario?

• We need to offer cross-channel support (buy online,
pick up in store)
• We need to be able to show shop stock levels
• We need to show store locations and directions
relative to the customer’s location
• We need to offer a reservation system so the PSP can
be put aside
• We need to cross sell suitable games by age group, or
at least classify games according to age group and
Expand your red routes into scenarios
gender

Does it describe a
Is it speciﬁc and complete activity
Is it really a red route?
measurable? (integrated, not simple
tasks)?

Does it describe what Does it include
Is the task “portable”
the user wants to do enough information to
to competitor
(not how the user will complete the task yet
products?
do it)? avoid hidden clues?

Six-step scenario checklist

You have 300 seconds.
Take one of the red routes
you created earlier.
Use it to create a scenario —
build in the ‘context’
information.

Create a scenario

Usability Test
How to moderate a usability test

Flight Attendant
- Safeguard the physical, psychological and
emotional wellbeing of the participant.

• Greet participants, hang up
their coats, offer a drink,
engage in small-talk
• Monitor participants for
signs of stress
• Provide reassurance and
assistance as needed
• Obtain informed consent

Hats moderators wear in a usability test (based on Carolyn Snyder’s Paper Prototyping)

Before starting — let participants know
they are going to be listened to and
that their opinions are valued.
Observe and read the participant’s
non-verbal behaviour — movement,
facial expressions, etc.
Reinforce the participant by nodding or
paraphrasing
Take some notes, but don’t write all the
time
Reﬂect back what you think you heard
and check your understanding.
Ask open-ended questions
Pace your talking speed to match the
participant’s
Listen to inconsistencies that may
need to be challenged

How to be a good moderator

To what degree does my non-
verbal behaviour show that I
value the participant (e.g. am I
shufﬂing papers)?
What attitudes am I expressing
non-verbally (e.g. am I focusing
on the recording technology)?
What attitudes am I expressing
verbally (e.g. am I making the
person feel like a “subject” rather
than a “participant”)?
What is stopping me from giving
my full attention to this
participant (e.g. noise, tiredness,
boredom)?

To be an effective listener, listen to yourself

What am I doing to handle these
distractions?
How intently am I listening to
what the participant is saying?
What am I doing to demonstrate
my willingness to listen to the
participant?
How well am I reading the
participant’s non-verbal
behaviours?
How careful am I not to over-
interpret non-verbal behaviour?

To be an effective listener, listen to yourself

- Saying, “Remember, we’re not testing
you,” more than three times.
- “Don’t worry, the last participant couldn’t
do it either.”
- “No-one’s ever done that before.”
- “HA! HA! HA!”
- “That’s amazing! I didn’t even know it
could go in upside down!”
- “Could we stop for a moment? Watching
you struggle is making me tired.”
- “I didn’t really mean you should press
any button.”
- “Yes, it’s normal for observers to laugh
during a test.”
- “Don’t feel bad, many people take 15 or
16 tries.”
- “Are you sure you’ve used a computer
before?”

What NOT to say to participants (slightly adapted from Rubin & Chisnell’s Handbook of Usability Testing)

Sports Commentator
- Ensure that observers get as much useful
information from the test as possible.


Getting participants to think aloud
Phase 1 — Give instructions


Phase 2 — Demonstrate and let them practice


Demonstrate thinking aloud with an example:
Turn your phone to silent

Get the participant to practice with a different example:
Adjust the height of his or her chair

Make sure your practice task is easy: you want the user to practice the
technique and feel successful!

Demonstrate and let them practice

Practise welcoming a
participant
- Set expectations and reassure your
participant.

- Ask for your participant’s consent to record
the session.

- Demonstrate thinking aloud.

- Get your participant to practise thinking
aloud.

Project Activity: Quick activity

Phase 3 — Probe with open questions


Keep
talking…

Tell me
more
about that What are you
thinking
right now?

PHASE 3: probe with open questions


Tell me
more
about that What are you
thinking
right now?

PHASE 3: probe with open questions

it was
easy!
what was
easy?


it was
easy!
what was
easy?

is that how is that how
it's supposed you expected
to work? it to work?


don't ask users to extrapolate…

Mmm, do I think that
someone, somewhere
is this a will find some use
useful for this feature?
feature?

would this feature, as it's
currently presented, be
valuable to the way you choose
…instead,
products today?
focus on
the present

instead of generic questions…

products today?
focus on
the present

instead of generic questions…

Mmm, there are some
good things and some
bad things, so what do
Is this a
i think overall?
good idea?

was there
anything you
particularly
liked or disliked
about the
what specifically?
…focus on prototype?
the specifics

don't be judgemental

crikey, he thinks this
don't you think this option should appear
option would be on the home page and
better if it was he won't like it if i say
available on the otherwise.
home page?

is there any other
a better place you'd like to see
question a feature like this?
would be…

don'T encourage confabulation

a better place you'd like to see
question a feature like this?
would be…

don'T encourage confabulation

do i have my 'surprised'
face on again? i better
you seem
think of something
surprised,
surprising now.
were you?

was that what
you expected? what did you
but keep the expect?
participant
talking

Scientist
- Responsible for avoiding test bias and
recording the data.

• Representative users.
• Appropriate task
scenarios.
• Balanced design
• Neutral questioning.
• Where bias can’t be
avoided, make a note.
• Decide when to intervene
to help a participant
• Log the data
• Record audio and video of
the session.
• Collect accurate
measures of task
completion, time on task
and satisfaction.


! -#('$!$*!8*49!7(+5.(5#! 0"&-8)*(98&?*()&-8*(98-&')*5"++6F&
;$54&('&*3&'2)-$5$'23-&>*1?&/239(29"&*,&,)(+-)2-$*3&*)& ! D,2!3*/!,&#')3)'!#%(@&7#,!G!#+'*./(5#!,$*/9$#77)+5!
233*?235"&231&21@(+-&$3-")25-$*3&255*)1$39/?6& B21M&DP*&?*(&82#"&-)*(>/"&%$-8&+"2)58GF&O**1M&DC"//&0"&
! :*+;$!4#3#+4!$0#!4#,)5+! 2>*(-&2&-$0"&%8"3&?*(&5*(/13=-&,$31&%82-&?*(&%")"&
:5-&2+&2&3"(-)2/&'2)-?&*(-+$1"&*,&-8"&1"+$93&-"20&A&?*(&2)"& +"2)58$39&,*)6F&
-8")"&,*)&)"+"2)58.&3*-&-*&1","31&1"+$93+6& ! D>*)4!7*+5!(+4!*>#/79!'*@&7#%!E.#,$)*+,!
! <($'0!3*/!,#73='#+,*/)+5! B21M&DK8"3&-8$+&+$-"&%2+&1"+$93"1.&-8")"&%2+&1$+5(++$*3&
;)*>"&,()-8")&%8"3&'2)-$5$'23-+&5"3+*)&-8"$)&%*)1+&*)&>/20"& %8"-8")IF&O**1M&DN*%&5*(/1&-8$+&+$-"&>"+-&0""-&?*()&
-8"0+"/#"+6& 3""1+GF&

! :#>#7*&!3/)#+479!/(&&*/$! ! :*+;$!(,2!&(/$)')&(+$,!$*!/#=4#,)5+!$0#!)+$#/3('#!
B"&%"/5*0$39&231&8"/',(/&-*&-8"&'2)-$5$'23-&%$-8*(-&>"$39& B21M&DK82-&%*(/1&>"&-8"&>"+-&%2?&-*&1"+$93&-8$+GF&O**1M&DH3&
-**&"0'2-8"-$5&$,&'2)-$5$'23-&$+&3"92-$#"&-*%2)1&2&1"+$936& ?*()&"R'")$"35".&%82-&*-8")&%">&+$-"+&1*&-8$+&%"//GF&

! ?/*>)4#!+#.$/(7!#+'*./(5#@#+$! ! D,2!'./)*.,!E.#,$)*+,!$*!#%&7*/#H!8.$!(>*)4!
C)"2-&'*+$-$#"&231&3"92-$#"&5*00"3-+&-8"&+20"&%2?6&DC8$+&$+& '*+3/*+$($)*+!
#")?&8"/',(/6&E*(=)"&1*$39&@(+-&%82-&%"&3""1&?*(&-*&1*F6& B21M&DP*&?*(&82#"&23?-8$39&"/+"&-*&+2?&2>*(-IGF&O**1M&
DS23&?*(&-"//&0"&0*)"&2>*(-IGF&
! -#&#($!
!"$-")2-"&5*00"3-+&#")>2-$0&-*&"35*()29"&"/2>*)2-$*36& ! I#$!8#0)+4!$0#!8#0(>)*./J!
Use a checklist to remind you how
DE*(&+2$1&?*(&(+(2//?&+-2)-&-8$+&2''/$52-$*3&,$)+-GF&
to phrase unbiassed questions.
B21M&DN*%&%*(/1&?*(&)2-"&?*()&"R'")$"35"&%$-8&-8"&+$-"GF&
O**1M&DN*%&%*(/1&?*(&)2-"&?*()&"R'")$"35"&%$-8&-8"&+$-"G&
You’ll receive a copy of this with your second refresher training e-mail.
! 10#'2!9*./!)+$#/&/#$($)*+!
K8?GF&
!"'8)2+"&-8"&'2)-$5$'23-=+&+-2-"0"3-&$3&?*()&*%3&%*)1+&-*&
"3+()"&255()25?6&DH&20&8"2)$39&?*(&+2?IF& ! ".@@(/),#!(+4!4/(K!(!'*+'7.,)*+!
DT*&%82-&H&20&8"2)$39&?*(&+2?&$+&-82-&-8"&%2?&-8"&LB**4&2&
! A)+)@)B#!@*4#/($*/!4),$/('$)*+,!
,25$/$-?=&$+&$0'/"0"3-"1&8")"&$+&5*3,(+$39.&>(-&-8"&$1"2&$+&
B"&2%2)"&*,&23?&233*?$39&*)&)"'"-$-$#"&9"+-()"+&231&
+$0'/"6F&
")21$52-"&-8"06&
! C+'*./(5#!$0)+2=(7*.4!
6-077'12*37*
!"0$31&-8"&'2)-$5$'23-&-*&-8$34J2/*(16&DK82-=+&9*$39& ! D,2!3*/!(+!*>#/(77!(,,#,,@#+$!
-8)*(98&?*()&8"21&2-&-8"&0*0"3-GF& DS23&?*(&+(002)$U"&?*()&"R'")$"35"&%$-8&Q&-*12?GF&*)&
! :*+;$!3*/'#!*&)+)*+,! DK82-&%*(/1&?*(&+2?&%")"&-8"&-*'&-%*&+-)"39-8+&231&
:+4&2>*(-&'2)-$5$'23-&*'$3$*3+&%8")"&2'')*')$2-".&>(-&1*3=-& %"243"++"+6F&
,*)5"&23&*'$3$*3&$,&-8"&'2)-$5$'23-&1*"+3=-&,""/&+-)*39/?6&
Usability Test Plan Toolkit ! -#E.#,$!'*@@#+$,!/#5(/4)+5!$*&)',!+*$!/(),#4!89!
! 17(/)39!(!$#/@!*/!'*+'#&$! @*4#/($*/!
DK82-&1*"+&L+?+-"0=&0"23&8")"GF&DK82-&1*&?*(&52//&-8*+"& DK82-&5*00"3-+&1*&?*(&82#"&*3&-*'$5+&%"&1$13=-&5*#")&
-*12?GF&*)&DK82-&"/+"&+8*(/1&H&82#"&2+4"1GF&

For example:
- Closed question: Don’t you think you should
try the Back button?

- Open question: How else might you return to
the home page?

Are you looking for search?
Did you just click on the Submit
button?
Didn’t you see the link to the site
map?
Don’t you think you should look in the
pull-down menu?
Do the radio buttons make your
options clear?
Do you think the design of this page
is good?
Do you think people will ﬁnd this easy
to use?

Quick Exercise: Improve these closed questions

Ask the people observing the test to write a short description of each
usability problem they see on a sticky note.
One problem per sticky note.
Identify the participant associated with the observation (e.g. P5).

At the end of the day’s testing, work with the observers to group,
prioritise and label the usability issues.

Getting clients and the development team to log usability issues

As you watch the test, you should note down the participant’s
behaviours and code your observations.
Good note taking saves hours in analysis time.

Why data logging is important

- Data logging ensures you note
all behaviours, not just the ones
that stand out (this helps reduce
bias in your observations).

- It helps you concentrate and
focus on the session.

- When lots of observations come at
once, you will be able to just note
the observation code — you can
then review it later on the
videotape.


- When scanning your notes, the
observation codes make it easy to
distinguish one class of
observation (e.g. the usability
issues) from other observations.

- It’s one of those things you’ll be
glad you did when there are
problems with the video
recording (e.g. when the sound is
poor or when the recording is
corrupted).


S Start task F Facial reaction (e.g. surprise)
E End task A Assist from moderator
X Usability problem Q
Gives up or wrongly thinks

ﬁnished
*
Video highlight — an “Ah-ha!”

moment H
Help or documentation

accessed
G Comment from P (general)
M
Misc (general observation by
P Positive opinion

logger)
N Negative opinion
C Clickstream (optional)
B Bug

How to classify your observation

S Start task B Bug
E End task F Facial reaction (e.g. surprise)
G General comment A Assist from moderator

As a rule of thumb, you P
N
X
Positive opinion
Negative opinion
Usability problem
Q
H
M
Gives up or wrongly thinks finished
Help or documentation accessed
Misc (general observation by logger)

should average about one * Video highlight — an “Ah-ha!” moment C Clickstream

observation per minute. Time Code Description

But remember this is an
average
- Observations are a bit like buses (none for
ages, then three come along at once).

What you should write down:
- The time

- The class of observation

- A short description

How to log usability problems

USERFOCUS
180 Piccadilly, London, W1J 9HF

Dr David Travis
Managing Director

Work 020 7917 9535
Mobile 07747 016132
Email david.travis@userfocus.co.uk
Twitter @userfocus

Thank You!

Usability Testing Bootcamp

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Usability Testing Bootcamp

Similar to Usability Testing Bootcamp (20)

Recently uploaded

Recently uploaded (20)

Usability Testing Bootcamp