Usability Testing: Targeting Mobile Devices

Introduction to User Testing
Wayne Pau, Emerging Technologies SAP
Mar28, 2013

Internal

Introduction to User Testing
Implementing Quick & Effective User Testing
Wayne Pau, Emerging Technologies SAP
Feb 4, 2013 – MEIC’s Mobile Accelerator Programme – Workshop #6

My Apology Slide

“If I'm curt with you it's because time is a
factor. I think fast, I talk fast and I need you
guys to act fast if you wanna get out of this.
So, pretty please... with sugar on top…”
- Winston Wolfe (Pulp Fiction)

© 2012 SAP AG. All rights reserved.

Internal

3

Background Info
Giving credit to Steve Krug

Buy this book…

•

Steve Krug

•

2nd Book

•

Cheap & Dirty Usability Testing

•

161 pages


Internal

5

… then buy this next… (and read it first)

•

Steve Krug

•

1st Book

•

Web-Focused

•

216 pages


Internal

6

Even more info…
•

Website:

Advanced Common Sense
http://www.sensible.com/


Internal

7

Steve Krug – Recorded Sample

http://www.rocketsurgerymadeeasy.com/
http://www.youtube.com/watch?feature=player_embedded&v=Q
ckIzHC99Xc
All of Chapter 2 in Rocket Surgery Made Easy


Internal

8

The “Basics”
Some general info about Usability Testing

What User Testing is Not…

•

Not statistically relevant Quantitative Research

•

Not a Market Focus Group

•

Not for “Baking-Off” UI Choices

•

Not mining for new UI Designs/Patterns

•

Not Contextual Design or “Voice of God” interviews


Internal

10

Why do User/Usability Testing

1.

No Product is perfect. Everything can be Improved.

2.

Serious problems are easy to Spot. Encountered by
most Users.

3.

User-Centered approach makes for better products.


Internal

11

How to do User Testing
•

One Morning per Month

•

Find 3x “Participants for 1 Hr
session

•

Find 3x Tasks, expand into
Scenarios

•

Record/Broadcast

•

Debrief in Afternoon


Internal

12

# of Subjects

•

3 or 4 for 2+ rounds

•

3 or 4 allows testing

•

AM, debrief in PM

•

2x Rounds w/ 3 > 1x round with 6

•

Recruit loosely, grade on a curve


Internal

14

Recruitment - Exceptions

•

If you site is only used a particular population/relevence

•

If you subject has conflicting or divergent interests

•

If you app is esoteric and needs domain expertise


Internal

15

Recruitment – Good Ides

•

Offer good Incentive (Gift Cert)

•

Keep invitation simple

•

Don’t offer too much info

•

Ask Relatives/Friends/Family

•

Everyone was Newbie once


Internal

16

Testing Script
A template for Usability Session

Script Breakdown

1.

Welcome

2.

Questions (Warm-up)

3.

Acclimatizing (Familiarizing Participant)

4.

Tasks (Prepared Scenario)

5.

Probing (Digging into issues)

6.

Wrapping Up (Close & Thanks)


Internal

18

Welcome – Part 1
Hi,
session.

. My name is Wayne Pau, and I'm going to be walking you through this

You probably already know, but let me explain why we've asked you to come here
today: We're testing a web site that we're working on to see what it's like for actual
people to use it.
I want to make it clear right away that we're testing the site, not you. You can't do
anything wrong here. In fact, this is probably the one place today where you don't
have to worry about making mistakes.


Internal

20

Welcome – Part 2
We want to hear exactly what you think, so please don't worry that you're going to
hurt our feelings. We want to improve it, so we need to know honestly what you think.
As we go along, I'm going to ask you to think out loud, to tell me what's going through
your mind. This will help us.
If you have questions, just ask. I may not be able to answer them right away, since
we're interested in how people do when they don't have someone sitting next to
them, but I will try to answer any questions you still have when we're done.
We have a lot to do, and I'm going to try to keep us moving, but we'll try to make sure
that it's fun, too.


Internal

21

Welcome – Part 3
You may have noticed the camera. With your permission, we're going to videotape
the computer screen and what you have to say. The video will be used only to help us
figure out how to improve the site, and it won't be seen by anyone except the people
working on the project. It also helps me, because I don't have to take as many notes.
There are also some people watching the video in another room.
If you would, I'm going to ask you to sign something for us. It simply says that we
have your permission to tape you, but that it will only be seen by the people working
on the project. It also says that you won't talk to anybody about what we're showing
you today, since it hasn't been made public yet.
Do you have any questions before we begin?


Internal

22

Welcome - Tips
•

Stick to the script, stick to the script, stick to the script…

•

Try to remain neutral & calm

•

Test Participant (Not Subject. They are not being tested)


Internal

23

Questions – Part 1
Before we look at the site, I'd like to ask you just a few quick questions. First, what's
your occupation?
Good. Now, roughly how many hours a week would you say you spend using the
Internet, including email?
How do you spend that time? In a typical day, for instance, tell me what you do, at
work and at home.

Do you have any favorite Web sites?


Internal

25

Questions – Part 2
Now, finally, have you bought anything on the Internet? How do you feel about buying
things on the Internet?
And what have you bought?
OK, great. We're done with the questions, and we can start looking at things.


Internal

26

Questions - Tips
•

Build some rapport with your Participant

•

Assist in your “Grade on a Curve”

•

Get Participant Talking…


Internal

27

Acclimatizing
OK, great. We’re done with the questions, and can start looking at things.

First I’m going to ask you to look at this page and tell me what you make
of it: what strikes you about it, whose site you think it is, what you can do
here, and what you can do here, and what it’s for. Just look around and
do a little narrative.
You can scroll if you want to, but don’t click on anything yet.


Internal

29

Acclimatizing - Tips
•

Get them used to listening to instructions

•

Get them used to verbalizing

•

Get them use to “thinking out loud”


Internal

30

Tasks
Now I’m going to ask you to try doing some specific tasks. I’m going to
read each one out loud and give you a printed copy.
I’m going to ask you to do these task without using Search. We’ll learn a
lot more about how well the site works that way.
And again, as much as possible, it will help us if you can try to think out
loud as you go along.


Internal

32

Tasks – FoodNetwork.ca


Internal

33

Tasks #1 – FoodNetwork.ca

Assume you are big fan of Anna Olsen and love all her shows. You are
going to be home tomorrow all day and would like to determine when she
is on first on FoodNetwork tomorrow. If possible, find out:
•

What TV series it is (she has multiple)

•

What is the Episode About

•

What recipes will be shown (in case you want follow along)


Internal

34


Assume you are big fan of Anna Olsen and love all her shows. You are going to be
home tomorrow all day and would like to determine when she is on first on

FoodNetwork tomorrow. If possible, find out:
•

What TV series it is (she has multiple)

•

What is the Episode About

•

What recipes will be shown (in case you want follow along)


Internal

35


You’ve recently come back from a short vacation to New Orleans. You had a great
time and especially loved the Cajun food. Check FoodNetwork.ca for possible main

course dinner dishes that you can try at home. Assume you will looking at following
criteria to help you narrow it down:
•

How it was rated

•

You like Fish over Red Meat

•

You really Blacken Anything


Internal

36


You’ve had a rough week and are looking at cheering yourself up with
some food. However you still want to eat healthy and really don’t like
nuts (actually you could be allergic). Can you find a good, quick recipe
that would entice you?


Internal

37

Tasks - Tips
•

When in doubt *SAY NOTHING*

•

Use every permutation of “uh huh” or “OK” possible

•

Aim for “somewhat” pleased demeanor

•

Deflect ( “What would you do if I was not here?”…)

•

Mine leading non-verbal clues (“Is that what you expected?”)


Internal

38

Probing
Thanks, that was very helpful.

If you’ll excuse me for a minute, I’m just about to see if the people on the
team have any follow-up questions they’d like me to ask you.


Internal

40

Tasks - Tips
•

Relax “hands-off” approach

•

Ask “Why”?

•

Avoid asking for “solutions” or “redesigning on the spot”


Internal

41

Wrap-up & Thank-you
Do you have any questions for me, now that we’re done?

Thanks. That was exactly what we need, it’s been very helpful.


Internal

43

Real Testing World

•

Review results right away

•

Triage the information

•

Find a Fix (or approach)

•

Do the *LEAST* you can do to
fix the problem


Internal

45

7 Reasons to Tweak (vs. Rebuild)
1.

Cheap & Cost effective

2.

Less work & resource intensive

3.

Less Stressful

4.

Implemented Quicker

5.

Small & more likely to be scheduled

6.

Less Risky than Overhaul

7.

Might know the issue, but not the “perfect” solution


Internal

46

Wayne’s Issue Triage

1.

Is it MVP? Can we “Dump” the culprit?

2.

Is there a Small Tweak we can do?

3.

Was it effective enough? (Do we need to Tweak the Tweak?)

4.

Did we Breaking anything more Important?

5.

Is it a new feature? DON’T ADD ANYTHING!!!


Internal

47

Something is Better than Nothing…
“Perfect is the Enemy of Good”
– Voltaire

“A good plan executed today is better than a perfect plan executed tomorrow.”
-

General George Patton

POGE – Principle of Good Enough


Internal

48

Thank you
Contact information:
Wayne Pau (wayne.pau@sap.com)
Emerging Technologies

Usability Testing: Targeting Mobile Devices

Recommended

Recommended

More Related Content

Similar to Usability Testing: Targeting Mobile Devices

Similar to Usability Testing: Targeting Mobile Devices (20)

More from Wayne Pau

More from Wayne Pau (6)

Recently uploaded

Recently uploaded (20)

Usability Testing: Targeting Mobile Devices

Editor's Notes