1. Keeping Your Moderating Skills Sharp
Susan Mercer
Senior Experience Researcher
smercer@madpow.com
@susanamercer

2. The Importance of Neutrality
• Clients want objective answers
• Asking biased questions lead to biased
answers
• Participants want to please us
• Providing types of feedback can subtly
communicate our expectations
• They are more likely to answer future questions
that way
2

3. The Reality of Moderation
• Observed 17 practitioners
• Many usability professionals get sloppy:
• Ask leading questions
• Participant:
“I would share this article with friends.”
• Moderator: “By printing it?”
• Ask closed-ended questions
• Moderator: “Would you print this?”
• Focus on known problems for “ammunition”
• Use biasing response phrases “Good job”, “Yes,
that’s a problem.”
3
Source: Norgaard & Hornbaek, 2006

4. What does this mean?
• Current practices are not consistent
• Moderating is taught through mentorship
• bad habits can easily be perpetuated
• We need to pay attention to our moderation
and remain neutral to capture unbiased data.
4

5. 5 Rules of Great Moderating
(A Quick Review)

6. 5 Rules of Great Moderating
1. Know your goals
2. Stay neutral
3. Be responsible
4. Be confident
5. Be genuine
6

7. 1. Know Your Goals
• What are you trying to achieve?
• Formative
• Summative
• What topics are highest priority?
These will influence
• What to focus on when time runs short
• How you fine-tune your moderating style
7

8. 5 Rules of Great Moderating
1. Know your goals
2. Stay neutral
3. Be responsible
4. Be confident
5. Be genuine
8

9. 2. Stay Neutral
• Be Quiet!
• Don’t insinuate they gave a wrong answer
✘ “Why did you do that?”
✘ “What made you think to click that link?”
• Don’t put ideas into their heads
✘ “Does entering your SSN make you feel uncomfortable?”
 “What do you think about entering your SSN?”
• Use their words
• If they call the dashboard a “chart area”, then call it a
“chart area”
• Keep your tone naturally curious, not derisive
• “What did you expect here?”
9

10. 2. Stay Neutral
• Don’t start with a verb
✘ id you see that?
D
✘ as that difficult?
W
✘ s this frustrating?
I
• Start with What, When, Where, Why, How
 hat would you change to make this easier?
W
 hen in this process would you be ready to buy?
W
 here did you expect to find that information?
W
 hy did you say this was easy?
W
 ow does that error message make you feel?
H
10

11. 2. Stay Neutral
• Use neutral, “continuing” response phrases
• Acknowledge that they gave feedback
• Don’t imply whether it is good or bad
✘ “Oh”, “Hmm”, “Interesting” - Assessing
✘ “OK”, “Yeah”, “That’s good” - Agreeing
 Uh huh”, “Mhmm”, “Tell me more” – Continuing
“
• Other tips to staying neutral?
11
Source: Boren & Ramey, 2000

12. 5 Rules of Great Moderating
1. Know your goals
2. Stay neutral
3. Be responsible
4. Be confident
5. Be genuine
12

13. 3. Be Responsible
You have responsibilities to:
• Participant
• Company
• Future Users
• Study Integrity
13

14. 3. Be Responsible
Situations happen during testing.
You may need to juggle your responsibilities.
What are your priorities?
• Study Integrity
• Company
• Future Users
• Participant
14

15. What to do?
If…
• Participant overshares personal information
• The participant rants about the website and
calls it “crap”
15

16. 5 Rules of Great Moderating
1. Know your goals
2. Stay neutral
3. Be responsible
4. Be confident
5. Be genuine
16

17. 4. Be Confident
• Project confidence
• If you make a mistake, keep going
• Don’t over-apologize
• Pause if you need to collect your thoughts
• “Fake it until you make it”
17

18. 5 Rules of Great Moderating
1. Know your goals
2. Stay neutral
3. Be responsible
4. Be confident
5. Be genuine
18

19. 5. Be Genuine
• Don’t play the notes on
the page…
…play the music
• Don’t just read the
questions from the page…
…have a conversation
19

20. 5 Rules of Great Moderating
1. Know your goals
2. Stay neutral
3. Be responsible
4. Be confident
5. Be genuine
6. What else?
20

21. Keeping sharp

22. Keep Improving
1. Identify your improvement areas
• Watch your videos
• Have colleagues give you feedback
22

23. Keep Improving
2. Learn from others
• Watch others moderate
• Be a participant
• Listen to talk radio interviews
23

24. Keep Improving
3. Practice skills in everyday life
• People watch (observe) in public
• Be quiet and listen in everyday conversations
24

25. Your Thoughts?
25

26. Thank You!
(Full presentation is on Slideshare. Follow me on twitter for link)
Susan Mercer
Senior Experience Researcher
smercer@madpow.com
@susanamercer
26

27. References / Photo Credits
Boren, T. and Ramey, J. (2000) Thinking aloud: reconciling theory and practice. IEEE
Transactions on Professional Communication, 43 (3), 261-278.
Dumas, J. and Loring, B. (2008) Moderating usability tests, Morgan Kaufman.
Dumas, J. and Redish, J. (1999) A practical guide to usability testing, Intellect Ltd.
Ericsson, K. and Simon, H. (1980) Verbal reports as data. Psychological review. 87 (3),
215-251.
Krahmer, E. and Ummelen, N. (2004) Thinking about thinking aloud: A comparison of two
verbal protocols for usability testing. IEEE Transactions on Professional Communication,
47 (2), 105-117.
Norgaard, M. and Hornbaek, K. (2006) What do usability evaluators do in practice? An
exploratory study of think-aloud testing. DIS 2006, 209-218.
Slide 4: Samantha Louras Photography
Slide 5: flickr: torgeaux
Slide 14: flickr: mkorcusa, flickr: Calsidyrose, flickr: marioanima, flickr: breity, flickr: Qole Pejorian
Slides 27, 29 and 30: flickr: Raphael Quinet, flickr: acme, flickr: betsyweber, flickr: KirkOls
Slide 32: flickr: bberburb
Slide 34: flickr: Tulane Public Relations
Slide 35: flickr: apdk, flickr: marktristan
Slide 38: flickr: Raphael Quinet
Slide 39: flickr: l-i-n-k
Slide 40: flickr: loppear
27