Giulio Michelon, Founder di @Belka – “Oltre le Stime: Sviluppare una Mentalit...
Maximizing your moderating_game
1. Maximizing Your Moderating Game!
Susan Mercer!
Senior Experience Researcher!
smercer@madpow.com!
@susanamercer!
2. What is Usability Testing?!
• It is NOT:!
• Market Research!
• The same as a Focus Group!
• About user preferences!
• “Fluffy”!
• It is:!
• A systematic way to observe user behavior!
• A method to discover usability issues!
• A ongoing field of scientific inquiry!
!!
2
3. Topics!
• What is think-aloud?!
• Your roles as a moderator!
• 5 rules of great moderating!
• Keeping sharp!
!
3!
5. Think-aloud Protocol!
• Most commonly-used usability testing method!
• 1 participant / 1 moderator!
• Participant does tasks and thinks aloud!
• Origins in Cognitive Psychology!
Original protocol:!
• To understand short-term memory!
• Moderator invisible observer!
• Only comments: “Keep talking”, etc.!
• Don’t interrupt short-term memory!
5! Source: Ericsson & Simon, 1980!
6. Think-aloud Protocol - Problems!
• Many practitioners feel the strict no-
comments moderation is not ideal:!
• Makes participants feel uncomfortable!
• Feels “unnatural” not to respond!
• Participants may get stuck and not be able to
continue!
6!
7. Think-aloud Protocols!
Original Think-aloud! Usability Testing!
Study this…!
Study this…!
…to understand this !
Short term
…to understand this !
memory!
System
usability
7!
8. Modified Think-aloud Protocol!
• Based on Conversational Theory!
• Set up a conversation!
• Participant is main speaker!
• Use short, neutral response phrases (continuers) !
• “Mhmm”, “Uh-huh”, “And now…?” !
• Utter phrases like questions !
• Ask for clarifications by repeating a phrase!
• Participant: “That was odd…”!
• Moderator: “Odd?”!
• Provide hints if necessary !
• Neutrality is key!
8! Source: Boren & Ramey, 2000!
9. The Reality of Think-aloud!
• Many usability professionals get sloppy:!
• Ask leading questions!
• Participant: “I would share this article with friends.”!
• Moderator: “By printing it?”!
• Ask closed-ended questions!
• Moderator: “Would you print this?”!
• Focus on known problems for “ammunition”!
• Use biasing response phrases “Good job”, “Yes,
that’s a problem.”!
!
9! Source: Norgaard & Hornbaek, 2006!
10. The Reality of Think-aloud!
• Current practice goes beyond neutrality and
includes probing questions!
• Some evidence that probing may influence what
usability problems are discovered!
• Evidence is only suggestive; more research is
needed!
10! Source: Krahmer & Ummelen, 2004!
11. What does this mean?!
• Current moderating practices are not
consistent!
• Moderating is often taught through
mentorship; so bad habits can easily be
perpetuated!
• Moderators should be willing to examine
their practices and adapt if needed!
• Therefore, we need to pay attention to our
moderation and remain neutral. !!
11!
17. 5 Rules of Great Moderating!
1. Know your goals!
2. Stay neutral !
3. Be responsible!
4. Be confident!
5. Be genuine!
!
17!
18. 5 Rules of Great Moderating!
1. Know your goals!
2. Stay neutral !
3. Be responsible!
4. Be confident!
5. Be genuine!
!
18!
19. 1. Know Your Goals!
• What are you trying to achieve?!
• Formative !
• Summative !
• What topics are highest priority?!
!
These will influence!
• Where you focus!
• What you skim when time runs short!
• How you fine-tune your moderating style!
19!
20. 5 Rules of Great Moderating!
1. Know your goals!
2. Stay neutral !
3. Be responsible!
4. Be confident!
5. Be genuine!
!
20!
21. 2. Stay Neutral!
• Be Quiet!!
• Don’t insinuate they gave a wrong answer!
✘ “Why did you do that?”!
✘ “What made you think to click that link?”!
• Don’t put ideas into their heads!
✘ “Does entering your SSN make you feel uncomfortable?”!
ü “What do you think about entering your SSN?”!
• Use their words!
• If they call the dashboard a “chart area”, then call it a
“chart area”!
• Keep your tone naturally curious, not derisive!
• “What did you expect here?”!
21!
22. 2. Stay Neutral!
• Avoid closed-ended (Yes/No) questions !
• Don’t start with a verb!
✘ Did you see that?!
✘ Was that difficult?!
✘ Is this frustrating?!
22!
23. 2. Stay Neutral!
• Use Open-ended questions!
• Start with What, When, Where, Why, How!
ü What would you change to make this easier?!
ü When in this process would you be ready to buy?!
ü Where did you expect to find that information?!
ü Why did you say this was easy?!
ü How does that error message make you feel?!
23!
24. 2. Stay Neutral!
• Use neutral, “continuing” response phrases!
• Acknowledge that they gave feedback!
• Don’t imply whether it is good or bad!
✘ “Oh”, “Hmm”, “Interesting” - Assessing!
✘ “OK”, “Yeah”, “That’s good” - Agreeing!
ü “Uh huh”, “Mhmm”, “Tell me more” - Continuing!
24! Source: Boren & Ramey, 2000
25. 5 Rules of Great Moderating!
1. Know your goals!
2. Stay neutral !
3. Be responsible!
4. Be confident!
5. Be genuine!
!
25!
26. 3. Be Responsible!
You have responsibilities to:!
!
• Participant!
• Company!
• Future Users !
• Study Integrity!
26!
27. 3. Be Responsible!
• Informed Consent:!
1. Voluntary!
2. Comprehension!
3. Disclosure!
• The purpose of the study!
• Any reasonably foreseeable risks!
• Potential benefits !
• Notification of recording and observers!
• Confidentiality protections !
• Ability to withdraw at any time without penalty!
27!
28. 3. Be Responsible!
Situations happen during testing.!
You may need to juggle your responsibilities.!
!
What are your priorities?!
!
• Study Integrity!
• Company!
• Future Users!
• Participant!
28!
29. What to do?!
If…!
• Participant doesn’t want to be recorded!
• You don’t have time to ask every question!
• The participant rants about the website and
calls it “crap”!
29!
30. 5 Rules of Great Moderating!
1. Know your goals!
2. Stay neutral !
3. Be responsible!
4. Be confident!
5. Be genuine!
!
30!
31. 4. Be Confident!
• Project confidence!
• If you make a mistake, keep going!
• Don’t over-apologize!
• Pause if you need to collect your thoughts!
• “Fake it until you make it”!
31!
32. 5 Rules of Great Moderating!
1. Know your goals!
2. Stay neutral !
3. Be responsible!
4. Be confident!
5. Be genuine!
!
32!
33. 5. Be Genuine!
• Don’t play the notes on
the page…!
…play the music!
!
• Don’t just read the
questions from the page…!
…have a conversation!
33!
34. 5. Be Genuine!
• LISTEN!
• Listen and respond!
• Stay in the moment!
• Don’t worry about past !
• Or the future!
• Create a conversation!
!
34!
35. 5 Rules of Great Moderating!
1. Know your goals!
2. Stay neutral !
3. Be responsible!
4. Be confident!
5. Be genuine!
!
35!
37. Keep Improving!
1. Identify your improvement areas!
• Watch your videos!
• Have colleagues give you feedback!
37!
38. Keep Improving!
2. Learn from others!
• Watch others moderate!
• Be a participant!
• Listen to talk radio interviews!
38!
39. Keep Improving!
3. Practice skills in everyday life!
• People watch (observe) in public!
• Be quiet and listen in everyday conversations!
!
39!
40. The most important thing is…!
PRACTICE!
• No one is perfect the first time!
• Do your best and continually improve!
• You’ll be awesome in no time!!
40!
42. Thank You!
(Slides will be posted to Slideshare..follow me on twitter for link)
Susan Mercer!
Senior Experience Researcher!
smercer@madpow.com!
@susanamercer!
42!
43. References / Photo Credits!
Boren, T. and Ramey, J. (2000) Thinking aloud: reconciling theory and practice. IEEE
Transactions on Professional Communication, 43 (3), 261-278.!
Dumas, J. and Loring, B. (2008) Moderating usability tests, Morgan Kaufman.!
Dumas, J. and Redish, J. (1999) A practical guide to usability testing, Intellect Ltd.!
Ericsson, K. and Simon, H. (1980) Verbal reports as data. Psychological review. 87 (3),
215-251.!
Krahmer, E. and Ummelen, N. (2004) Thinking about thinking aloud: A comparison of two
verbal protocols for usability testing. IEEE Transactions on Professional Communication,
47 (2), 105-117.!
Norgaard, M. and Hornbaek, K. (2006) What do usability evaluators do in practice? An
exploratory study of think-aloud testing. DIS 2006, 209-218.!
!
!
!
Slide 4: Samantha Louras Photography !
Slide 5: flickr: torgeaux!
Slide 14: flickr: mkorcusa, flickr: Calsidyrose, flickr: marioanima, flickr: breity, flickr: Qole Pejorian!
Slides 27, 29 and 30: flickr: Raphael Quinet, flickr: acme, flickr: betsyweber, flickr: KirkOls!
Slide 32: flickr: bberburb!
Slide 34: flickr: Tulane Public Relations!
Slide 35: flickr: apdk, flickr: marktristan!
Slide 38: flickr: Raphael Quinet!
Slide 39: flickr: l-i-n-k!
Slide 40: flickr: loppear!
!
!
43!
