EffectiveUI's Ari Weissman (Lead Experience Architect) and Lys Maitland (Senior Experience Planner) spoke at Denver Startup Week 2016. Discussion description: Test early, test often. It’s a mantra that’s been proven successful time and again when it comes to innovation and design. So why aren’t you doing it? In the start-up world, when everything is moving so quickly, it can be easy to overlook or postpone collecting feedback from real people because of cost, time, or lack of preparation. Don’t let those things stop you. Valid data can be captured cheaply, quickly, and with half-finished products and strategies. This talk will cover: What is user testing and why is it important How to plan for user testing What are ways to make testing cheaper What are ways to make testing quicker How to test with different fidelities of concept and design How to collect data more frequently Opportunities for getting the whole team engaged What to do with the insights/outcomes of research

2. Sponsors

3. User Testing: Adapt to Fit
Your Needs
#DenStartupWeek
#EffectiveUI

4. Agenda
1. Intro
2. What is evaluative research?
3. How To
4. Engaging the whole team
5. What to do next

5. Ari Weissman
Lead Experience Architect888.310.5327
• 10+ years global
experience
• Responsible for all
things UX, from
research through
experience design
• Project lead for
TimeWarner Cable,
AMEX, FreemanCo, and
Securian
@ EffectiveUI

6. Lys Maitland
Senior Experience Planner888.310.5327
• 15+ years global
experience
• Responsible for all
things UX, from
research through
experience design
• Research lead for
Davita, Scottrade,
NextGear, Securian,
CenturyLink
@ EffectiveUI

7. Insight-driven digital
product company

8. Locations
2162 Market Street
Denver, CO 80205
DENVER
274 North Goodman St,
Unit B264
Rochester, NY 14607
ROCHESTER
85 Broad Street, 18th fl.
New York, NY 10004
NEW YORK CITY

9. Our clients

10. What is evaluative research?
[a.k.a. user testing]

11. Generative v. Evaluative
Generative research
Helps define the problem.
The goal of generative research is to look to
the activities, environments, interactions,
objects and people to find opportunities for
solutions and innovation. These solutions
could be new products or experiences or
they could be an update or improvement to
an existing one.
Evaluative research
Evaluates an existing design (in prototype form or
in final form).
The goal of evaluative research is to test the
existing solution to see if it meets people’s needs,
is easy to access and use, and is enjoyable. This
type of research should be conducted throughout
the development lifecycle, from early concept
design (think rough sketches or prototypes) to the
final site, app, or product.

12. DEFINE DISCOVER DESIGN DEVELOP DEPLOY
EVALUATIVE RESEARCH
Field Research Iterative Concept Testing Usability Testing
Define Problem
Problem Approach
Gather Data
Observational Research
Benchmark Current Site
Distinguish Insights
Analytics Assessment
Create
Concepts
Concept Test
Explore Solutions
Build
Prototypes
Test &
Refine
Collaborate On Options
Discover Gaps &
Problems in Experience
Analytics Strategy
Implementation/
Change Management
Continuous
Measurement
Test &
Refine

13. Some examples of evaluative research
Testing an IA or design learnability
CLICK TEST
Testing flows and content
USABILITY TESTING
Testing and measuring current
state for redesign
BENCHMARKING
Closed card sort to validate an IA
CARD SORTING
Iterative testing of flows and
concepts, some generative research
ITERATIVE DESIGN TESTING

14. Benefits of User Testing
1 Validate assumptions 4 Provide justification for
additional effort
2 Decide between options 5 Reveal how users think about the
problems your solution solves
3 Discover hidden issues 6 Removes opinion from the
design process

15. HowTo

16. How To
1. Define the goals
2. Define the audience
3. What to test
4. $ -> $$$
5. Debrief
6. Analysis
7. Outcomes/Deliverables

17. What are
your goals?
888.310.5327
Some questions to ask:
- Why are you testing and what will
you do with the data?
- What do you need to know / learn /
validate? Why?

18. Who is your
audience?
888.310.5327
Some questions to ask:
- Who will use of the thing you’re
building?
- Are there different groups of users?
If so, which have the highest
priority?
- Where are they?

19. Recruiting
Screener
• Script used to recruit
• Intro to the study
• Questions to find
participants who
demonstrate desired
behaviors
• Offer an incentive
Schedule
1 hour interviews with at
least 30 minutes between
sessions
Recruit
• 5-7 participants per
segment
• 5 participants finds 85%
of usability issues
1 2 3

20. Participants are good to the extent they represent your target. If your
participants don't match your target, your study will be useless. You can
learn valuable things by asking the right people the wrong questions.
If you are talking to the wrong people it doesn't matter what you ask.
Bad participants undermine everything you’re trying to do.
— Erika Hall, author of Just Enough Research
“
”

21. Run the first session as a
pilot to work out the kinks
and refine the moderators
guide. Schedule an hour
after the first to re-tool as
needed.
PRO TIP

22. What can I
test?
ANYTHING
- CONCEPT
- HAND SKETCHED MESSY
- HAND SKETCHED NEAT
- WIREFRAMES
- VISUAL DESIGN
- CLICK-THROUGH PROTOTYPE
- CODED PROTOTYPE
- FINAL PRODUCT

24. The quick and dirty

25. Why quick and
dirty?
- NO TIME
- NO MONEY
- NO FACILITIES
- TESTING SOMETHING SMALL

26. Quick and dirty
Location
Anywhere!
Logistics
Find a place
Find some people
Ask some questions
Write down answers
Costs
Minimal to none
Fidelity to test
Any
1 2 3 4

27. Middle of the road

28. Why middle of
the road?
- NEED SOME DOCUMENTATION TO
SHARE WITH TEAM OR SAVE
- TOO MUCH CONTENT FOR QUICK
AND DIRTY
- WANT TO HAVE OBSERVERS
- DESIRED USERS ARE NOT READILY
AVAILABLE

29. Middle of the road
Location
Preferably in context,
though may be remote
In a quiet room at a
table
Logistics
Schedule a location and
time for one hour
Recruit participants and
consider incentive
Prep discussion guide
Create data capture
sheet
Audio/Video recording
Costs
Location
Recruiting and
incentives
Screen sharing/
Recording tool
Fidelity to test
Typically wireframe or
above
1 2 3 4

30. Formal

31. Why formal
testing?
- REQUIRES FORMAL DOCUMENTATION
- DESIRED USERS ARE NOT READILY
AVAILABLE
- STRICT DISCUSSION GUIDE
REQUIREMENTS
- LOTS OF THINGS TO TEST
- NEED TO INCLUDE NOTETAKERS AND
OBSERVERS
- MULTIPLE DEVICES OR COMPLEX
TECHNOLOGY REQUIREMENTS

32. Formal
Location
Quiet room with a table
Separate location for
observers
Quality phone if remote
Logistics
Schedule a location and
time for two hours
Recruit participants and
distribute incentive
Prep/share/revise
discussion guide
Create data capture
sheet
Audio/Video recording
Costs
Location
Recruiting and
incentives
Screen sharing/
Recording tool
Snacks and beverages
Transcriptions
Fidelity to test
Typically wireframe or
above
1 2 3 4

33. Facilitation
Do you have 3 more hours?

34. Facilitation: The highlights
Asking questions
Ground your questions
• Will you How have
you…?
• In general In this
specific instance…?
• Tell me how Show me
how…?
Ask why? A lot!
Rhythm
• Ask
• Listen
• Probe
• Validate
3 Types of tasks
• Specific verb based
tasks (do, send, complete)
• Scavenger hunt (find,
discover)
• Interview (tell me about)
1 2 3

35. So what I heard was, you
do X and Y because of A
and B, but a little bit of 123
would make it better. Is
that right?
PRO TIP

36. The Debrief
888.310.5327
Debrief after EVERY session
It’s a quick assessment of a session:
- Summary of the participant
- What went well
- What needs improvement
- Insights or surprises
- Opportunities

37. Analysis and Synthesis
Trends
Look for trends across
all participants or
within a participant
segment
Good and Bad
Document what
doesn't work AND
what does work
Prioritize
Choose the most
important findings to
share and focus on
solving
Think big picture
Is the issue identified a
symptom of a bigger
problem?
Quotes
Identify quotes that
capture the essence of
your insights
Solutions
What solutions were
identified by
participants or
observers?
Future
Opportunities
What do the outcomes
suggest would be a
good next step?
Centralize and
share
All data in one place
for access and
redundancy. Shit
happens.

38. Examples of analysis
OUTPUTS

39. Spreadsheet
39

40. Post-its
40

41. Live

42. Examples of evaluative
research results
OUTPUTS

43. $
quick and
dirty

44. $
quick and
dirty

45. $$
middle of
the road

46. $$$$
formal

47. Engaging theWholeTeam

48. Engage your
team at every
step
- CHOOSING GOALS AND AUDIENCE
- REVIEWING DISCUSSION GUIDE
- OBSERVING AND NOTE TAKING
- DEBRIEFING
- PROBING DURING ANALYSIS
- PRIORITIZING RESULTS
- DETERMINING NEXT STEPS

49. What to do with outcomes

50. What to do with outcomes
1 4
2 5
3 6
Share your findings
(email, present, print, posterize)
Create a roadmap with priorities
Identify opportunities Support ideation
Stay engaged!
Be available to answer questions.
Make recommendations
7 Test Again!

51. Questions?
Vragen
Preguntas
問題
Des questions
‫שאלות‬
вопросов
คำถาม
frågor

52. Thank you.
Ari + Lys
P.S. We’re hiring!
http://www.effectiveui.com/company/careers/

53. www.linkedin.com/company/effectiveui
LINKEDIN
https://twitter.com/effectiveui
TWITTER
www.facebook.com/InsideEffectiveUI
FACEBOOK
www.instagram.com/effectiveui
INSTAGRAM
https://medium.com/@effectiveui
MEDIUM
Stay connected with EffectiveUI

54. https://abookapart.com/products/just-enough-research
JUST ENOUGH RESEARCH BY ERIKA HALL
http://rosenfeldmedia.com/books/interviewing-users
INTERVIEWING USERS BY STEVE PORTIGAL
https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users
WHY YOU ONLY NEED TO TEST WITH 5 USERS BY JAKOB NIELSEN
Mentioned References