Slides from a presentation given at the September 2013 UXMad meetup

1. Research Methods in UX
UXMad
9/19/13

2. Schedule
● Socializing/Mingling
● Intros/Announcements
○ Jobs/Opportunities
● Brief Presentation
○ Why research?
○ What research is
○ A few methods
● Workshop!
● Afterparty?

3. About Me

4. Brad Orego
me@bradorego.com
@bradorego
B.S. in Computer Science, B.A. in Psychology
UX Lead at Nextt
UXMad Meetup Co-Organizer
Entrepreneur
Dancer, Curler, Ultimate-er, Cyclist, Beer-batter pancake blogger

5. Why research?

6. Why?
● UX is important
● UX sells
● UX is the key differentiator
● People matter
● We aren’t our users
● Build a solution that people want
● Solve a problem people have

7. How?
● Qualitative
○ Focus Groups/Interviews
○ Design Ethnography
○ Usability Studies
○ Diary Studies
● Quantitative
○ Surveys
○ Metrics
○ Usability studies

8. Details!
Qualitative

9. Qualitative
● Goals
○ Meaning - how people see the world
○ Context - the world in which people act
○ Process - what actions and activities people do
○ Reasoning - why people behave the way they do
● Attributes
○ Creativity - what might be interesting to see?
○ Subjective - influence by personal opinions
○ Inductive reasoning - few examples, then extrapolate
● Data
○ field notes, recordings, diary entries, …

10. Focus Groups & Interviews
● Benefits
○ may be structured, directed, detailed
○ put real people/faces/stories behind data
● Problems
○ highly subjective; open to bias by interviewer
○ not in-situ
● Tools
○ Flow - keep them talking when it’s interesting
○ Non-direction - let them meander
○ Transition - close current; open new conversation
○ Depth - tell me more

11. Design Ethnography
● Comes from anthropology
○ Science!
● In-situ
○ observation, interviews
○ get the real story
● How
○ What people say, do, and use
○ Why they do it this way. Why not another way?
● Be…
○ Non-disruptive, non-interventionist, unbiased

12. Usability Studies
● More appropriate for actual artifacts
○ mock-ups, existing systems, prototypes, etc
● Methods
○ Heuristic Evaluation - rate it against a set of criteria
○ Conceptual Modelling - show & tell
○ Direct Observation - watch people play with it
● Interviews
○ Retrospective - record it then show them
○ Critical Incidence - the parts that stood out
● Data
○ audio/video recordings, handwritten notes

13. Diary Studies
● Participants answer over a period of time
○ Same questions? Different questions? Open/closed?
● Pros
○ Longitudinal, in-depth self-report data
○ Highly detailed; very realistic
○ Trends over time
● Cons
○ Expensive; difficult to get right
○ Self-report data
○ Availability Bias

14. Numbers!
Quantitative

15. Quantitative
● Goals
○ Numbers
○ Trends in data
○ Read between the lines
○ Answer specific questions
● Attributes
○ Large(r) sample sizes
○ Phone, online, in-person
○ Open vs. closed questions
● Data
○ Means, medians, SDs, regressions, correlations

16. Surveys
● Questionnaires with specific measures,
hypotheses, and results
○ SPECIFIC
● Types of Questions
○ Multiple Choice; Scalar; Matching/Association
○ Open-ended
● Things to consider
○ What to ask? How to ask it?
○ Ordering effects
○ Validity threats

17. Metrics
● Measuring real behavior, interactions, use
○ Doesn’t have to be on a real system
● Numbers!
○ Click-through rates
○ Time spent on an activity
○ Error rate
● Method
○ Have a specific thing to measure
○ Figure out how to measure it
○ Figure out why the results are the way they are

18. Usability Studies
● Not as hard-and-fast as other Qualitative Data
○ In-person, so you can dive deeper
● Be specific (numbers!)
○ Measure success/failure rate
○ Measure time
○ A/B test
● Continuous Evaluation
○ Measure real systems in real use
○ Metrics, analytics, log files

19. How?
● Qualitative
○ Focus Groups/Interviews
○ Design Ethnography
○ Usability Studies
○ Diary Studies
● Quantitative
○ Surveys
○ Metrics
○ Usability studies

20. Now what?

21. Tonight's Meetup
● Socializing/Mingling
● Intros/Announcements
○ Jobs
● Brief Presentation
○ Why research?
○ What research is
○ A few methods
● Workshop!
● Afterparty?

22. Workshop!

23. References/Further Reading
● http://carmster.com/hci/
● http://www.cs.rochester.edu/courses/212/fall2012/
● http://www.upassoc.
org/usability_resources/usability_in_the_real_world/roi_of_usability.html
● http://www.youtube.com/watch?v=O94kYyzqvTc
● http://uxmag.com/topics/research-methods-and-techniques
● http://www.usability.gov/
● http://www.nngroup.com/articles/which-ux-research-methods/
● https://speakerdeck.com/tsharon/validating-assumptions-with-12-ux-
research-methods
● http://www.uxbooth.com/articles/complete-beginners-guide-to-design-
research/
● http://www.slideshare.net/dgcooley/introduction-to-ux-research-methods
● http://www.nngroup.com/articles/discount-usability-20-years/