The document provides guidance on conducting diary studies for user experience research. It defines a diary study as having participants record daily events, tasks, and perceptions around a given subject over time to gain insights. Examples are given of diary studies conducted to examine usability of an existing system, understand how users work with an email marketing system over time, and gauge experience when upgrading to a new product version. Key aspects of designing, recruiting for, managing, analyzing, and reporting on diary studies are discussed, with tips provided for each stage of the process.
You want me to what a practical guide to diary studies
1. You Want me to What?
A Practical Guide to Diary Studies
2. What is a Diary Study?
Definition*:
“Research method that involves providing
participants with the materials and structure
to record daily events, tasks and perceptions
around a given subject in order to gain
insight into their behaviour and needs over
time.”
*http://www.usability.gov/what-and-why/glossary/diary-study.html
3. Mike’s Study
Purpose: to examine an existing system to prepare for a
global redesign
Why a diary study?
• To address the difficulty of determining the impact of
known usability issues during onboarding
• What prevents a motivated user from forming patterns of
use?
• Examined patterns of use, sticking points, goals and
motivations
4. Vicky’s Study
Purpose: Understand how different user types work with an Email Marketing System over
time.
Why a diary study:
● Diary study enabled us to track user actions and satisfaction level OVER TIME – what
changes for the user as they become more skilled?
● Factors we were gauging: blockers to success, satisfaction points, business goals and
motivations, which features were used most frequently
5. Liz’s Study
Purpose: Gain an understanding of the experience of current customers when they are
upgraded to a newer version of the product.
Why a diary study:
● Study conditions recreate the context for transitioning users as closely as possible
● Can gauge how quickly participants adjust to the new version
● Participants have the freedom to report any type of issue or concern
● Longer timeline distinguishes initial learning from persistent issues
6. What can you learn from a Diary Study?
It can provide a better understanding of usage patterns over time:
• Learnability
• Acceptance of a product or product features
• Issues or challenges that impede usage
Generally: most any type of change in behavior or attitude
7. Advantages of Diary Studies
• Authenticity of information collected:
– User’s environment rather than a lab setting
– Participants feel safer so they’re more likely to be open
– Can get past initial learning
• Benefits for participants:
– Less intrusive
– More flexible (e.g., scheduling)
• Can uncover unexpected issues
8. Disadvantages & Challenges
• Overall:
– Time- and resource-intensive
– Logistically unwieldy
– Less control over study environment
• Participants:
– Recruitment, Management, & Retention
• Data:
– Self-reported, but use the resources available to you
– Overwhelming!
9. A Practitioner’s Checklist:
❏ Designing the Study
❏ Recruiting
❏ Managing Participants
❏ Managing the Data
❏ Analyzing the Data
❏ Reporting the Results
10. Designing the Study
Managing Length of Study
• Long enough to capture desired
behaviors
• Stay meaningful for participants and
stakeholders
• Reflect users’ natural work cadence as
much as possible
11. Designing the Study
Components
• Ongoing feedback on user experience
– Diary entries
• Interim check-ins
– Surveys / interviews
• Final review touch point
– Follow-up interviews
12. Designing the Study
Unstructured
• Allows for more unexpected, possibly more realistic data
• BUT, may not get desired data, and lots of manual labor
sorting through results
Structured
• Less time consuming data analysis, higher probability of
getting relevant data
• BUT, may not reflect user’s natural workflow, and a lot of
dictating may feel too formal and restrictive to users
13. Example for study design - Diary Questions
• Name
• What did you do with the tool today?
• What did you want to accomplish?
• Did you accomplish what you wanted?
• What specific features did you use?
• What did you like about your experience today?
• What did you not like about your experience today?
• What else would you like us to know?
• Screen shots (optional)
14. Example for study design - Setup
“Is the tool meeting your
business goals?”
“What features have you
used?”
“Which features do you like
most / least?”
“How has your experience
changed since 3 weeks ago?”
“Have you tried any new
features?”
“Do you feel more skilled than
you did 3 weeks ago?”
15. Example for study design - Phone Interviews
• Follow-up on diary entries and
surveys
• Kept conversation less structured
• Gave deeper understanding of user
goals, issues, etc.
• Benefit of hearing user’s tone
(excitement or frustration)
16. A Practitioner’s Checklist:
✔ Designing the Study
❏ Recruiting
❏ Managing Participants
❏ Managing the Data
❏ Analyzing the Data
❏ Reporting the Results
18. Recruiting
Be realistic about how long this is likely to take
• It will probably take a lot longer than you think to hit your number
• Be prepared to be flexible, prioritize screener criteria if necessary
Strategies for minimizing attrition:
• Graduated payments
– Have payments tied to something you can control (e.g., interviews)
• If you can get participants with intrinsic motivation, that’s a bonus - but don’t hinge your study on it
19. A Practitioner’s Checklist:
✔ Designing the Study
✔ Recruiting
❏ Managing Participants
❏ Managing the Data
❏ Analyzing the Data
❏ Reporting the Results
21. Managing Participants
• Make it as easy as possible for them
– Online data entry
– Include links in every communication
– Have some structure (not completely open-ended)
• Give participants permission to be brief and/or repeat
themselves, and let them be themselves!
– Remember, it’s about a truer contextual perspective
• Guard against being too invasive
– Let participants choose the frequency of reminders
– Make it explicit that reminders are automated
• Monitor for compliance and to answer questions or
redirect to support
22. A Practitioner’s Checklist:
✔ Designing the Study
✔ Recruiting
✔ Managing Participants
❏ Managing the Data
❏ Analyzing the Data
❏ Reporting the Results
23. Managing the Data
There are tons of great online resources -- make use of them!
• Online form for diary entries
– Easy for participants to access
– Researcher has access to data in real time to monitor
• Online self-scheduler for interviews/checkins
• Online form for data entry (e.g., for interviews)
• Survey tools for questionnaires, rating questions etc.
But watch out for:
• Privacy issues
• Appropriateness for your participants
24. A Practitioner’s Checklist:
✔ Designing the Study
✔ Recruiting
✔ Managing Participants
✔ Managing the Data
❏ Analyzing the Data
❏ Reporting the Results
26. Analyzing the Data
It’s largely a manual process:
1. Read through the data looking for issues, patterns etc.
2. Make a list of tags and cross reference with your data
3. Repeat
Qualitative data analysis tools:
• Lots of them available
• Traditionally have been more helpful for ongoing data collection needs than
for a one-time study
• Worth keeping an eye out
27. Analyzing the Data
Strategies:
• Try to Identify categories and tags in advance as part of defining
the study
• “Find” is your friend but watch out for:
– misspellings
– different word forms
– different words for the same/similar concepts
– different meanings for the same words
• Have a method for collecting or flagging verbatims as you go
• Use learnings from other touch-points to help identify tags &
categories
28. Analyzing the Data
Don’t wait until the end!
• Keep the job manageable
• Keep analysis timely
• Allow for mid-course corrections
Include some selected-response measures:
• Easy to analyze
• Consistent across participants and over time
• Provide metrics to supplement open-ended responses
Look for the surprises!
29. My Approach
Two spreadsheets:
1. Raw data: diary entries, numbered chronologically
2. “Processed” data, with columns for:
1. Tags
2. Entry numbers
3. Verbatims
Same 3 Likert questions in all diary entries:
• “easy” results
30. My approach: Get your participants to help you!
Two weeks before the final interview:
• Created individual spreadsheets
• Asked participants to review and think about:
1. Their overall experience making the transition
2. What they liked best
3. What was the biggest challenge
4. What we could have done to make the transition easier
31. DIY vs. DIFM
• Emerging tools like
dscout can reduce
overhead
– The trade-off can be
flexibility
– Decide what you want to
accomplish with your
methodology before you
approach any tools
• We were curious, so we’re
running a study right now
with thanks to our UXPA
organizers!
32. A Practitioner’s Checklist:
✔ Designing the Study
✔ Recruiting
✔ Managing Participants
✔ Managing the Data
✔ Analyzing the Data
❏ Reporting the Results
33. Reporting the Results
Changes over time
• Learnability
– (Qual) Frequency with which issues or problems are reported
– (Quant) How easy/difficult was it for you to meet your goals in this session? To
what extent were you able to meet your goals in this session?
• Long-term patterns of use (what features remain important
over time)
– (Qual) Changes in users’ stated goals
– (Quant) % of use of different features
• Changes in user opinions over time
– (Qual) % of comments classified is positive/negative/neutral
– (Quant) How satisfied/dissatisfied are you with the product?
36. A Practitioner’s Checklist:
✔ Designing the Study
✔ Recruiting
✔ Managing Participants
✔ Managing the Data
✔ Analyzing the Data
✔ Reporting the Results
37. Lessons Learned
Mike:
1. Planned vs. Unplanned learnings (be prepared to dive back in)
2. It all comes down to Codifying open responses
3. Visualize learnings for broad adoption
Vicky:
1. Try to start all participants at the same time
2. Automate as much as possible
3. Review the data as you go - don’t wait until the end
Liz:
1. Structure can be your friend
2. Control what you can but don’t try to control what you can’t
3. Embrace the chaos!
