Effective Digital Diary Studies / UXPA Webinar May 2016

1. Eﬀective Digital Diary Studies Sara Cambridge sara@gotoresearch.com UXPA Webinar: May 18, 2016

2. What do diary studies oﬀer that interviews don’t?

3. 3 Interviews don’t capture the range of ways people do things or the rich, in-the-moment details that people quickly forget.

4. Fitness Tracker Case Study 40 people who use ﬁtness trackers were recruited to participate in several digital diary studies over 8 months. The ﬁrst diary study had them report on 5 things they do with their tracker. With every entry, they submitted a short video and answered a few questions. 44

5. 5 Erica, 33 “It’s 10:53pm, and I’m only at 7388 steps….we’re about to go take the dog for a walk. Got to make sure we get to our 10,000 steps before the end of the night.” Show us how you use your ﬁtness tracker

6. Digital diary studies excel at: • Gathering contextual information cost-eﬀectively • Capture repeated behaviors in context • Providing instant feedback on products • Uncovering latent needs in general problem space

7. What This Presentation Will Cover What a Diary Study is Advantages of Diary Studies Platforms for Conducting Digital Diary Studies How to Conduct a Digital Diary Study Highlights for Ensuring an Eﬀective Digital Diary Study 7

8. What is a Diary Study?

9. A Diary Study is a Cultural Probe Cultural probes use a collection of tools, artifacts and tasks intended to provoke the user to think about their environment in new ways….  With minimal intrusion, researchers can glean insights into participants’ environments that can help to identify problems and uncover new opportunities. 9 Adam Starkman & Jennifer Chow http://designresearchtechniques.com/casestudies/cultural-probes/

10. Key Elements of Diary Studies Theme-based: the subject and questions are suggested by the researcher. Self-reported: the actual reporting is done by the participant, using whatever platform/prompts were provided. In-the-moment: the participant is asked to do the reporting immediately after engaging in the activity, in order to capture as much detail as possible. Longitudinal: the goal is to capture a range of uses, rather than a singular experience. This can include how the user’s experience evolves over time. 10

11. Traditional Diary Study Mozilla used paper diary studies in 2012 to learn how participants were using bookmarks in Firefox. They asked 10 people to record their experiences with bookmarks over 3 days. 11 Brian Groudan, Mozilla research, 2012 / https://blog.mozilla.org/ ux/page/24/

12. Traditional Diary Study People came in for an in-person interview afterwards, where they discussed their entries and sorted them by theme. Guiding design principles were culled from the analysis that were the basis for design better bookmarking tools. 12

13. Why Use   Diary Studies?

14. What’s so great about ‘in-the-moment’ feedback? Can’t you get the same information   in an interview a few days later? 14

15. 15 Most see it like a video camera…. How Does Memory Work? but really it’s like a slow leak

16. The Psychology of Habits 16The Power of Habit, Duhigg Behavior trigger Sequence of actions occurs repeatedly Reward / Finish Our brains turn the sequence into one automatic routine; “chunked”

17. The Psychology of Habits 17The Power of Habit, Duhigg We can’t access the details of habitual actions until we start them Diary studies ask people to report when details are still accessible This also allows for collecting workarounds people are not aware of Behavior trigger Details of actions become accessible Reward / Finish Habit sequence executed

18. What is a Digital Diary Study?

19. 19 Getting people to actually do the reporting is a major obstacle for traditional diary studies. http://kotaku.com/hong-kong-residents-supposedly-cant-live-without-their-476758731

20. 20 Smartphones are ideal for digital diary studies since people use them in short spurts throughout the day. http://kotaku.com/hong-kong-residents-supposedly-cant-live-without-their-476758731 Average use: 85 times a day* *Study of 23 people aged 18-33: http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0139004

21. Tools for Executing Digital Diary Studies Platform (web/app) SMS/Text Email Blog Video 21

22. Digital Diary Study Platforms 22 Name Platform Cost 20/20 Research Web/app diary study high 24Tru Web/app diary study high CrowdTap Web/app diary study high UX 360 Web diary study high Dscout App diary study medium Dedoose Web diary study low Typeform online survey low

23. 23 Diary Study Platform Advantages: • Oﬀers analysis; faster to insights • Lots of data collection options • Many have panels for recruiting • Guidance, support, education • Team members can participate • Many participants = quant data Disadvantages: • Cost • New tool • For more tech-savvy participants • Can’t adjust once launched $595 for recruiting $295 for each study

24. 24 Most diary studies are a mix of multiple choice and open ended questions.

25. Example Dscout Diary Study:  New Product Launch Goal: Get feedback on a new digital entertainment product Details: 5 families, 4 months, in-person interviews throughout Platform: Dscout app, asking them to report every time they used the service, answering a series of questions. Key Finding: The service was most robust for mobile viewing, but the families did the bulk of their viewing on TVs. Surprising Finding: The study uncovered signiﬁcant technical errors that were unknown to our client. 25

26. 26 SMS/Text Advantages: • Free • Good for less tech savvy participants • Can adjust on the ﬂy • More likely to get immediate response Disadvantages: • All analysis is manual • Limited data input

27. Example SMS Diary Study: Tracking Diabetes Patients Goal: Learn how often Type 2 Diabetes patients actually check their glucose level, as well as how often they have their glucose monitor with them Details: 6 participants, 4 weeks, in-person interviews at the beginning and the end Daily Text: How are you feeling today? Have you checked your blood glucose today? Where is your meter now? Key Finding: People check their glucose & have meter less often than was assumed. Several aspects were identiﬁed that dictate how often people check their glucose Study Insight: Asking how they were feeling made them feel cared for, rather than reminded/judged 27

28. 28 Email Advantages: • Free • Good for less tech savvy participants • Can adjust on the ﬂy • Likely to get more details than if using SMS Disadvantages: • All analysis is manual • Emails are easy to overlook

29. Example Email Diary Study:  New Product Launch Goal: Understand how and why people use different channels when managing their investment account Details: 18 participants, 2 weeks, in-person interviews at the beginning and the end Daily Emails: Questions to understand any activity taken regarding investment accounts (activity taken, channel used, how felt about it, etc), plus occasional tasks for probing deeper on specific aspects Surprising insight: People sometimes conflate how often they think about their money with how often they check their accounts Biggest Challenge: Not having enough time to fully analyze data 29

30. 30 http://www.uxmatters.com/mt/archives/2012/01/video- diaries-a-method-for-understanding-new-usage- patterns.php Video Advantages: • Can collect rich content • Edited video = rich storytelling • Most people send cameras back Disadvantages: • All analysis is manual • Transcribing video • Cost of multiple cameras • Ease of use is critical

31. 31 http://www.eﬀectiveui.com/blog/2012/03/29/using-digital-diary- studies-to-understand-the-customer-experience/ Blog Advantages: • Free • Can adjust on the ﬂy • Likely to get more input than SMS Disadvantages: • All analysis is manual • Researcher must set up accounts ahead of time

32. How to Conduct a Digital Diary Study

33. The 3 Phases of a Diary Studies 33 The goal: a concise, focused study with the right participants. The goal: high participation and high quality data. The goal: to turn data into relevant insights.

34. • Clarify your research question • Reﬁne and pretest your study • Minimize the work for your participants • Recruit the right participants

35. Clarify Your Goals & Research Question Before Building the Study Deﬁne the goals for the study. What do you want to learn? Turn that into a research question that will guide your eﬀorts as you create your diary study. 36

36. Fitness Tracker Study Research Question: How are activity trackers used by people attempting to make concrete improvements to their ﬁtness habits? 37

37. What Type of Diary Study Will   You Build? Ideation: Collect ideas/suggestions.   “Show me your ideal…” Inventory: Understand key things participants use, own or do.   “Show me all of your...” Moments: Capture experiences as they happen in real time.   “Show me every time you…” Process: Understand key steps of a process in real time.   “Show me your process for…” Reﬂection: Explore likes, dislikes, memories and motivators.   “Tell me how you feel about….” 38 http://dscout.helpscoutdocs.com/article/121-tutorial-2-building-your-mission

38. 39 Consider how often you ask them to report. It should be enough to collect useful data, but not enough to cause fatigue. Help Your Participants Succeed By Reducing the Reporting Burden

39. 40 Pairing themed multiple-choice questions with open-ended ones creates a logical ﬂow for people and answers both “what” and “why”.

40. 41 Another beneﬁt of multiple-choice is you can quickly see trends as the data comes in.

41. 42 Open-ended questions provide the richest data but are more work for both for you and the participant. Use them sparingly. I think it should send reminders on when to be more active when I'm lagging in my goal such as vibrating. I do wish that other activities could be tracked with the Fitbit without having to input it myself. Maybe if I could get some sort of notiﬁcation on my phone if a speciﬁc friend or group of friends had passed me.

42. 43 Pretest by sitting down with someone as they go through every step. Look for awkward or unclear phrasing.

43. 45 Diary studies generate a lot of data. It is good to start small. 10 x 10 x 10 =questions participants entries data points 1000

44. 46 If recruiting through Dscout, you’ll only pay incentives for participants that ﬁnish. Expect to pay $5-$20 per entry. We paid $25 for a total of 6 entries. If you’re working with a specialized or busy population, you may need to pay as much as $20/entry.

45. 47 We looked for participants who have a ﬁtness tracker and were regularly using it as part of a ﬁtness or weight loss goal. Find the Right Participants

46. • Review & make notes from your data as it’s coming in • Be persistent with participants • Ask them questions for clarity • Do post-study interviews

47. Watch the data as it comes in. This is the fun part. Are your questions holding up?   If not, can you ﬁx it?   Either way, learn for next time. 50

48. 51 Why did you get your first fitness tracker? Larry, 31 “I got a fitness tracker because was about 300 lbs, on a diet, exercising, wasn't losing weight. I knew there was something missing. I wanted to learn how many calories I was burning so could know how little or much I should be eating. That was the missing link. I have lost 150 lbs, and the only way did it was because I started using Jawbone Up.”

49. 52 Take notes of interesting patterns and questions to ask participants as you go through it.

50. 54 Minimize your drop oﬀ rate. Stay in touch with your participants. Be friendly and persistent.

51. 55 Ask them questions if you want more clarity about their answers. Be appreciative and interested in what they have to say.

52. 56 Do follow-up interviews with the most engaged participants. You already have rapport and can dig deeper into their reporting and actual behaviors.

53. • Use the built-in analysis tools • Start by answering your initial question(s) • Identify patterns and/or exceptions • Share your ﬁndings with real examples

54. 58 The good news is that by simply reviewing and making notes on the data, you have already started on the analysis.

55. 59 Diary Study platforms oﬀer tools to analyze the multiple choice and rating questions, reducing the work the researcher has to do.

56. 60 If analyzing manually, data should be entered into a spreadsheet so it can be coded, reﬁned and sorted.

57. What Further Analysis Should be Done? That depends on many variables, such as: • What your initial question/s were • How surprised you are by the data that has come in • How it ﬁts into other research methods • Your ﬁnal deliverables 61

58. Digital Diary Study   Key Takeaways

59. Main Benefits of Using Diary Studies • To collect rich stories about how and why people do things • Get intimate glimpses of how a product fits into people’s lives • Allows for understanding evolving use of a product • Uncover latent needs for new product development • Cost-effective way to collect contextual insights 63

60. Checklist for a Great Diary Studies Phase 1, Prepare: • Have a clear, focused research question to guide your eﬀorts • Have an appropriate mix of closed/open-ended questions • Pre-testing it to ensure clarity and understanding • Start small & recruit the right participants • Reduce the burden on them Phase 2, Execute: • Watch the data and follow up with participants during the study • Do follow-up interviews with the best participants Phase 3, Analyze: • Answer your initial question ﬁrst • Give yourself enough time to analyze the data 64

61. Tips for Building Your Own Diary Study Before you can use it, you may have to sell it. Here are a few things to keep in mind: • It’s an extremely ﬂexible tool; make it work for your needs • Start small, let it prove it’s value to your organization and yourself (no-cost platform, 5-6 participants) • Evangelize the insights and stories you collect to build more support. • You become an expert by doing. So dive in! 65

62. For a 4-min video about diary studies:   www.gotoresearch.com/speaking/sara/ Questions about diary studies? Ask! sara@gotoresearch.com @ maybanks