Growing pains: Launching our first longitudinal study
•Télécharger en tant que PPTX, PDF•
1 j'aime•739 vues
In our ongoing quest to discover how our users think, our UX group conducted a four-month-long study. As we would soon discover, this was no mean feat, as there was very little precedent from the UX community on the nuts and bolts of longitudinal studies. We learned the hard way how to actually set up a longitudinal study, how to maintain it, how to collect and analyze data, and perhaps most importantly, how to get real insights that can go into design solutions. In this session, I'll present a case study of our methodology, our insights, what mistakes we made, and what you can do to make qualitative research a success in your organization.
Recommandé
Recommandé
Contenu connexe
En vedette
Similaire à Growing pains: Launching our first longitudinal study
Similaire à Growing pains: Launching our first longitudinal study (16)
Dernier
Dernier (20)
Growing pains: Launching our first longitudinal study
- 1. Growing pains Leanna Gingras - @leegoesplaces Internet User Experience 2011
- 2. Hi. IUE October 2011 @leegoesplaces
- 3. IUE October 2011 @leegoesplaces
- 4. IUE October 2011 @leegoesplaces
- 5. IUE October 2011 @leegoesplaces
- 6. IUE October 2011 @leegoesplaces
- 7. IUE October 2011 @leegoesplaces
- 8. Grab ‘n Go Gus “this looks good enough.” “that’s not it…that’s not it either…” Savvy Sally IUE October 2011 @leegoesplaces
- 9. IUE October 2011 @leegoesplaces
- 10. 1. Our research need IUE October 2011 @leegoesplaces http://www.flickr.com/photos/lwr/2446626352/sizes/l/in/photostream/
- 11. “this looks good enough.” Grab ‘n Go Gus
- 12. Selecting our tools IUE October 2011 @leegoesplaces
- 13. “A longitudinal study is a correlational research study that involves repeated observations of the same variables over long periods of time.” --Wikipedia IUE October 2011 @leegoesplaces
- 14. UX resources are sparse http://dux.typepad.com/dux/2009/05/usability-over-time-longitudinal-research-studies.html “I participated in the "Best Practices in Longitudinal Research" workshop at CHI 2009 with a small group of academics and practitioners from five countries. … Here's a brief sampling of a few topics from the day's discussions.” User experience over time: An initial framework “This paper presents an in-depth, five-week ethnographic study that followed 6 individuals during an actual purchase of the Apple iPhone". We found prolonged use to be motivated by different qualities than the ones that provided positive initial experiences.” (and many abstracts and workshops) IUE October 2011 @leegoesplaces
- 15. IUE October 2011 @leegoesplaces
- 16. Psychology resources galore To learn how to retain participants: Minimizing respondent attrition in longitudinal research: Practical implications from a cohort study of adolescent drinking More on retaining participants: Retaning Participants in Longitudinal Community Research: A Comprehensive Protocol Examples of how much to pay: Psychology Department at the University of Vermont IUE October 2011 @leegoesplaces
- 17. IUE October 2011 @leegoesplaces
- 18. 2. Planning http://www.flickr.com/photos/dacotahsgirl/299198494/sizes/o/in/photostream/ IUE October 2011 @leegoesplaces
- 19. Decide the research question “The goal of this study is to learn how a novice researcher’s research patterns and strategies change over time.” IUE October 2011 @leegoesplaces
- 20. Decide who to study “The goal of this study is to learn how a novice researcher’s research patterns and strategies change over time.” IUE October 2011 @leegoesplaces
- 21. Decide what data to collect “All of it.” IUE October 2011 @leegoesplaces
- 22. Decide what data to collect “We will be collecting data from usability tests, interviews, Twitter entries, diary entries, and focus group activities.” IUE October 2011 @leegoesplaces
- 23. Plan touchpoints and activities 1 - Initial interview & usability test 1 - Contact the recruits for longitudinal study 1 - Orientation, 1 – Follow-up email 3 - check-in emails 1 - Mid-semester interview based on tweets 1 - End-semester interview 1 - Focus group IUE October 2011 @leegoesplaces
- 24. Plan compensation strategy IUE October 2011 @leegoesplaces
- 25. Plan retention strategy IUE October 2011 @leegoesplaces
- 26. 3. Execution http://www.flickr.com/photos/goincase/2720494170/sizes/o/in/photostream/ IUE October 2011 @leegoesplaces
- 27. The actual study 1 usability test 10-30 tweets At least 3 diary entries 3 in-person interviews Focus group IUE October 2011 @leegoesplaces
- 28. Screener usability tests IUE October 2011 @leegoesplaces
- 29. Screener usability tests IUE October 2011 @leegoesplaces
- 31. Plan dates for touchpoints throughout semesterIUE October 2011 @leegoesplaces
- 32. Twitter guidelines IUE October 2011 @leegoesplaces
- 33. Diary due dates IUE October 2011 @leegoesplaces
- 34. Diary entries IUE October 2011 @leegoesplaces
- 35. Interviews Standard set of questions: “Tell me about your toughest moment.” Observations from Twitter/diary to build interview from: “Does she understand where to find scholarly stuff?” Exercise: “Let’s revisit one of your earlier Tweets. Walk me through finding resources on Faust. Is there anything you’d do differently?” IUE October 2011 @leegoesplaces
- 36. End of semester focus group “so how did your research change over the semester?” IUE October 2011 @leegoesplaces
- 37. End of semester focus group “so how did your research change over the semester?” IUE October 2011 @leegoesplaces
- 38. End of semester workshop “so how did your research change over the semester?” IUE October 2011 @leegoesplaces
- 39. IUE October 2011 @leegoesplaces
- 40. Building activities IUE October 2011 @leegoesplaces
- 41. Articulating activities IUE October 2011 @leegoesplaces
- 42. YAY! We’re done, right? http://www.flickr.com/photos/markfive/4121088992/ IUE October 2011 @leegoesplaces
- 43. 4. Analysis IUE October 2011 @leegoesplaces
- 45. What keywords did she try? http://www.island94.org/2009/09/boston-subway-in-vector-format-svg/ IUE October 2011 @leegoesplaces
- 46. Asked specific questions of the data “The goal of this study is to learn how a novice researcher’s research patterns and strategies change over time.” What do they learn? What tools and hacks are useful to them and why? What do they hate and why? IUE October 2011 @leegoesplaces
- 47. “What do they hate and why?” IUE October 2011 @leegoesplaces
- 48. Archived everything IUE October 2011 @leegoesplaces
- 49. What’d we find out? IUE October 2011 @leegoesplaces
- 50. Finding: Emotions run high IUE October 2011 @leegoesplaces
- 51. IUE October 2011 @leegoesplaces
- 52. Finding: Wikipedia is a broad resource IUE October 2011 @leegoesplaces
- 53. Finding: Resources drive research IUE October 2011 @leegoesplaces
- 54. Yeah, but what about the deliverables? IUE October 2011 @leegoesplaces
- 55. Cross-unit brainstorming IUE October 2011 @leegoesplaces
- 56. Persona-building
- 57. Project steering IUE October 2011 @leegoesplaces
- 58. Recycle, recycle, recycle IUE October 2011 @leegoesplaces
- 59. 5. Lessons learned IUE October 2011 @leegoesplaces
- 60. Study management takes time IUE October 2011 @leegoesplaces
- 61. Retain ‘em IUE October 2011 @leegoesplaces
- 62. Nag, nag, nag IUE October 2011 @leegoesplaces
- 63. Make it easy for them IUE October 2011 @leegoesplaces
- 64. Adapt. IUE October 2011 @leegoesplaces
- 65. Define research need Planning Execution Analysis and findings Lessons learned IUE October 2011 @leegoesplaces
- 66. Questions? http://www.flickr.com/photos/picken/5694078937/sizes/o/in/photostream/ IUE October 2011 @leegoesplaces
- 67. Thanks Leanna Gingras @leegoesplaces usabilityfail.com
Notes de l'éditeur
- This IS: The insights and the many many failures that we experienced doing our very first longitudinal study. Didn’t have many UX resources to look to. So I’m going to give a case study and Hopefully you can learn from us. This is NOT: a step by step guide, or me telling you that a longitudinal study is right for you.
- Who I amWho JSTOR is
- Our mission is to help scholars do better work. Who works here?: Developers, librarians, UX people, business people.
- This website providesscholarly materials, journals, etcHow it usually works – a library will purchase access to our website for the university, and the people
- Demographic-wise, our user base looks roughly like this ON PAPER. These are the people who have access to jstor.
- Through user research and surveys, etc, we’ve learned that in terms of who actually uses JSTOR and forms a relationship with JSTOR, the picture is a lot like this.
- Different researchers, different goals and methodsGUS: the typical undergrad student. Just wants to finish an assignment. Google, google scholar. Might not even really understand who JSTOR is and how they got there.Sally: the typical grad student. They have a topic they care about and are undertaking an effort to get stuff into their brains, and then use their brains to produce new and interesting stuff. Know what tools they have available.
- However, we’re missing the full picture. What we did: Inhouse, offiste, quantitative, qualitative. We understand how people use JSTOR, but not how it fits into their goals.
- People come into our lab for an hour and we never see them againWe see many students with many skills, but we never see how they develop those skillsNeed to go BEYOND JSTOR into the Lifecycle.
- In order to learn how people develop skills over time, we decided to learn more about Grab ‘n Go Gus.We’re interested not just in how they use JSTOR, but how they do stuff.
- Now that we’ve defined our RESEARCH NEED, it’s time to find the BEST METHOD to fulfill this need.
- We found a couple cool resources, but not many
- WE Turned to psychology for inspiration
- WE WE WE WE
- WE googledgoogledgoogled. Every situation is different, google it for yourself.
- I’m going to run through a case study of how we did the longitudinal study. This won’t be an extensive, easy-to-follow guide on launching a longitudinal study but you can get ideas, dos, don’ts.
- WE narrowed our research focus to a single articulable question. This question did NOT focus just on JSTOR or Google, but on their entire research workflow.
- Your “who” is very important and will be highly targeted
- WE needed to decide, based on OUR resources (time&money), what info to collect. Prioritize. What can we realistically collect?
- We decided: Of all the methods available to us, What will help US answer that question most effectively?We needed a mix of in-person AND offsite methods, so this. The POINT of tweets: ____
- We know the going rate of a regular usability test – the usability profession does this all the time. We tried to break down what we required and price everything out. This was very iterativeHow do we value their time and the work they give us? Finding the right balance of amount of work to ask of them;How much does it cost to interest an undergrad for the duration of a semester?
- MONEY - How do we space out payments.Keep them interested, but don’t pay too much.Keep them enticed, but don’t pay in advanceTRUST Cultivating a good trust relationship with themTOUCHPOINTS – clear ones. Finding easy ways for them to give us that information
- Multipurpose.We evaluate some site functionality as normal, but ALSO evaluate whether they’re good fits for our studyAre they suited to do diary studies and tweets - Do they seem able to recognize and verbalize their problems? Story: “JSTOR isn’t working right” versus “I use Google at the library all the time and it usually gives me stuff”
- AND we evaluate whether they’re good participants for the study. Red flagsUsers who show up lateUsers who give up on a task easily Users who don’t reply to email promptlyUsers who don’t complete forms as instructed
- Was essential to set expectations and lay out study requirementsand make sure everything’s understood. Also help participants to know they’re not the only ones.Study expectationspay schedule Contact information, etc
- Reminder: Point of tweets – in situWe were concerned that they wouldn’t know what we want them to tweet, “tweet” is kind of vague. Want them to tweet things that are helpful to US
- Orientation, let them know about study requirements, give them their materials and twitter accounts
- The interviews were TONS OF WORK.Did one at beginning, middle, end of semester.Opportunity to roll in data and follow up.
- We did individual end-of-semester retrospective interviews. AFTER THAT, We wanted to bring people together and have them bounce ideas off each other. While we were planning the focus group we shifted it a little bit.
- Not a traditional focus group with questions and answers, self-report data has too many problems with it. And it’s not engaging.
- What makes it a workshop? We weren’t asking questions so much as we were asking participants to construct models of their research experience
- Designed the workshop to get us the best conversation possible: burritos for incentive, sharpies, and post-its
- STORY: They did: arranged from least useful to most useful “forums” “scholarly articles”What we actually see: actually a credibility continuum.
- Activity designed not to give us data or a model, but to get them talking and thinking in a tangible way.Affinity diagram!
- Nope, not done.We had a firehose of data.
- Firehose of data. I have some tips for how to deal with your data.
- Refer back to an example from earlier in this presentation – we built interview questions on the data we already had. Touchpoints did not stand alone.
- Example: how do they define “bad results”?
- Example
- Forgive me if I’m stating the obvious, but … don’t let the data overwhelm you. Don’t assume you’ll always have it. Store it. Archive it. SAVE EVERYTHING. We saved everything. We put the tweets in spreadsheets. We even savedassignments they wrote. You’ll find new stuff to do with it.
- Good to keep in mind – in our labs, users are neutral.twitter/blogs/etc are users at their most frustrated, letting them down at finals time. Good to keep perspective of what a bad experience means.
- The Wikipedia stereotype
- Wikipedia is popular because it’s an aggregation resource; it did the research for themAnd also, Wikipedia is popular because it’s a BROAD resource.
- STORY: participant who saw topic A, switched to topic B. Frustration when she can’t find enough stuff for topic B, switches to topic C…As inexperienced researchers, they don’t know how to effectively survey a research topic and craft searches.
- As mentioned, WE had some solid understanding of novice researchers preferring broad resources to specific ones. And some examples of why. Took this to the metadata librarians and brainstormed whether there were ways to help identify entry-level, broad, or novice resources. EXAMPLE: identify specific journals as being more general, identify metareviews, lots of ideas.
- Longitudinal studies are perfect for building personas: rich info on behavior, motivations, goals, pain points
- Study also helps us more broadly. We combined insights from the study with other studies we’ve done – interviews, usability tests,etc - Helped us build understanding of how to serve our users
- So first: was it worth it? YES. We are still recycling data
- NO REALLY. Interview prep is intense!Focus group activities take a long timeUsers are flaky – leave extra timeTechnology fails – have a backup
- Did a good job of initial screening, but we’re used to that first round of filtering. HOWEVER, still didn’t screen perfectly.DON’T want it to be like a beauty pageant where you find out throughout the thing that someone doesn’t fit.
- We asked them for lots of data, which is ok, cuz we paid them for it. But we should still have made it easier to give us everything.Future: compiled interface. Give participants a single place to monitor their progress, submit us their stuff, and check due dates
- They get busier near the end of the semester. We don’t….Need to shape study requirements to this better