Speaker: Caroline Jarrett
To help us get the best out of this tricky research method, Caroline will describe the Survey Octopus, a friendly creature that helps her to tackle all the issues that may lie between 'What we want to ask, and who we want to ask', and a solid, reliable number that can be used to make decisions.
Along the way, we'll encounter the key concept in survey methodology, Total Survey Error, and the various types of error that can affect your survey.
5. People will only respond if they trust
you. After that, it's a balance between
the perceived reward from filling in the
survey compared to the perceived
effort that's required. Strangely
enough, if a reward seems 'too good to
be true' that can also reduce the
response.
Diagram inspired by Dillman, D.A. (2000)
“Internet, Mail and Mixed Mode Surveys: The Tailored Design Method” 5
Response
6. Caroline Jarrett @cjforms (CC) BY SA-4.0 6
I’m a forms specialist
6Image credit: Flickr, taxrebate.org.uk
7. Caroline Jarrett @cjforms (CC) BY SA-4.0 7
Why do people answer questions?
7Caroline Jarrett @cjforms (CC) BY SA-4.0
8. People ask me about surveys
“Please have a look at this survey”
“How many people do I need in my sample?”
“Tell me whether this is a good question”
“I prefer 5 points in a rating scale, but my boss likes 7.
Who’s is right?”
8
9. Caroline Jarrett @cjforms (CC) BY SA-4.0 9
“Please have a look at this survey?”
9
Kill survey!
Kill! Kill!
10. Caroline Jarrett @cjforms (CC) BY SA-4.0 10
“Please have a look at this survey?”
10
What number do
you need to make
your decision?
11. 11
The survey is a systematic method
for gathering information from
(a sample of) entities for the purpose of
constructing quantitative descriptors of
the attributes of the larger population of
which the entities are members.
Groves, Robert M.; Fowler, Floyd J.; Couper, Mick P.; Lepkowski, James M.; Singer, Eleanor &
Tourangeau, Roger (2004).Survey methodology. Hoboken, NJ: John Wiley & Sons.
12. Caroline Jarrett @cjforms (CC) BY SA-4.0 12
The aim of a survey is to get a number
that helps you to make a decision
12
14. Caroline Jarrett @cjforms (CC) BY SA-4.0 14
“Yes, to make <decision> we need <number>”
14
Great!
Let’s work
through a few
key topics
15. 15
Total Survey Error diagram as presented in
Groves, R. M., F. J. Fowler, M. P. Couper, J. M.
Lepkowski, E. Singer and R. Tourangeau (2009).
Survey methodology. Hoboken, N.J., Wiley.
17. People ask me about surveys
“Please have a look at this survey”
“How many people do I need in my sample?”
“Tell me whether this is a good question”
“I prefer 5 points in a rating scale, but my boss likes 7.
Who’s is right?”
17
18. Caroline Jarrett @cjforms (CC) BY SA-4.0 18
To work that out, let’s visit the Octopus
Caroline Jarrett
@cjforms
(CC) BY SA-4.0 18
19. Caroline Jarrett @cjforms (CC) BY SA-4.0 19
Start with how many will answer
19
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
Fieldwork:
Who answers?
20. Caroline Jarrett @cjforms (CC) BY SA-4.0 20
Whether they’ll answer depends on effort
20
Questions:
What are you asking about?
How many questions?
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
21. Caroline Jarrett @cjforms (CC) BY SA-4.0 21
And on the reward you’re offering
Goals and resources:
Why are you asking?
Is helping you a reward in itself?
Are you offering any other incentive?
21
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
22. Then there’s the ‘Justin Bieber North Korea’ problem
22http://www.bbc.com/news/10506482
23. Caroline Jarrett @cjforms (CC) BY SA-4.0 23
If we ask ‘anyone’, we’ll have extra work here
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
Response:
Whose answers
can we use?
24. Caroline Jarrett @cjforms (CC) BY SA-4.0 24
So it matters where we get our sample
24
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
Sample:
the list you
sample from
25. Caroline Jarrett @cjforms (CC) BY SA-4.0 2525
And now it’s easy to work out how many to ask
Sample:
the number of
people to ask
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
26. Caroline Jarrett @cjforms (CC) BY SA-4.0 26
We thought about a lot of topics to work that out
Goals
Sample
Questions
Fieldwork
26
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
Response
28. Caroline Jarrett @cjforms (CC) BY SA-4.0 28
What about this bit?
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
29. People ask me about surveys
“Please have a look at this survey”
“How many people do I need in my sample?”
“Tell me whether this is a good question”
“I prefer 5 points in a rating scale, but my boss likes 7.
Who’s is right?”
29
30. Caroline Jarrett @cjforms (CC) BY SA-4.0 30
A good question gets good answers
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
Response:
Is the question
easy to answer?
31. Caroline Jarrett @cjforms (CC) BY SA-4.0 31
Good answers help you to make decisions
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
Insight:
Is the answer
useful?
32. 32
In your last five days at work, what
percentage of your work time do you
estimate that you spent using publicly-
available online services (not including
email, instant messaging, and search) to
do your work using a work computer or
other device?
%
Caroline Jarrett @cjforms (CC) BY SA-4.0
33. Caroline Jarrett @cjforms (CC) BY SA-4.0 33
All the topics are connected
Goals
Sample
Questions
Fieldwork
33
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
Response
Insight
Response
35. Caroline Jarrett @cjforms (CC) BY SA-4.0 35
People will only respond if they trust
you. After that, it's a balance between
the perceived reward from filling in the
survey compared to the perceived
effort that's required. Strangely
enough, if a reward seems 'too good to
be true' that can also reduce the
response.
Diagram inspired by Dillman, D.A. (2000)
“Internet, Mail and Mixed Mode Surveys: The Tailored Design Method”
Response depends on effort, reward, and trust
35
36. Caroline Jarrett @cjforms (CC) BY SA-4.0 36
A good question works in three ways
36
Appropriate
Obvious Interesting
37. Caroline Jarrett @cjforms (CC) BY SA-4.0 3737
Why did you visit our website today?
Appropriate
Obvious Interesting
38. Caroline Jarrett @cjforms (CC) BY SA-4.0 38
Would you recommend us to a friend or family member?
In a shop,
buying a baby carriage
In a hospital,
having a miscarriage
Obvious Yes
Interesting Yes
Appropriate Yes Cruelly inappropriate
38
39. Tip
Test your questions by
interviewing in context
Caroline Jarrett @cjforms (CC) BY SA-4.0 39
43. Caroline Jarrett @cjforms (CC) BY SA-4.0 43
Big Honkin’ Survey = think hard about everything
Goals
Sample
Questions
Fieldwork
43
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
Response
Insight
Response
44. 2015 mindset: the Light Touch survey
• Choose ONE question
• Find ONE person
• Ask the question, face-to-face
• See if you can make ONE decision
• Improve, iterate, increase
44
47. From goals to insight - quickly
• Choose ONE question
• Find ONE person
• Ask the question, face-to-face
• See if you can make ONE decision
• Improve, iterate, increase
47
48. Time for new question
48
One way to
iterate,
improve,
increase
Caroline Jarrett @cjforms (CC) BY SA-4.0
49. People ask me about surveys
“Please have a look at this survey”
“How many people do I need in my sample?”
“Tell me whether this is a good question”
“I prefer 5 points in a rating scale, but my boss likes 7.
Who’s is right?”
49
50. Caroline Jarrett @cjforms (CC) BY SA-4.0 50
Likert had several different types
of question in his response formats
Likert, Rensis. (1932). A Technique for the Measurement of Attitudes.
Archives of Psychology, 140, 1–55. 50
51. You can find an academic paper to support
almost any number of points
Krosnick and Presser refer to over 80 papers
51
Krosnick, J. A. and S. Presser (2009). Question and Questionnaire Design.
Handbook of Survey Research (2nd Edition) J. D. Wright and P. V. Marsden, Elsevier.
http://comm.stanford.edu/faculty/krosnick/docs/2010/2010 Handbook of Survey Research.pdf
52. Caroline Jarrett @cjforms (CC) BY SA-4.0 52
Respondents focus on the actual question
Likert, Rensis. (1932). A Technique for the Measurement of Attitudes.
Archives of Psychology, 140, 1–55. 52
53. Tip
Don’t stress too much about
the number of points in your
rating scale
53Picture credit: Flickr - Bill Soderman (BillsoPHOTO)
54. Caroline Jarrett @cjforms (CC) BY SA-4.0 54
Well, OK, stress a little bit.
54
This scale is
downright peculiar.
Avoid.
56. Caroline Jarrett @cjforms (CC) BY SA-4.0 5656
The aim is to get the best number you can,
within the resources you have
57. Caroline Jarrett @cjforms (CC) BY SA-4.0 57
What you want to ask about
The resources you have
The questions you ask
The answers you get
The answers you use
The number
Who you want to ask
The list that you sample from
The sample you ask
The ones who answer
The ones whose answers
you can use
57
The aim is to get the best number you can,
within the resources you have
58. Caroline Jarrett @cjforms (CC) BY SA-4.0 58
The aim is to get the best number you can,
within the resources you have
What you want
to ask about
The resources
you have
The questions
you ask
The answers
you get
The answers
you use
Who you want
to ask
The list you use
to sample from
The ones you
ask
The ones who
answer
The ones whose
answers you can use
The number
59. Caroline Jarrett @cjforms (CC) BY SA-4.0 5959
Survey statistic
Post-survey
adjustments
Respondents
Sample
Sampling frame
Representation
Edited response
Response
Measurement
Construct
The aim is to get the best number you can,
within the resources you have
Resources
What you want
to ask about
The resources
you have
The questions
you ask
The answers
you get
The answers
you use
Who you want
to ask
The list you use
to sample from
The ones you
ask
The ones who
answer
The ones whose
answers you can use
60. Caroline Jarrett @cjforms (CC) BY SA-4.0 6060
Survey statistic
Post-survey adjustments
Respondents
Sample
Sampling frame
Representation
Edited response
Response
Measurement
Construct
By making good choices, you minimise the error
Resources
61. 61
Total Survey Error diagram as presented in
Groves, R. M., F. J. Fowler, M. P. Couper, J. M.
Lepkowski, E. Singer and R. Tourangeau (2009).
Survey methodology. Hoboken, N.J., Wiley.
62. What number do you need to make the decision?
Use your UX skills
(especially technical communication)
Go for Light Touch surveys
Use the Survey Octopus to make good choices
Caroline Jarrett @cjforms (CC) BY SA-4.0 62
This is a genuine invitation from local government, but the layout and images in the invitation make it look as if it's an approach from some sort of spammer or scammer.
People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response.
The survey sits between 'what you want to ask', 'who you want to ask' and 'the number'
The survey octopus has 8 tentacles. We'll visit each one in the next few slides. We’ll get our survey to the people who will answer in what the survey methodologists call ‘fieldwork’ – that might be a pop-up a website, a mail survey, or face-to-face interviews.
The survey octopus has 8 tentacles. We'll visit each one in the next few slides. We’ll get our survey to the people who will answer in what the survey methodologists call ‘fieldwork’ – that might be a pop-up a website, a mail survey, or face-to-face interviews.
The octopus again. This time we're looking at 'the questions we ask'.
The resources you have will help you to decide on the reward you’re offering
Prank leaves Justin Bieber facing tour of North Korea
By Daniel Emery Technology reporter, BBC News
5 July 2010
Image caption It is highly unlikely Bieber would be given permission to enter North Korea Canadian singer Justin Bieber's has become the target of a viral campaign to send him to North Korea.
A website polled users as to which country he should tour next, with no restrictions on the nations that could be voted on.
There are now almost half a million votes to send the singer to the secretive communist nation.
The contest, which ends at 0600 on 7 July, saw North Korea move from 24th to 1st place in less than two days.
Many of the votes are thought to originate from imageboard website 4chan, which has built a reputation for triggering online viral campaigns.
The octopus, with focus on 'The list you sample from'
The octopus again; we've looked at 6 of the 8 tentacles.
The octopus again; we've looked at 6 of the 8 tentacles.
People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response.
Obvious questions require no effort to answer. Interesting questions are rewarding to answer. Appropriate questions are ones that inspire trust in the respondent.
People come to the web with their own questions, so they’re likely to know why they are on your website. It’s a relatively interesting question to answer, and it’s appropriate to ask visitors why they’re visiting.
If you're buying a baby carriage, then the 'recommend to a friend' question would probably be obvious, interesting and appropriate. If you're in hospital having a miscarriage, it would be cruelly inappropriate.
The octopus again; we've looked at 6 of the 8 tentacles.
Excerpt from Death to ‘it depends’ by Leisa Reichelt (@leisa)
Death to ‘it depends’
Lately I find myself on a mission for mass simplification. Possibly over simplification, but I’m not sure it matters.
It’s one of the things I care most about at the moment – how can we simplify what we are asking people to do so that there is nothing else they can do but start doing it, instead of following their natural inclination to make a list, hire a consultant, write a white paper, do anything but doing the thing.
It requires that I stop saying (or even thinking) one of the things I have probably said most in my entire working life – ‘it depends’. That’s hard, but I think it’s the right thing to do.
It depends is paralysing.
Read more at:http://www.disambiguity.com/death-to-it-depends/
The octopus again; we've looked at 6 of the 8 tentacles.
A process starting with one person face to face, continues through 10 people by phone, gets to 100 people by email or pop-up.
It’s best to check that your question works with one person before you hassle 10 people with it. Then check it works with 10 people before you send it to 100. Once you’ve tried it on 100 people, you might be more interested in a new question than getting more answers on this question
This is a more conventional way of looking at the octopus tentacles
If we just look at the issues (no tentacles) we get this slide
This slide translates the issues into the technical terms used by survey methodologists
This slide translates the issues into the technical terms used by survey methodologists