Vovici Vision 2011: New Zen and the Art of Questionnaire Design
1. Zen and the Art of Questionnaire Design
Vovici Vision
May 18, 2011
2. Our Agenda: The Eightfold Path to Survey Enlightenment
1. Right Perspective
2. Right Intention
3. Right Order
4. Right Attention
5. Right Speech
6. Right Choice
7. Right Length
8. Right Action
2
3. More Questionnaire Design, Less Zen
“Now let me correct you on a couple of
things, OK?
“Aristotle was not Belgian.
“The central message of Buddhism is not
‘Every man for himself.’
“And the London Underground is not a
political movement.”
Not Vovici specific
For other aspects, see 7 Habits of Highly
Successful Surveys
3
4. The Eightfold Path to Survey Enlightenment
Right Perspective:
The Respondent as Human
4
5. Asking a Lot of the Respondent
Literally and Figuratively
1. Interpret the meaning of a
question
2. Recall all relevant facts related to
question
3. Internally summarize those facts
4. Report summary judgment
accurately
5
6. Asking a Lot of the Respondent
Literally and Figuratively
1. Interpret the meaning of a
question
2. Recall all relevant facts related
to question
3. Internally summarize those facts
4. Report summary judgment
accurately
6
7. Respondent Coping Strategies
Different Types of Processing
1. Interpret question meaning
2. Recall relevant facts
3. Internally summarize facts
4. Report summary judgment
7
9. Weak Satisficing Strong Satisficing
Selecting the first choice that Endorsing the status quo instead
appears reasonable of change
Agreeing with assertions Failing to differentiate in ratings
(“acquiescence response bias”) Selecting “Don’t know” rather
than giving an opinion
Randomly choosing
Source: Krosnick, J. A. (1991). “Response
Strategies for Coping with the Cognitive
Demands of Attitude Measures in Surveys.”
Applied Cognitive Psychology, 5, 213-236.
9
10. Other Signs Respondents are Human
Cognitive Social Survey
Behaviors Behaviors Behaviors
Satisficing Acquiescence bias Response styles
Memory biases Social desirability bias Response substitution
Economic behavior Halo error
Mode effects
Practice effects
Panel conditioning
10
12. Acquiescence Bias
Some respondents are simply agreeable, and indicate
agreement out of politeness
Other respondents expect that the researchers agree with
the listed items and defer to their judgment
Most respondents find agreeing takes less effort than
carefully weighing each optional level of disagreement and
agreement
Source: Saris, Krosnick and Shaeffer, 2005
12
13. Social Desirability Bias, by Topic
Health initiatives - Respondents exaggerate frequency of
exercise and compliance with medical regimens
Voting behavior - Respondents exaggerate intent to vote
Illegal behavior - Respondents underreport drug usage
and criminal history
Sexual behavior - Respondents deny, sanitize or
mainstream aspects of their sexual lives
Bigotry - Respondents downplay any prejudices
Salary - Poor respondents overstate income; rich
respondents understate it
13
14. Mode Effects
Face-to-face: a “guest” script
Social desirability bias highest:
Phone interviews: a “solicitor”
1. Telephone surveys
script
2. Face-to-face surveys
IVR interviews: a “voice mail”
3. IVR surveys
script
4. Mail surveys
Internet surveys: a “web
form” script 5. Web surveys
Mail surveys: a “form” script
14
16. Response Styles by Country: Informed by Culture
Source: Johnson, Kulesa, Cho, Shavitt, 2003; Vovici
16
17. Halo Error & Response Substitution
• What makes cheesesteaks and Tastykakes® taste even
better?
• Fans like the food 11% more when the Eagles win.
Victory tastes delicious.
• Halo error confuses the true strengths and weaknesses
of products and services.
• Makes benchmarking attributes across competing
brands and products unreliable (brands are well
documented as introducing halo effects).
• Leads to misinterpretation of satisfaction attributes.
• Response substitution is when respondents' answers to
questions might sometimes reflect attitudes that respondents
want to convey, but that the researcher has not asked about
17
18. What Respondents Like About Taking Surveys
Nothing 34%
Dislike surveys 24%
Chance to voice their opinion 13%
Earn an incentive 11%
Like to be helpful 10%
Interesting topics 9%
Answers make a difference 7%
0% 20% 40% 60% 80% 100%
Source: Vovici Survey Nation study, N = 100 RDD sampled U.S. adults
18
20. The Second Factor: Worthy Purpose For a Survey
Focus on a Goal
Be precise about what information you need to
gather and what you plan on doing with it
If your organization hasn’t done a survey in a
while, the tendency is for every department to
chime in with questions they want you to ask
A narrow goal will help you to relentlessly
simplify the survey
"In the case of archery, the hitter
and the hit are no longer two
opposing objects, but one reality."
- Zen in the Art of Archery
20
21. Is a Survey the Right Arrow to Hit the Target?
Sometimes the best survey is to not do a
survey at all
Talk to stakeholders who will use the data to
understand their wants and needs
Is someone elsewhere in the organization
doing a survey on this topic or researching
this issue?
Are customers (or employees or …) the only
source of this information?
"Two hands clap and there is
Do your CRM, web analytics or other systems a sound. What is the sound
hold data that would address this issue? of one hand?"
- Hakuin Ekaku
21
22. Examples of Goals
Good Goals Bad Goals
Determine how often callers into It’s been a while; we should do a
help desk check the customer- survey.
service web site before calling What are our customers thinking
Prioritize the feature ideas for a right now?
new product with prospects Maybe customers would like it if we
Find out how satisfied customers provided support after 6 pm.
are with tech support Let’s create a committee and see
Determine if employees feel that what they want to find out from a
organization is living up its core survey.
values
22
24. The Third Factor: Questions’ Proper Sequence
Screener Use the inverted
pyramid approach,
Open-Ended Questions drilling down
General Questions Ask harder questions
first, before respondents
Specific Questions
grow tired of the survey
Demographics
Follow-up
24
25. The Third Factor: Questions’ Proper Sequence
Screener Conventionally, screeners
route people out of survey
Open-Ended Questions depending on answers to
initial question
General Questions
Build panel profiles that
Specific Questions
including the screening
Demographics
information you need
You have their attention; skip
Follow-up
them out to another survey
Otherwise they get the
impression their feedback isn’t
valuable
25
26. The Third Factor: Questions’ Proper Sequence
Screener Capture their views in their
own words before biasing
Open-Ended Questions them with your later
questions
General Questions
o “What, if anything, do you
Specific Questions
like about…?”
Demographics o “What, if anything, do you
dislike…”
Follow-up
Answers provide color
commentary to later closed-
ended questions
Answers validate choice lists
26
27. Trade Off Increased Abandonment for More Verbatim Responses
Survey Concluding with Survey Beginning with
Open-Ends Open-Ends
Abandonment rate 1% 6%
Responses with at least 61% 90%
one verbatim answered
Average length of 13 words 13 words
verbatim answer
Sample size 70 79
Source: Vovici research on research, 9-24-09
27
28. The Fourth Habit: Order Questions Logically
Screener Use Skip Use
Patterns Branching
Open-Ended Questions
General Questions
Specific Questions
Demographics
Follow-up
28
29. Sequential Filtering Provides Better Conversational Flow
Sequential Filtering Grouped Filtering
Source: Lisa Carley-
Baxter, Andy Peytchev
and Michele C.
Black, 2010
29
30. The Third Factor: Questions’ Proper Sequence
Screener Use demographic and
firmographic questions to
Open-Ended Questions profile respondents and their
organizations
General Questions
Enables you to cross-tabulate
Specific Questions
& compare subgroups
Demographics Pre-populate from CRM
systems where possible
Follow-up
Place near end as tedious and
intrusive but can be
answered on “autopilot”
30
31. The Third Factor: Questions’ Proper Sequence
Screener Ask for any final comments
about any aspect of survey or
Open-Ended Questions topic
General Questions Ask for permission to follow-
up with them about their
Specific Questions
answers
Demographics Prompt if they have an issue
they want to be contacted
Follow-up
about
31
32. The Third Factor: Questions’ Proper Sequence
Screener Only at the very end ask
respondent satisfaction
Open-Ended Questions questions evaluating the
survey
General Questions
Use to drive continual
Specific Questions
improvement to your
Demographics
research process itself
Key measures:
Follow-up How interesting was the survey
How long was it
Open comments
32
34. Four Basic Question Types
Open-Ended Questions Closed-Ended Questions
Essay Question Fill in the Blank Choose One Choose Many
34
35. Open-Ended Questions vs. Closed-Ended Questions
beige blue Bluw green purpel red
fire
engine
Black blue red green purple red
sea
black blue Gray grey red green
sky
black blue Green orange red blue
blue
Blue r0x!! green orange red yellow
35
36. Best Practices
Open-Ended Questions Closed-Ended Questions
Great for hearing from the respondent in Make sure the list contains all common
their own words choices
Provides unbiased, unfiltered answers Better to have too many items rather than
Good for catching anything you missed at too few, but try not to clutter the list
end of a survey Provide the respondent with an “Other –
Limit use, though, because: please specify” choice
o Time consuming and taxing to Arrange the choices in logical order
answer If no logical order, then
o Difficult to analyze randomize the order
36
37. Yes/No Questions: Common Pitfalls
Force-fitting a question into a yes/no format by
overriding what “Yes” or “No” means.
Asking for a single Yes/No to multiple items.
Letting the user select both Yes and No.
Providing caveats to the Yes/No choices.
Asking questions that can’t be answered
Yes/No.
Listing a bunch of similar yes/no questions in a
matrix.
Asking questions that no one wants to say “no”
to (“acquiescence response bias”).
37
38. Choose-Many Questions
Use whenever more than one choice is
applicable
Always include a “none of the above” as an
exclusive choice, otherwise can’t tell if
respondent answered question
Avoid providing choices that can be
synonymous or subsets/supersets of one
another
Don’t use a list box to show the choices
If the choice list has no natural order,
randomize the order (anchoring “None of the
above” to bottom)
38
39. Matrix/Table/Grid Questions
Concise technique for combining questions with common topics
Can be 50% faster for respondent to complete but speed may lead to
mistakes
Source: SSI, “Grid Test Summary”, 2009
39
40. Reimagine Matrix Questions when You Can
"We know respondents don't like grids... Now, we're beginning to
learn that not only are grids frustrating for respondents - they actually
produce inferior data." - Jackie Lorch, SSI VP
Break each row of the matrix into a separate question or group
of questions on its own page
Rewrite each row of the matrix into separate questions, replacing
checkboxes with fully labeled scales
Refactor importance matrixes into choose-many questions
Refactor Yes/No Matrix questions into choose-many questions
(checkbox lists) instead
Be creative
40
41. Mental Effort of Rating vs. Ranking Questions
Ranking Questions are More Taxing for Respondents
70
60 Rating
Comparisons
50 Ranking
40
30
20
10
0
2 3 4 5 6 7 8 9 10 11 12
# Choices
41
42. When in Doubt, Use Rating Questions
Ranking Questions Rating Questions
Taxing for respondents, requiring Lead to less differentiation among
them to compare multiple items choices
against one another Ratings often fall into narrow upper
Difficulty increases disproportionately band
as choices are added Personal variations in rating styles
Take three times longer to answer Possible spurious positive correlations
than rating questions (Munson and due to individuals’ personal variations
McIntyre, 1979)
Limit range of statistical analysis
Alwin, D. F., & Krosnick, J. A. (1985). “The
measurement of values in surveys: A
comparison of ratings and rankings.”
42
43. Constant Sum & Allocation Questions
Limit the number of items
If you have too many items, break
them into categories and then ask
follow-on allocation questions
If appropriate, include an Other
category as a safety valve for
respondents
Use matrix questions instead if
respondents cannot easily recollect
and quantify their past behavior
Provide visual feedback to the
current working sum
43
45. The Fifth Factor: Right Speech
Asking Objective Questions
Respondents should not be able to determine where
you stand on any topic
o Use nonjudgmental wording
o Choose neutral terms
Don’t ask leading questions
o Not “What do you like about your service?” Nan-in served tea. He poured his visitor's
cup full, and then kept on pouring.
The professor watched the overflow
o But “What, if anything, do you like…?” until he no longer could restrain himself. "It
is overfull. No more will go in!"
Write from the respondent’s perspective not your "Like this cup," Nan-in said, "you are
full of your own opinions and speculations.
perspective How can I show you Zen unless you first
empty your cup?"
45
46. The Fifth Factor: Right Speech
Asking Objective Questions
Remove ambiguity: “What is your favorite drink?”
(drink = beverage or drink = alcoholic beverage)
Ask one item at a time
not: “How would you rate our price and service?”
not: “How easy to reach someone to help?”
Avoid industry jargon
Specify how you use general terms
Don’t make subtle distinctions
Have others proofread your questions for clarity
Pre-test survey with a segment of your audience
46
47. The “Shoulds” of Question Wording
Q. should be focused on a single issue or topic
Q. should be interpreted the same way by all respondents
Q. should use the respondent’s core vocabulary
Q. should be a grammatically simple sentence if possible
Q. should be brief
47
48. The “Should Nots” of Question Wording
Q. should not assume criteria that are not obvious
Q. should not be beyond the respondent’s ability or experience
Q. should not use a specific example to represent a general case
Q. should not request recall of specifics when only generalities will be remembered
Q. should not require the respondent to guess a generalization
Q. should not ask for details that cannot be related
Q. should not contain words that overstate the condition
Q. should not have ambiguous wording
Q. should not be “double-barreled”
Q. should not lead the respondent to a particular answer
Q. should not have “loaded” wording or phrasing.
48
49. Test, Test, Test
Self-Test Pre-test Pilot Test Publish
Self-Test
Question flow Answer validation
Question wording Required answers
Question types Skip patterns
Scale consistency Errors of omission
49
51. What’s Your Favorite Color?
Poll A – 6 Choices Poll H – 15 Choices
Green
Blue Purple
Pink
Green Orange
Red
Black White
Blue
Red Black
Maroon
Yellow Magenta
Brown
None of those! Gray
Lime Green
Sky Blue
Hot Pink
Whatever Hayley's favorite color is ;)
51
52. Provide Exhaustive Lists of Choices
Top 3 Favorite Colors As Determined by Questions
with Different Numbers of Choices
15
10
5
0
A B C D E F G H
Correct Ranking
52
54. Impact of Showing Logos
May increase brand awareness recall
May be more accurate for assessing print, web
and video campaigns
May be less accurate for assessing radio
campaigns
54
55. To Label or Not Label Each Point of a Scale
Many Variations Possible Best Practices
Respondents prefer fully labeled scales
Fully labeled scales have greater reliability
and validity
Numeric values alter the meaning of labels
and should be avoided
5-point unipolar and 7-point bipolar scales
have greatest reliability and validity
Where possible use standard scales rather
than write your own
Source: Krosnick, J. A., & Fabrigar, L. R.
(1997). “Designing rating scales for effective
measurement in surveys.”
55
56. Are Respondents Too Agreeable?
Likert Scale Best Practices
Completely disagree The traditional Likert scale is
Disagree obsolete
Somewhat disagree Over 100 studies have
demonstrated acquiescence bias
Neither agree or disagree
Use “construct-specific response
Somewhat agree
options” instead – common rating
Agree scales and custom scales
Completely agree - Saris, Krosnick & Shaeffer (2005)
56
57. Patterns to Use for Scales
“He's embiggened
that role with his
Unipolar Scale (0..100)
cromulent Bipolar Scale (-1..0..+1)
performance.” Completely* disembiggened
Not at all cromulent
Slightly cromulent Mostly disembiggened
Somewhat disembiggened
Moderately cromulent
Neither embiggened nor
Very cromulent disembiggened
Completely* cromulent Somewhat embiggened
Mostly embiggened
*or “Extremely” where appropriate Completely* embiggened
57
58. Common Survey Rating Scales
Acceptability Totally unacceptable, Unacceptable, Slightly unacceptable, Neutral, Slightly acceptable, Acceptable, Perfectly acceptable
Agreement Completely disagree, Disagree, Somewhat disagree, Neither agree or disagree, Somewhat agree, Agree, Completely agree
Amount of Use Never use, Almost never, Occasionally/Sometimes, Almost every time, Frequently use
Appropriateness Absolutely inappropriate, Inappropriate, Slightly inappropriate, Neutral, Slightly appropriate, Appropriate, Absolutely appropriate
Awareness Not at all aware, Slightly aware, Moderately aware, Very aware, Extremely aware
Beliefs Very untrue of what I believe, Untrue of what I believe, Somewhat untrue of what I believe, Neutral, Somewhat true of what I believe, True of what I believe,
Very true of what I believe
Concern Not at all concerned, Slightly concerned, Moderately concerned, Very concerned, Extremely concerned
Familiarity Not at all familiar, Slightly familiar, Moderately familiar, Very familiar, Extremely familiar
Frequency Never, Rarely, Sometimes, Often, Always
Importance Not at all important, Slightly important, Moderately important, Very important, Extremely important
Influence Not at all influential, Slightly influential, Moderately influential, Very influential, Extremely influential
Likelihood Not at all likely, Slightly likely, Moderately likely, Very likely, Completely likely
Priority Not a priority, Low priority, Medium priority, High priority, Essential
Probability Not at all probable, Slightly probable, Moderately probable, Very probable, Completely probable
Quality Very poor, Poor, Fair, Good, Excellent
Reflect Me Very untrue of me, Untrue of me, Somewhat untrue of me, Neutral, Somewhat true of me, True of me, Very true of me
Satisfaction (bipolar)
Completely dissatisfied, Mostly dissatisfied, Somewhat dissatisfied, Neither satisfied or dissatisfied, Somewhat satisfied, Mostly satisfied, Completely satisfied
Satisfaction (unipolar) Not at all satisfied, Slightly satisfied, Moderately satisfied, Very satisfied, Completely satisfied
58
59. Developing Customer Scales
Labels Reflect Equal Intervals Common Scales
100 Liking
Very Poor, Poor, Fair, Good, Excellent
80
Amount
60 None, Some, Half, Most, All
Frequency
40
Never, Rarely, Sometimes, Often, Always
20 Likelihood
Not at all likely, Slightly likely, Moderately likely,
0
Very likely, Completely likely
Source: Krosnick, J. A., & Fabrigar, L. R.
(1997). “Designing rating scales for effective
measurement in surveys.”
59
60. Use Dropdowns Only for Known Lists
Great for letting respondents pick from a long list of
known choices (e.g., states, provinces)
Even casual users will use the keyboard effectively
(e.g., click M three times to cycle from Maine to
Maryland to Massachusetts)
Make sure not to make any choice the default (e.g.,
having Alabama selected)
Make the default choice an instruction like “Click
here to choose”
Don’t use when the choices have to be read to be
understood, as in lists of industries or job titles
60
61. “Don’t Know” or “No Opinion” Choices
When “satisficing”, respondents will select a
no-opinion choice if presented if one,
sometimes even if they have an opinion.
When no such choice is presented, most
respondents will choose from the other
choices.
Omit a no-opinion choice when asking for
attitude. Provide a “Don’t Know” choice when
prompting to recall specifics.
Source: Krosnick, J. A., & Fabrigar, L. R.
(1997). “Designing rating scales for
effective measurement in surveys.”
61
62. Juxtaposing Next & Previous Buttons
1.20%
Rate of Previous Button Use
1.00%
0.80%
0.60%
0.40%
0.20%
0.00%
9.4 9.5 9.6 9.7 9.8 9.9 10 10.1 10.2 10.3
Completion Time (minutes)
Source: Couper, Baker, Mechling
62
63. Juxtaposing Next & Previous Buttons
1.20%
Rate of Previous Button Use
1.00%
0.80%
0.60%
0.40%
0.20%
0.00%
0 2 4 6 8 10 12
Completion Time (minutes)
Source: Couper, Baker, Mechling
63
65. Right Length: Not Too Short, Not Too Long
2-4 questions: Transactional Survey
5-10 questions: Event Evaluation
10-20 questions: Customer Satisfaction
20-30 questions: Planning
50-70 questions: Major Account Review
70-90 questions: Employee Satisfaction
65
67. The Sixth Habit: Shorten the Survey
100%
Abandonment Rate Decrease the number
90%
Of 180-Question of matrix/grid
80%
Survey questions
70%
60% Reduce the number of
50% open-ended questions
40% Put demographic
30% questions at the end
20%
Shorten the survey!
10%
0%
Intro
10
12
13
14
15
16
17
18
19
20
1
2
3
4
5
6
7
8
9
11
Average 6 questions per block
Source: “Dropouts on the Web”, Galesic, 2006
67
68. Causes of Survey Incompletes
Primary Reason Respondent Abandoned Survey
Subject matter 35%
Media downloads 20%
Survey length 20%
Grids 15%
Too many open-ends 5%
0% 20% 40% 60% 80% 100%
Source: Lightspeed Research
68
69. Interesting Questionnaires are Perceived as Shorter
Optimal 1
length 0.9
0.8
0.7
} perception gap
response rates”, 2002, Mirta Galešic
0.6
“Effects of questionnaire length on
Somewhat
0.5
too long Higher interest in questionnaire
0.4
Lower interest in questionnaire
0.3
0.2
Absolutely 0.1
Source:
too long 0
Fewer Average More
questions number of questions
questions
69
70. Tips for Shortening the Survey
Keep Your Focus – Remove questions that don’t
directly address the goal of the survey
Ask Only Most Important Questions – Common
research tactic to have three similar questions
on similar topic: use one
Don’t Ask Esoteric Questions – Cut questions
that make distinctions only apparent to those
within your organization
Don’t Set False Expectations – Remove
questions that raise issues that can’t be
addressed (for customers, free services; for
employees, extended vacation time)
70
71. Shorten the Survey from the Respondent’s Perspective
Skip Respondents Past Inapplicable Sections – Don’t
subject respondent to survey about products or
services they don’t have or can’t have
Import Answers – Use CRM data to pipe in answers to
“hidden questions”
Randomize Displayed Sections – For less important
sections, randomly display only one section to each
respondent
Break into Multiple Questionnaires – Maybe questions
around different target groups are so different that
they are best served with different questionnaires
Use Fewer Pages – Page submits add a burden, so the
fewer pages the better for the most part.
Keep the Questionnaire Interesting – Respondents
perceive interesting surveys as shorter!
71
73. Right Action
Sharing the Survey’s Karma
Honor the time and energy of your
respondents
Share the results
Change and grow to serve them
better
Use survey triggers to immediately
take action
73
74. CRM Survey &
System Panel Database
Aggregate
Responses
Poor Rating Notification Manager
Customer Survey with CRM
Appended w/Customer
Data Integration
& Product Line Data
or
Customer Recovery
74
75. Use Survey Alerts/Email Triggers to Improve Satisfaction
Don’t Just Measure Satisfaction—Intervene to Improve It!
Email appropriate department when a respondent
provides a negative rating of service
Reactive
o Help-desk ticket satisfaction
o General dissatisfaction
o Hotel-stay satisfaction
o Major-account satisfaction
Proactive
o Customer-service satisfaction
o Literature fulfillment
75
76. The Eightfold Path
1. Right Perspective: The Respondent as Human
2. Right Intention: Worthy Purpose for Survey
3. Right Order: Questions in the Right Order
4. Right Attention: How Each Question Listens
5. Right Speech: Asking Objective Questions
6. Right Choice: Prompting with Care
7. Right Length: Not Too Short, Not Too Long
8. Right Action: Fulfilling the Survey’s Karma
76
77. Questions & Answers
To request a demonstration contact
your sales executive or:
1-800-787-8755
sales@vovici.com
I welcome your questions:
Jeffrey Henning,
jhenning@vovici.com
Free e-book
available from
http://blog.vovici.com
77
Egocentric bias: recalling the past in a self-serving manner, e.g., remembering one's exam grades as being better than they were, or remembering a caught fish as bigger than it really wasLeveling and Sharpening >: memory distortions introduced by the loss of details in a recollection over time, often concurrent with sharpening or selective recollection of certain details that take on exaggerated significance in relation to the details or aspects of the experience lost through leveling. Both biases may be reinforced over time, and by repeated recollection or re-telling of a memory.Telescoping effect > >: the tendency to displace recent events backward in time and remote events forward in time, so that recent events appear more remote, and remote events, more recent.</li>