SlideShare a Scribd company logo
1 of 77
Zen and the Art of Questionnaire Design
Vovici Vision
May 18, 2011
Our Agenda: The Eightfold Path to Survey Enlightenment

1. Right Perspective
2. Right Intention
3. Right Order
4. Right Attention
5. Right Speech
6. Right Choice
7. Right Length
8. Right Action




                                                         2
More Questionnaire Design, Less Zen

 “Now let me correct you on a couple of
   things, OK?
    “Aristotle was not Belgian.
    “The central message of Buddhism is not
    ‘Every man for himself.’
    “And the London Underground is not a
    political movement.”
 Not Vovici specific
 For other aspects, see 7 Habits of Highly
    Successful Surveys




                                              3
The Eightfold Path to Survey Enlightenment




                                Right Perspective:
                                The Respondent as Human




                                                          4
Asking a Lot of the Respondent
Literally and Figuratively


1. Interpret the meaning of a
   question
2. Recall all relevant facts related to
   question
3. Internally summarize those facts
4. Report summary judgment
   accurately



                                          5
Asking a Lot of the Respondent
Literally and Figuratively


1. Interpret the meaning of a
   question
2. Recall all relevant facts related
   to question
3. Internally summarize those facts
4. Report summary judgment
   accurately



                                       6
Respondent Coping Strategies
Different Types of Processing


 1. Interpret question meaning


        2. Recall relevant facts


  3. Internally summarize facts


 4. Report summary judgment


                                   7
Respondent Behavior Degrades as Survey Lengthens


                            Optimizing



                            Weak Satisficing



                            Strong Satisficing



                            Cheating




                                                   8
Weak Satisficing                            Strong Satisficing
  Selecting the first choice that             Endorsing the status quo instead
  appears reasonable                          of change
  Agreeing with assertions                    Failing to differentiate in ratings
  (“acquiescence response bias”)              Selecting “Don’t know” rather
                                              than giving an opinion
                                              Randomly choosing
Source: Krosnick, J. A. (1991). “Response
Strategies for Coping with the Cognitive
Demands of Attitude Measures in Surveys.”
Applied Cognitive Psychology, 5, 213-236.




                                                                                    9
Other Signs Respondents are Human




 Cognitive       Social                     Survey
 Behaviors       Behaviors                  Behaviors
 Satisficing     Acquiescence bias          Response styles
 Memory biases   Social desirability bias   Response substitution
                 Economic behavior          Halo error
                                            Mode effects
                                            Practice effects
                                            Panel conditioning



                                                                    10
Memory Biases
 Choice-supportive bias                Generation effect                   Osborn effect      Source Confusion
           Change bias            Illusion-of-truth effect         Part-list cueing effect       Spacing effect
    Childhood amnesia                          Lag effect                Peak-end effect     Stereotypical bias
       Consistency bias        Leveling and Sharpening                        Persistence          Suffix effect
         Context effect       Levels-of-processing effect       Picture superiority effect        Suggestibility
      Cross-race effect                List-length effect                Positivity effect   Telescoping effect
         Cryptomnesia             Misinformation effect                    Primacy effect         Testing effect
        Egocentric bias                   Misattribution      Processing difficulty effect    Tip of the tongue
      Fading affect bias                 Modality effect            Reminiscence bump           Verbatim effect
         Hindsight bias    Mood congruent memory bias                 Rosy retrospection     Von Restorff effect
          Humor effect                Next-in-line effect           Self-relevance effect       Zeigarnik effect


                                                       Source: http://en.wikipedia.org/wiki/Memory_bias




                                                                                                               11
Acquiescence Bias

   Some respondents are simply agreeable, and indicate
   agreement out of politeness
   Other respondents expect that the researchers agree with
   the listed items and defer to their judgment
   Most respondents find agreeing takes less effort than
   carefully weighing each optional level of disagreement and
   agreement
                         Source: Saris, Krosnick and Shaeffer, 2005




                                                                      12
Social Desirability Bias, by Topic

   Health initiatives - Respondents exaggerate frequency of
   exercise and compliance with medical regimens
   Voting behavior - Respondents exaggerate intent to vote
   Illegal behavior - Respondents underreport drug usage
   and criminal history
   Sexual behavior - Respondents deny, sanitize or
   mainstream aspects of their sexual lives
   Bigotry - Respondents downplay any prejudices
   Salary - Poor respondents overstate income; rich
   respondents understate it




                                                              13
Mode Effects
  Face-to-face: a “guest” script
                                    Social desirability bias highest:
  Phone interviews: a “solicitor”
                                    1. Telephone surveys
  script
                                    2. Face-to-face surveys
  IVR interviews: a “voice mail”
                                    3. IVR surveys
  script
                                    4. Mail surveys
  Internet surveys: a “web
  form” script                      5. Web surveys
  Mail surveys: a “form” script




                                                                        14
Answers Patterns for Common Response Styles
                                                    Completely              Somewhat    Neither agree   Somewhat           Completely
                                                                 Disagree                                          Agree
                             Response Style          disagree                disagree   nor disagree      agree              agree

                             Optimal Responding
                             Extreme Response
                             Style (ERS)
 Truncated Scales




                             Response Range (RR)

                             Mild Response Style
                             Midpoint Response
                             (MPR)
                             Acquiescence
 Social/Anti-Social Styles




                             Response Style (ARS)
                             Disacquiescence
                             Response Style
                             (DARS)
                             Socially Desirable
                             Responding (SDR)
                             Noncontingent
                             Responding (NCR)


                                                                                                                                        15
Response Styles by Country: Informed by Culture




                          Source: Johnson, Kulesa, Cho, Shavitt, 2003; Vovici

                                                                                16
Halo Error & Response Substitution

              •   What makes cheesesteaks and Tastykakes® taste even
                  better?
                    •   Fans like the food 11% more when the Eagles win.
                        Victory tastes delicious.
                    •   Halo error confuses the true strengths and weaknesses
                        of products and services.
                    •   Makes benchmarking attributes across competing
                        brands and products unreliable (brands are well
                        documented as introducing halo effects).
                    •   Leads to misinterpretation of satisfaction attributes.
              •   Response substitution is when respondents' answers to
                  questions might sometimes reflect attitudes that respondents
                  want to convey, but that the researcher has not asked about

                                                                                 17
What Respondents Like About Taking Surveys

                     Nothing                              34%

              Dislike surveys                      24%

Chance to voice their opinion               13%

           Earn an incentive               11%

            Like to be helpful            10%

            Interesting topics           9%

 Answers make a difference              7%

                                 0%          20%          40%           60%          80%          100%
                                      Source: Vovici Survey Nation study, N = 100 RDD sampled U.S. adults

                                                                                                            18
The Eightfold Path




                     Right Intention:
                     Worthy Purpose for Survey




                                                 19
The Second Factor: Worthy Purpose For a Survey
Focus on a Goal
   Be precise about what information you need to
   gather and what you plan on doing with it
   If your organization hasn’t done a survey in a
   while, the tendency is for every department to
   chime in with questions they want you to ask
   A narrow goal will help you to relentlessly
   simplify the survey


                                                     "In the case of archery, the hitter
                                                          and the hit are no longer two
                                                    opposing objects, but one reality."
                                                            - Zen in the Art of Archery



                                                                                       20
Is a Survey the Right Arrow to Hit the Target?

   Sometimes the best survey is to not do a
   survey at all
   Talk to stakeholders who will use the data to
   understand their wants and needs
   Is someone elsewhere in the organization
   doing a survey on this topic or researching
   this issue?
   Are customers (or employees or …) the only
   source of this information?
                                                   "Two hands clap and there is
   Do your CRM, web analytics or other systems       a sound. What is the sound
   hold data that would address this issue?                       of one hand?"
                                                                - Hakuin Ekaku



                                                                              21
Examples of Goals

Good Goals                             Bad Goals
  Determine how often callers into       It’s been a while; we should do a
  help desk check the customer-          survey.
  service web site before calling        What are our customers thinking
  Prioritize the feature ideas for a     right now?
  new product with prospects             Maybe customers would like it if we
  Find out how satisfied customers       provided support after 6 pm.
  are with tech support                  Let’s create a committee and see
  Determine if employees feel that       what they want to find out from a
  organization is living up its core     survey.
  values



                                                                             22
The Eightfold Path




                     Right Order:
                     Questions’ Proper Sequence




                                             23
The Third Factor: Questions’ Proper Sequence


            Screener                 Use the inverted
                                     pyramid approach,
       Open-Ended Questions          drilling down
         General Questions           Ask harder questions
                                     first, before respondents
          Specific Questions
                                     grow tired of the survey
             Demographics


              Follow-up




                                                                 24
The Third Factor: Questions’ Proper Sequence


             Screener                Conventionally, screeners
                                     route people out of survey
        Open-Ended Questions         depending on answers to
                                     initial question
          General Questions
                                     Build panel profiles that
           Specific Questions
                                     including the screening
              Demographics
                                     information you need
                                     You have their attention; skip
               Follow-up
                                     them out to another survey
                                     Otherwise they get the
                                     impression their feedback isn’t
                                     valuable

                                                                      25
The Third Factor: Questions’ Proper Sequence


             Screener                Capture their views in their
                                     own words before biasing
       Open-Ended Questions          them with your later
                                     questions
         General Questions
                                    o “What, if anything, do you
           Specific Questions
                                      like about…?”
             Demographics           o “What, if anything, do you
                                      dislike…”
               Follow-up
                                     Answers provide color
                                     commentary to later closed-
                                     ended questions
                                     Answers validate choice lists

                                                                     26
Trade Off Increased Abandonment for More Verbatim Responses

                          Survey Concluding with      Survey Beginning with
                          Open-Ends                   Open-Ends
Abandonment rate          1%                          6%


Responses with at least   61%                         90%
one verbatim answered

Average length of         13 words                    13 words
verbatim answer
Sample size               70                          79


                                     Source: Vovici research on research, 9-24-09




                                                                                    27
The Fourth Habit: Order Questions Logically


              Screener             Use Skip      Use
                                   Patterns   Branching
        Open-Ended Questions

         General Questions

           Specific Questions

              Demographics


                Follow-up




                                                          28
Sequential Filtering Provides Better Conversational Flow
         Sequential Filtering    Grouped Filtering




                                                     Source: Lisa Carley-
                                                     Baxter, Andy Peytchev
                                                     and Michele C.
                                                     Black, 2010




                                                                             29
The Third Factor: Questions’ Proper Sequence


             Screener                  Use demographic and
                                       firmographic questions to
        Open-Ended Questions           profile respondents and their
                                       organizations
          General Questions
                                       Enables you to cross-tabulate
           Specific Questions
                                       & compare subgroups
             Demographics              Pre-populate from CRM
                                       systems where possible
               Follow-up
                                       Place near end as tedious and
                                       intrusive but can be
                                       answered on “autopilot”



                                                                 30
The Third Factor: Questions’ Proper Sequence


             Screener                  Ask for any final comments
                                       about any aspect of survey or
        Open-Ended Questions           topic
          General Questions            Ask for permission to follow-
                                       up with them about their
           Specific Questions
                                       answers
              Demographics             Prompt if they have an issue
                                       they want to be contacted
               Follow-up
                                       about




                                                                  31
The Third Factor: Questions’ Proper Sequence


             Screener                  Only at the very end ask
                                       respondent satisfaction
        Open-Ended Questions           questions evaluating the
                                       survey
          General Questions
                                       Use to drive continual
           Specific Questions
                                       improvement to your
              Demographics
                                       research process itself
                                       Key measures:
               Follow-up               How interesting was the survey
                                       How long was it
                                       Open comments




                                                                        32
The Eightfold Path




                     Right Attention:
                     How Each Question Listens




                                             33
Four Basic Question Types


     Open-Ended Questions                Closed-Ended Questions




  Essay Question   Fill in the Blank   Choose One    Choose Many




                                                                   34
Open-Ended Questions vs. Closed-Ended Questions



 beige   blue    Bluw    green   purpel red
                 fire
                 engine
 Black   blue    red    green    purple red
                                        sea
 black   blue    Gray    grey    red    green
                                        sky
 black   blue    Green   orange red     blue
         blue
 Blue    r0x!!   green   orange red     yellow




                                                  35
Best Practices

Open-Ended Questions                         Closed-Ended Questions
  Great for hearing from the respondent in     Make sure the list contains all common
  their own words                              choices
  Provides unbiased, unfiltered answers        Better to have too many items rather than
  Good for catching anything you missed at     too few, but try not to clutter the list
  end of a survey                              Provide the respondent with an “Other –
  Limit use, though, because:                  please specify” choice

         o   Time consuming and taxing to      Arrange the choices in logical order
             answer                            If no logical order, then
         o   Difficult to analyze              randomize the order




                                                                                           36
Yes/No Questions: Common Pitfalls


  Force-fitting a question into a yes/no format by
  overriding what “Yes” or “No” means.
  Asking for a single Yes/No to multiple items.
  Letting the user select both Yes and No.
  Providing caveats to the Yes/No choices.
  Asking questions that can’t be answered
  Yes/No.
  Listing a bunch of similar yes/no questions in a
  matrix.
  Asking questions that no one wants to say “no”
  to (“acquiescence response bias”).


                                                     37
Choose-Many Questions

  Use whenever more than one choice is
  applicable
  Always include a “none of the above” as an
  exclusive choice, otherwise can’t tell if
  respondent answered question
  Avoid providing choices that can be
  synonymous or subsets/supersets of one
  another
  Don’t use a list box to show the choices
  If the choice list has no natural order,
  randomize the order (anchoring “None of the
  above” to bottom)


                                                38
Matrix/Table/Grid Questions




     Concise technique for combining questions with common topics
     Can be 50% faster for respondent to complete but speed may lead to
     mistakes
     Source: SSI, “Grid Test Summary”, 2009



                                                                          39
Reimagine Matrix Questions when You Can

"We know respondents don't like grids... Now, we're beginning to
learn that not only are grids frustrating for respondents - they actually
produce inferior data." - Jackie Lorch, SSI VP
     Break each row of the matrix into a separate question or group
     of questions on its own page
     Rewrite each row of the matrix into separate questions, replacing
     checkboxes with fully labeled scales
     Refactor importance matrixes into choose-many questions
     Refactor Yes/No Matrix questions into choose-many questions
     (checkbox lists) instead
     Be creative


                                                                            40
Mental Effort of Rating vs. Ranking Questions
Ranking Questions are More Taxing for Respondents

                    70
                    60       Rating
      Comparisons




                    50       Ranking
                    40
                    30
                    20
                    10
                     0
                         2   3   4     5   6   7   8   9   10   11   12
                                           # Choices




                                                                          41
When in Doubt, Use Rating Questions

Ranking Questions                           Rating Questions
  Taxing for respondents, requiring           Lead to less differentiation among
  them to compare multiple items              choices
  against one another                         Ratings often fall into narrow upper
  Difficulty increases disproportionately     band
  as choices are added                        Personal variations in rating styles
  Take three times longer to answer           Possible spurious positive correlations
  than rating questions (Munson and           due to individuals’ personal variations
  McIntyre, 1979)
  Limit range of statistical analysis
                                                Alwin, D. F., & Krosnick, J. A. (1985). “The
                                                measurement of values in surveys: A
                                                comparison of ratings and rankings.”



                                                                                               42
Constant Sum & Allocation Questions

                                  Limit the number of items
                                  If you have too many items, break
                                  them into categories and then ask
                                  follow-on allocation questions
                                  If appropriate, include an Other
                                  category as a safety valve for
                                  respondents
                                  Use matrix questions instead if
                                  respondents cannot easily recollect
                                  and quantify their past behavior
                                  Provide visual feedback to the
                                  current working sum




                                                                        43
The Eightfold Path




                     Right Speech:
                     Asking Objective Questions




                                              44
The Fifth Factor: Right Speech
Asking Objective Questions

  Respondents should not be able to determine where
  you stand on any topic
         o   Use nonjudgmental wording
         o   Choose neutral terms
   Don’t ask leading questions
         o   Not “What do you like about your service?”   Nan-in served tea. He poured his visitor's
                                                          cup full, and then kept on pouring.
                                                              The professor watched the overflow
         o   But “What, if anything, do you like…?”       until he no longer could restrain himself. "It
                                                          is overfull. No more will go in!"
   Write from the respondent’s perspective not your           "Like this cup," Nan-in said, "you are
                                                          full of your own opinions and speculations.
   perspective                                            How can I show you Zen unless you first
                                                          empty your cup?"




                                                                                                   45
The Fifth Factor: Right Speech
Asking Objective Questions

  Remove ambiguity: “What is your favorite drink?”
  (drink = beverage or drink = alcoholic beverage)
  Ask one item at a time
  not: “How would you rate our price and service?”
  not: “How easy to reach someone to help?”
  Avoid industry jargon
  Specify how you use general terms
  Don’t make subtle distinctions
  Have others proofread your questions for clarity
  Pre-test survey with a segment of your audience



                                                      46
The “Shoulds” of Question Wording



  Q. should be focused on a single issue or topic
  Q. should be interpreted the same way by all respondents
  Q. should use the respondent’s core vocabulary
  Q. should be a grammatically simple sentence if possible
  Q. should be brief




                                                             47
The “Should Nots” of Question Wording

   Q. should not assume criteria that are not obvious
   Q. should not be beyond the respondent’s ability or experience
   Q. should not use a specific example to represent a general case
   Q. should not request recall of specifics when only generalities will be remembered
   Q. should not require the respondent to guess a generalization
   Q. should not ask for details that cannot be related
   Q. should not contain words that overstate the condition
   Q. should not have ambiguous wording
   Q. should not be “double-barreled”
   Q. should not lead the respondent to a particular answer
   Q. should not have “loaded” wording or phrasing.


                                                                                         48
Test, Test, Test




  Self-Test             Pre-test              Pilot Test   Publish

                    Self-Test
 Question flow           Answer validation
 Question wording        Required answers
 Question types          Skip patterns
 Scale consistency       Errors of omission




                                                                     49
The Eightfold Path




                     Right Choice:
                     Prompting with Care




                                           50
What’s Your Favorite Color?

 Poll A – 6 Choices           Poll H – 15 Choices
                              Green
   Blue                       Purple

                              Pink
   Green                      Orange

                              Red
   Black                      White

                              Blue
   Red                        Black

                              Maroon
   Yellow                     Magenta

                              Brown
   None of those!             Gray

                              Lime Green

                              Sky Blue

                              Hot Pink

                              Whatever Hayley's favorite color is ;)



                                                                       51
Provide Exhaustive Lists of Choices

        Top 3 Favorite Colors As Determined by Questions
                with Different Numbers of Choices
 15


 10


  5


  0
       A      B      C       D      E       F      G        H
                                                  Correct Ranking


                                                                    52
Source:
Ocucom



          53
Impact of Showing Logos


  May increase brand awareness recall
  May be more accurate for assessing print, web
  and video campaigns
  May be less accurate for assessing radio
  campaigns




                                                  54
To Label or Not Label Each Point of a Scale

Many Variations Possible       Best Practices
                                 Respondents prefer fully labeled scales
                                 Fully labeled scales have greater reliability
                                 and validity
                                 Numeric values alter the meaning of labels
                                 and should be avoided
                                 5-point unipolar and 7-point bipolar scales
                                 have greatest reliability and validity
                                 Where possible use standard scales rather
                                 than write your own


                                         Source: Krosnick, J. A., & Fabrigar, L. R.
                                         (1997). “Designing rating scales for effective
                                         measurement in surveys.”




                                                                                          55
Are Respondents Too Agreeable?


Likert Scale                  Best Practices
  Completely disagree            The traditional Likert scale is
  Disagree                       obsolete

  Somewhat disagree              Over 100 studies have
                                 demonstrated acquiescence bias
  Neither agree or disagree
                                 Use “construct-specific response
  Somewhat agree
                                 options” instead – common rating
  Agree                          scales and custom scales
  Completely agree            - Saris, Krosnick & Shaeffer (2005)




                                                                    56
Patterns to Use for Scales
                “He's embiggened
                 that role with his
Unipolar Scale (0..100)
                    cromulent       Bipolar Scale (-1..0..+1)
                  performance.”       Completely* disembiggened
  Not at all cromulent
  Slightly cromulent                 Mostly disembiggened
                                     Somewhat disembiggened
  Moderately cromulent
                                     Neither embiggened nor
  Very cromulent                     disembiggened
  Completely* cromulent              Somewhat embiggened
                                     Mostly embiggened
*or “Extremely” where appropriate    Completely* embiggened



                                                                  57
Common Survey Rating Scales
         Acceptability    Totally unacceptable, Unacceptable, Slightly unacceptable, Neutral, Slightly acceptable, Acceptable, Perfectly acceptable

           Agreement      Completely disagree, Disagree, Somewhat disagree, Neither agree or disagree, Somewhat agree, Agree, Completely agree

       Amount of Use      Never use, Almost never, Occasionally/Sometimes, Almost every time, Frequently use

     Appropriateness      Absolutely inappropriate, Inappropriate, Slightly inappropriate, Neutral, Slightly appropriate, Appropriate, Absolutely appropriate

           Awareness      Not at all aware, Slightly aware, Moderately aware, Very aware, Extremely aware

               Beliefs    Very untrue of what I believe, Untrue of what I believe, Somewhat untrue of what I believe, Neutral, Somewhat true of what I believe, True of what I believe,
                          Very true of what I believe

              Concern     Not at all concerned, Slightly concerned, Moderately concerned, Very concerned, Extremely concerned

            Familiarity   Not at all familiar, Slightly familiar, Moderately familiar, Very familiar, Extremely familiar

            Frequency     Never, Rarely, Sometimes, Often, Always

           Importance     Not at all important, Slightly important, Moderately important, Very important, Extremely important

             Influence    Not at all influential, Slightly influential, Moderately influential, Very influential, Extremely influential

            Likelihood    Not at all likely, Slightly likely, Moderately likely, Very likely, Completely likely

               Priority   Not a priority, Low priority, Medium priority, High priority, Essential

           Probability    Not at all probable, Slightly probable, Moderately probable, Very probable, Completely probable

               Quality    Very poor, Poor, Fair, Good, Excellent

            Reflect Me    Very untrue of me, Untrue of me, Somewhat untrue of me, Neutral, Somewhat true of me, True of me, Very true of me

 Satisfaction (bipolar)
                          Completely dissatisfied, Mostly dissatisfied, Somewhat dissatisfied, Neither satisfied or dissatisfied, Somewhat satisfied, Mostly satisfied, Completely satisfied

Satisfaction (unipolar)   Not at all satisfied, Slightly satisfied, Moderately satisfied, Very satisfied, Completely satisfied




                                                                                                                                                                                          58
Developing Customer Scales

Labels Reflect Equal Intervals   Common Scales

100                              Liking
                                          Very Poor, Poor, Fair, Good, Excellent
 80
                                 Amount
 60                                       None, Some, Half, Most, All

                                 Frequency
 40
                                          Never, Rarely, Sometimes, Often, Always
 20                              Likelihood
                                          Not at all likely, Slightly likely, Moderately likely,
  0
                                          Very likely, Completely likely


                                                 Source: Krosnick, J. A., & Fabrigar, L. R.
                                                 (1997). “Designing rating scales for effective
                                                 measurement in surveys.”




                                                                                                   59
Use Dropdowns Only for Known Lists

  Great for letting respondents pick from a long list of
  known choices (e.g., states, provinces)
  Even casual users will use the keyboard effectively
  (e.g., click M three times to cycle from Maine to
  Maryland to Massachusetts)
  Make sure not to make any choice the default (e.g.,
  having Alabama selected)
  Make the default choice an instruction like “Click
  here to choose”
  Don’t use when the choices have to be read to be
  understood, as in lists of industries or job titles



                                                           60
“Don’t Know” or “No Opinion” Choices

  When “satisficing”, respondents will select a
  no-opinion choice if presented if one,
  sometimes even if they have an opinion.
  When no such choice is presented, most
  respondents will choose from the other
  choices.
  Omit a no-opinion choice when asking for
  attitude. Provide a “Don’t Know” choice when
  prompting to recall specifics.
                                                  Source: Krosnick, J. A., & Fabrigar, L. R.
                                                  (1997). “Designing rating scales for
                                                  effective measurement in surveys.”




                                                                                           61
Juxtaposing Next & Previous Buttons
                              1.20%
Rate of Previous Button Use




                              1.00%

                              0.80%

                              0.60%

                              0.40%

                              0.20%

                              0.00%
                                      9.4   9.5   9.6   9.7    9.8    9.9     10      10.1     10.2     10.3
                                                        Completion Time (minutes)
                                                                             Source: Couper, Baker, Mechling

                                                                                                               62
Juxtaposing Next & Previous Buttons
                              1.20%
Rate of Previous Button Use




                              1.00%

                              0.80%

                              0.60%

                              0.40%

                              0.20%

                              0.00%
                                      0   2   4          6           8            10            12
                                              Completion Time (minutes)
                                                                   Source: Couper, Baker, Mechling

                                                                                                     63
The Eightfold Path




                     Right Length:
                     Not Too Short, Not Too Long




                                              64
Right Length: Not Too Short, Not Too Long

                       2-4 questions: Transactional Survey

                       5-10 questions: Event Evaluation

                       10-20 questions: Customer Satisfaction

                       20-30 questions: Planning

                       50-70 questions: Major Account Review

                       70-90 questions: Employee Satisfaction


                                                                65
66
The Sixth Habit: Shorten the Survey

 100%
              Abandonment Rate                               Decrease the number
  90%
               Of 180-Question                               of matrix/grid
  80%
                   Survey                                    questions
  70%
  60%                                                        Reduce the number of
  50%                                                        open-ended questions
  40%                                                        Put demographic
  30%                                                        questions at the end
  20%
                                                             Shorten the survey!
  10%
   0%
        Intro




          10

          12
          13
          14
          15
          16
          17
          18
          19
          20
            1
            2
            3
            4
            5
            6
            7
            8
            9

          11




                            Average 6 questions per block
              Source: “Dropouts on the Web”, Galesic, 2006



                                                                                    67
Causes of Survey Incompletes

              Primary Reason Respondent Abandoned Survey

     Subject matter                     35%


  Media downloads                 20%


      Survey length               20%


              Grids             15%


Too many open-ends         5%


                      0%        20%     40%   60%         80%            100%
                                                    Source: Lightspeed Research


                                                                                  68
Interesting Questionnaires are Perceived as Shorter
    Optimal      1
     length    0.9
               0.8
               0.7
                                                      } perception gap




                                                               response rates”, 2002, Mirta Galešic
               0.6




                                                               “Effects of questionnaire length on
  Somewhat
               0.5
    too long               Higher interest in questionnaire
               0.4
                           Lower interest in questionnaire
               0.3
               0.2
  Absolutely   0.1




                                                               Source:
    too long     0
                      Fewer         Average          More
                     questions     number of       questions
                                   questions


                                                                                                      69
Tips for Shortening the Survey

   Keep Your Focus – Remove questions that don’t
   directly address the goal of the survey
   Ask Only Most Important Questions – Common
   research tactic to have three similar questions
   on similar topic: use one
   Don’t Ask Esoteric Questions – Cut questions
   that make distinctions only apparent to those
   within your organization
   Don’t Set False Expectations – Remove
   questions that raise issues that can’t be
   addressed (for customers, free services; for
   employees, extended vacation time)


                                                     70
Shorten the Survey from the Respondent’s Perspective
  Skip Respondents Past Inapplicable Sections – Don’t
  subject respondent to survey about products or
  services they don’t have or can’t have
  Import Answers – Use CRM data to pipe in answers to
  “hidden questions”
  Randomize Displayed Sections – For less important
  sections, randomly display only one section to each
  respondent
  Break into Multiple Questionnaires – Maybe questions
  around different target groups are so different that
  they are best served with different questionnaires
  Use Fewer Pages – Page submits add a burden, so the
  fewer pages the better for the most part.
  Keep the Questionnaire Interesting – Respondents
  perceive interesting surveys as shorter!



                                                         71
The Eightfold Path




                     Right Action:
                     Sharing the Survey’s Karma




                                              72
Right Action
Sharing the Survey’s Karma


  Honor the time and energy of your
  respondents
  Share the results
  Change and grow to serve them
  better
  Use survey triggers to immediately
  take action




                                       73
CRM                          Survey &
System                    Panel Database



                                           Aggregate
                                           Responses




                                     Poor Rating Notification   Manager
Customer   Survey with CRM
                                     Appended w/Customer
           Data Integration
                                      & Product Line Data

                              or


                Customer Recovery




                                                                          74
Use Survey Alerts/Email Triggers to Improve Satisfaction
Don’t Just Measure Satisfaction—Intervene to Improve It!
   Email appropriate department when a respondent
   provides a negative rating of service
   Reactive
         o    Help-desk ticket satisfaction
         o    General dissatisfaction
         o    Hotel-stay satisfaction
         o    Major-account satisfaction
   Proactive
         o    Customer-service satisfaction
         o    Literature fulfillment


                                                           75
The Eightfold Path

1. Right Perspective: The Respondent as Human
2. Right Intention: Worthy Purpose for Survey
3. Right Order: Questions in the Right Order
4. Right Attention: How Each Question Listens
5. Right Speech: Asking Objective Questions
6. Right Choice: Prompting with Care
7. Right Length: Not Too Short, Not Too Long
8. Right Action: Fulfilling the Survey’s Karma




                                                 76
Questions & Answers
     To request a demonstration contact
     your sales executive or:
             1-800-787-8755
             sales@vovici.com

     I welcome your questions:
             Jeffrey Henning,
             jhenning@vovici.com




                                          Free e-book
                                          available from
                                          http://blog.vovici.com




                                                                   77

More Related Content

Similar to Vovici Vision 2011: New Zen and the Art of Questionnaire Design

Navigating Social Skills Training: Designing a Musical Roadmap for Individual...
Navigating Social Skills Training: Designing a Musical Roadmap for Individual...Navigating Social Skills Training: Designing a Musical Roadmap for Individual...
Navigating Social Skills Training: Designing a Musical Roadmap for Individual...Mundana Music Therapy
 
Decoding word association 2 jung wat and its relevance
Decoding word association 2  jung wat and its relevanceDecoding word association 2  jung wat and its relevance
Decoding word association 2 jung wat and its relevanceCol Mukteshwar Prasad
 
1. Describe physical security threats to the United States as a re.docx
1. Describe physical security threats to the United States as a re.docx1. Describe physical security threats to the United States as a re.docx
1. Describe physical security threats to the United States as a re.docxjeremylockett77
 
Why study communication ppt @ bec doms
Why study communication ppt @ bec domsWhy study communication ppt @ bec doms
Why study communication ppt @ bec domsBabasab Patil
 
Why study communication ppt @ bec doms
Why study communication ppt @ bec domsWhy study communication ppt @ bec doms
Why study communication ppt @ bec domsBabasab Patil
 
Why study communication ppt @ bec doms
Why study communication ppt @ bec domsWhy study communication ppt @ bec doms
Why study communication ppt @ bec domsBabasab Patil
 
Hillary Rodham Clinton’S Political Advertisement ‘Invisibles’
Hillary Rodham Clinton’S Political Advertisement ‘Invisibles’Hillary Rodham Clinton’S Political Advertisement ‘Invisibles’
Hillary Rodham Clinton’S Political Advertisement ‘Invisibles’Sigit Ricahyono
 
Effectiveness-in-communication
 Effectiveness-in-communication Effectiveness-in-communication
Effectiveness-in-communicationravikantpvs
 
Speech act-andrew-d-cohen-1206263596754620-5
Speech act-andrew-d-cohen-1206263596754620-5Speech act-andrew-d-cohen-1206263596754620-5
Speech act-andrew-d-cohen-1206263596754620-5xiongying
 
Can We Talk Presentation
Can We Talk PresentationCan We Talk Presentation
Can We Talk Presentationphilleb
 
Working with difficult people
Working with difficult peopleWorking with difficult people
Working with difficult peopleSugata Chowdhuri
 
Customers services - SEM Internship
Customers services - SEM InternshipCustomers services - SEM Internship
Customers services - SEM InternshipRoû MahMoûd
 
The importance of behavioural psychology on digital strategy
The importance of behavioural psychology on digital strategyThe importance of behavioural psychology on digital strategy
The importance of behavioural psychology on digital strategyReading Room
 
Marital counseling and infidelity
Marital counseling and infidelityMarital counseling and infidelity
Marital counseling and infidelityBrooke Schauder
 

Similar to Vovici Vision 2011: New Zen and the Art of Questionnaire Design (20)

Navigating Social Skills Training: Designing a Musical Roadmap for Individual...
Navigating Social Skills Training: Designing a Musical Roadmap for Individual...Navigating Social Skills Training: Designing a Musical Roadmap for Individual...
Navigating Social Skills Training: Designing a Musical Roadmap for Individual...
 
Decoding word association 2 jung wat and its relevance
Decoding word association 2  jung wat and its relevanceDecoding word association 2  jung wat and its relevance
Decoding word association 2 jung wat and its relevance
 
1. Describe physical security threats to the United States as a re.docx
1. Describe physical security threats to the United States as a re.docx1. Describe physical security threats to the United States as a re.docx
1. Describe physical security threats to the United States as a re.docx
 
Why study communication ppt @ bec doms
Why study communication ppt @ bec domsWhy study communication ppt @ bec doms
Why study communication ppt @ bec doms
 
Why study communication ppt @ bec doms
Why study communication ppt @ bec domsWhy study communication ppt @ bec doms
Why study communication ppt @ bec doms
 
Why study communication ppt @ bec doms
Why study communication ppt @ bec domsWhy study communication ppt @ bec doms
Why study communication ppt @ bec doms
 
Hillary Rodham Clinton’S Political Advertisement ‘Invisibles’
Hillary Rodham Clinton’S Political Advertisement ‘Invisibles’Hillary Rodham Clinton’S Political Advertisement ‘Invisibles’
Hillary Rodham Clinton’S Political Advertisement ‘Invisibles’
 
Effectiveness-in-communication
 Effectiveness-in-communication Effectiveness-in-communication
Effectiveness-in-communication
 
Speech act-andrew-d-cohen-1206263596754620-5
Speech act-andrew-d-cohen-1206263596754620-5Speech act-andrew-d-cohen-1206263596754620-5
Speech act-andrew-d-cohen-1206263596754620-5
 
Can We Talk Presentation
Can We Talk PresentationCan We Talk Presentation
Can We Talk Presentation
 
Working with difficult people
Working with difficult peopleWorking with difficult people
Working with difficult people
 
LC Session Three
LC Session ThreeLC Session Three
LC Session Three
 
Perception
PerceptionPerception
Perception
 
Questionnaire design
Questionnaire designQuestionnaire design
Questionnaire design
 
Ch9
Ch9Ch9
Ch9
 
Presentation
PresentationPresentation
Presentation
 
Customers services - SEM Internship
Customers services - SEM InternshipCustomers services - SEM Internship
Customers services - SEM Internship
 
The importance of behavioural psychology on digital strategy
The importance of behavioural psychology on digital strategyThe importance of behavioural psychology on digital strategy
The importance of behavioural psychology on digital strategy
 
Fourth day
Fourth dayFourth day
Fourth day
 
Marital counseling and infidelity
Marital counseling and infidelityMarital counseling and infidelity
Marital counseling and infidelity
 

Recently uploaded

Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationSafe Software
 
My Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationMy Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationRidwan Fadjar
 
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Wonjun Hwang
 
Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Manik S Magar
 
The Future of Software Development - Devin AI Innovative Approach.pdf
The Future of Software Development - Devin AI Innovative Approach.pdfThe Future of Software Development - Devin AI Innovative Approach.pdf
The Future of Software Development - Devin AI Innovative Approach.pdfSeasiaInfotech2
 
Developer Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLDeveloper Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLScyllaDB
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024Lorenzo Miniero
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebUiPathCommunity
 
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Mark Simos
 
SAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptxSAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptxNavinnSomaal
 
Vertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering TipsVertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering TipsMiki Katsuragi
 
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticsKotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticscarlostorres15106
 
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machineInstall Stable Diffusion in windows machine
Install Stable Diffusion in windows machinePadma Pradeep
 
Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Enterprise Knowledge
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek SchlawackFwdays
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Commit University
 
Search Engine Optimization SEO PDF for 2024.pdf
Search Engine Optimization SEO PDF for 2024.pdfSearch Engine Optimization SEO PDF for 2024.pdf
Search Engine Optimization SEO PDF for 2024.pdfRankYa
 
Story boards and shot lists for my a level piece
Story boards and shot lists for my a level pieceStory boards and shot lists for my a level piece
Story boards and shot lists for my a level piececharlottematthew16
 

Recently uploaded (20)

Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
 
My Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationMy Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 Presentation
 
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
 
Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!
 
The Future of Software Development - Devin AI Innovative Approach.pdf
The Future of Software Development - Devin AI Innovative Approach.pdfThe Future of Software Development - Devin AI Innovative Approach.pdf
The Future of Software Development - Devin AI Innovative Approach.pdf
 
Developer Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLDeveloper Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQL
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio Web
 
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
 
SAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptxSAP Build Work Zone - Overview L2-L3.pptx
SAP Build Work Zone - Overview L2-L3.pptx
 
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptxE-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
 
Vertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering TipsVertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering Tips
 
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticsKotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
 
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machineInstall Stable Diffusion in windows machine
Install Stable Diffusion in windows machine
 
Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!
 
Search Engine Optimization SEO PDF for 2024.pdf
Search Engine Optimization SEO PDF for 2024.pdfSearch Engine Optimization SEO PDF for 2024.pdf
Search Engine Optimization SEO PDF for 2024.pdf
 
Story boards and shot lists for my a level piece
Story boards and shot lists for my a level pieceStory boards and shot lists for my a level piece
Story boards and shot lists for my a level piece
 

Vovici Vision 2011: New Zen and the Art of Questionnaire Design

  • 1. Zen and the Art of Questionnaire Design Vovici Vision May 18, 2011
  • 2. Our Agenda: The Eightfold Path to Survey Enlightenment 1. Right Perspective 2. Right Intention 3. Right Order 4. Right Attention 5. Right Speech 6. Right Choice 7. Right Length 8. Right Action 2
  • 3. More Questionnaire Design, Less Zen “Now let me correct you on a couple of things, OK? “Aristotle was not Belgian. “The central message of Buddhism is not ‘Every man for himself.’ “And the London Underground is not a political movement.” Not Vovici specific For other aspects, see 7 Habits of Highly Successful Surveys 3
  • 4. The Eightfold Path to Survey Enlightenment Right Perspective: The Respondent as Human 4
  • 5. Asking a Lot of the Respondent Literally and Figuratively 1. Interpret the meaning of a question 2. Recall all relevant facts related to question 3. Internally summarize those facts 4. Report summary judgment accurately 5
  • 6. Asking a Lot of the Respondent Literally and Figuratively 1. Interpret the meaning of a question 2. Recall all relevant facts related to question 3. Internally summarize those facts 4. Report summary judgment accurately 6
  • 7. Respondent Coping Strategies Different Types of Processing 1. Interpret question meaning 2. Recall relevant facts 3. Internally summarize facts 4. Report summary judgment 7
  • 8. Respondent Behavior Degrades as Survey Lengthens Optimizing Weak Satisficing Strong Satisficing Cheating 8
  • 9. Weak Satisficing Strong Satisficing Selecting the first choice that Endorsing the status quo instead appears reasonable of change Agreeing with assertions Failing to differentiate in ratings (“acquiescence response bias”) Selecting “Don’t know” rather than giving an opinion Randomly choosing Source: Krosnick, J. A. (1991). “Response Strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys.” Applied Cognitive Psychology, 5, 213-236. 9
  • 10. Other Signs Respondents are Human Cognitive Social Survey Behaviors Behaviors Behaviors Satisficing Acquiescence bias Response styles Memory biases Social desirability bias Response substitution Economic behavior Halo error Mode effects Practice effects Panel conditioning 10
  • 11. Memory Biases Choice-supportive bias Generation effect Osborn effect Source Confusion Change bias Illusion-of-truth effect Part-list cueing effect Spacing effect Childhood amnesia Lag effect Peak-end effect Stereotypical bias Consistency bias Leveling and Sharpening Persistence Suffix effect Context effect Levels-of-processing effect Picture superiority effect Suggestibility Cross-race effect List-length effect Positivity effect Telescoping effect Cryptomnesia Misinformation effect Primacy effect Testing effect Egocentric bias Misattribution Processing difficulty effect Tip of the tongue Fading affect bias Modality effect Reminiscence bump Verbatim effect Hindsight bias Mood congruent memory bias Rosy retrospection Von Restorff effect Humor effect Next-in-line effect Self-relevance effect Zeigarnik effect Source: http://en.wikipedia.org/wiki/Memory_bias 11
  • 12. Acquiescence Bias Some respondents are simply agreeable, and indicate agreement out of politeness Other respondents expect that the researchers agree with the listed items and defer to their judgment Most respondents find agreeing takes less effort than carefully weighing each optional level of disagreement and agreement Source: Saris, Krosnick and Shaeffer, 2005 12
  • 13. Social Desirability Bias, by Topic Health initiatives - Respondents exaggerate frequency of exercise and compliance with medical regimens Voting behavior - Respondents exaggerate intent to vote Illegal behavior - Respondents underreport drug usage and criminal history Sexual behavior - Respondents deny, sanitize or mainstream aspects of their sexual lives Bigotry - Respondents downplay any prejudices Salary - Poor respondents overstate income; rich respondents understate it 13
  • 14. Mode Effects Face-to-face: a “guest” script Social desirability bias highest: Phone interviews: a “solicitor” 1. Telephone surveys script 2. Face-to-face surveys IVR interviews: a “voice mail” 3. IVR surveys script 4. Mail surveys Internet surveys: a “web form” script 5. Web surveys Mail surveys: a “form” script 14
  • 15. Answers Patterns for Common Response Styles Completely Somewhat Neither agree Somewhat Completely Disagree Agree Response Style disagree disagree nor disagree agree agree Optimal Responding Extreme Response Style (ERS) Truncated Scales Response Range (RR) Mild Response Style Midpoint Response (MPR) Acquiescence Social/Anti-Social Styles Response Style (ARS) Disacquiescence Response Style (DARS) Socially Desirable Responding (SDR) Noncontingent Responding (NCR) 15
  • 16. Response Styles by Country: Informed by Culture Source: Johnson, Kulesa, Cho, Shavitt, 2003; Vovici 16
  • 17. Halo Error & Response Substitution • What makes cheesesteaks and Tastykakes® taste even better? • Fans like the food 11% more when the Eagles win. Victory tastes delicious. • Halo error confuses the true strengths and weaknesses of products and services. • Makes benchmarking attributes across competing brands and products unreliable (brands are well documented as introducing halo effects). • Leads to misinterpretation of satisfaction attributes. • Response substitution is when respondents' answers to questions might sometimes reflect attitudes that respondents want to convey, but that the researcher has not asked about 17
  • 18. What Respondents Like About Taking Surveys Nothing 34% Dislike surveys 24% Chance to voice their opinion 13% Earn an incentive 11% Like to be helpful 10% Interesting topics 9% Answers make a difference 7% 0% 20% 40% 60% 80% 100% Source: Vovici Survey Nation study, N = 100 RDD sampled U.S. adults 18
  • 19. The Eightfold Path Right Intention: Worthy Purpose for Survey 19
  • 20. The Second Factor: Worthy Purpose For a Survey Focus on a Goal Be precise about what information you need to gather and what you plan on doing with it If your organization hasn’t done a survey in a while, the tendency is for every department to chime in with questions they want you to ask A narrow goal will help you to relentlessly simplify the survey "In the case of archery, the hitter and the hit are no longer two opposing objects, but one reality." - Zen in the Art of Archery 20
  • 21. Is a Survey the Right Arrow to Hit the Target? Sometimes the best survey is to not do a survey at all Talk to stakeholders who will use the data to understand their wants and needs Is someone elsewhere in the organization doing a survey on this topic or researching this issue? Are customers (or employees or …) the only source of this information? "Two hands clap and there is Do your CRM, web analytics or other systems a sound. What is the sound hold data that would address this issue? of one hand?" - Hakuin Ekaku 21
  • 22. Examples of Goals Good Goals Bad Goals Determine how often callers into It’s been a while; we should do a help desk check the customer- survey. service web site before calling What are our customers thinking Prioritize the feature ideas for a right now? new product with prospects Maybe customers would like it if we Find out how satisfied customers provided support after 6 pm. are with tech support Let’s create a committee and see Determine if employees feel that what they want to find out from a organization is living up its core survey. values 22
  • 23. The Eightfold Path Right Order: Questions’ Proper Sequence 23
  • 24. The Third Factor: Questions’ Proper Sequence Screener Use the inverted pyramid approach, Open-Ended Questions drilling down General Questions Ask harder questions first, before respondents Specific Questions grow tired of the survey Demographics Follow-up 24
  • 25. The Third Factor: Questions’ Proper Sequence Screener Conventionally, screeners route people out of survey Open-Ended Questions depending on answers to initial question General Questions Build panel profiles that Specific Questions including the screening Demographics information you need You have their attention; skip Follow-up them out to another survey Otherwise they get the impression their feedback isn’t valuable 25
  • 26. The Third Factor: Questions’ Proper Sequence Screener Capture their views in their own words before biasing Open-Ended Questions them with your later questions General Questions o “What, if anything, do you Specific Questions like about…?” Demographics o “What, if anything, do you dislike…” Follow-up Answers provide color commentary to later closed- ended questions Answers validate choice lists 26
  • 27. Trade Off Increased Abandonment for More Verbatim Responses Survey Concluding with Survey Beginning with Open-Ends Open-Ends Abandonment rate 1% 6% Responses with at least 61% 90% one verbatim answered Average length of 13 words 13 words verbatim answer Sample size 70 79 Source: Vovici research on research, 9-24-09 27
  • 28. The Fourth Habit: Order Questions Logically Screener Use Skip Use Patterns Branching Open-Ended Questions General Questions Specific Questions Demographics Follow-up 28
  • 29. Sequential Filtering Provides Better Conversational Flow Sequential Filtering Grouped Filtering Source: Lisa Carley- Baxter, Andy Peytchev and Michele C. Black, 2010 29
  • 30. The Third Factor: Questions’ Proper Sequence Screener Use demographic and firmographic questions to Open-Ended Questions profile respondents and their organizations General Questions Enables you to cross-tabulate Specific Questions & compare subgroups Demographics Pre-populate from CRM systems where possible Follow-up Place near end as tedious and intrusive but can be answered on “autopilot” 30
  • 31. The Third Factor: Questions’ Proper Sequence Screener Ask for any final comments about any aspect of survey or Open-Ended Questions topic General Questions Ask for permission to follow- up with them about their Specific Questions answers Demographics Prompt if they have an issue they want to be contacted Follow-up about 31
  • 32. The Third Factor: Questions’ Proper Sequence Screener Only at the very end ask respondent satisfaction Open-Ended Questions questions evaluating the survey General Questions Use to drive continual Specific Questions improvement to your Demographics research process itself Key measures: Follow-up How interesting was the survey How long was it Open comments 32
  • 33. The Eightfold Path Right Attention: How Each Question Listens 33
  • 34. Four Basic Question Types Open-Ended Questions Closed-Ended Questions Essay Question Fill in the Blank Choose One Choose Many 34
  • 35. Open-Ended Questions vs. Closed-Ended Questions beige blue Bluw green purpel red fire engine Black blue red green purple red sea black blue Gray grey red green sky black blue Green orange red blue blue Blue r0x!! green orange red yellow 35
  • 36. Best Practices Open-Ended Questions Closed-Ended Questions Great for hearing from the respondent in Make sure the list contains all common their own words choices Provides unbiased, unfiltered answers Better to have too many items rather than Good for catching anything you missed at too few, but try not to clutter the list end of a survey Provide the respondent with an “Other – Limit use, though, because: please specify” choice o Time consuming and taxing to Arrange the choices in logical order answer If no logical order, then o Difficult to analyze randomize the order 36
  • 37. Yes/No Questions: Common Pitfalls Force-fitting a question into a yes/no format by overriding what “Yes” or “No” means. Asking for a single Yes/No to multiple items. Letting the user select both Yes and No. Providing caveats to the Yes/No choices. Asking questions that can’t be answered Yes/No. Listing a bunch of similar yes/no questions in a matrix. Asking questions that no one wants to say “no” to (“acquiescence response bias”). 37
  • 38. Choose-Many Questions Use whenever more than one choice is applicable Always include a “none of the above” as an exclusive choice, otherwise can’t tell if respondent answered question Avoid providing choices that can be synonymous or subsets/supersets of one another Don’t use a list box to show the choices If the choice list has no natural order, randomize the order (anchoring “None of the above” to bottom) 38
  • 39. Matrix/Table/Grid Questions Concise technique for combining questions with common topics Can be 50% faster for respondent to complete but speed may lead to mistakes Source: SSI, “Grid Test Summary”, 2009 39
  • 40. Reimagine Matrix Questions when You Can "We know respondents don't like grids... Now, we're beginning to learn that not only are grids frustrating for respondents - they actually produce inferior data." - Jackie Lorch, SSI VP Break each row of the matrix into a separate question or group of questions on its own page Rewrite each row of the matrix into separate questions, replacing checkboxes with fully labeled scales Refactor importance matrixes into choose-many questions Refactor Yes/No Matrix questions into choose-many questions (checkbox lists) instead Be creative 40
  • 41. Mental Effort of Rating vs. Ranking Questions Ranking Questions are More Taxing for Respondents 70 60 Rating Comparisons 50 Ranking 40 30 20 10 0 2 3 4 5 6 7 8 9 10 11 12 # Choices 41
  • 42. When in Doubt, Use Rating Questions Ranking Questions Rating Questions Taxing for respondents, requiring Lead to less differentiation among them to compare multiple items choices against one another Ratings often fall into narrow upper Difficulty increases disproportionately band as choices are added Personal variations in rating styles Take three times longer to answer Possible spurious positive correlations than rating questions (Munson and due to individuals’ personal variations McIntyre, 1979) Limit range of statistical analysis Alwin, D. F., & Krosnick, J. A. (1985). “The measurement of values in surveys: A comparison of ratings and rankings.” 42
  • 43. Constant Sum & Allocation Questions Limit the number of items If you have too many items, break them into categories and then ask follow-on allocation questions If appropriate, include an Other category as a safety valve for respondents Use matrix questions instead if respondents cannot easily recollect and quantify their past behavior Provide visual feedback to the current working sum 43
  • 44. The Eightfold Path Right Speech: Asking Objective Questions 44
  • 45. The Fifth Factor: Right Speech Asking Objective Questions Respondents should not be able to determine where you stand on any topic o Use nonjudgmental wording o Choose neutral terms Don’t ask leading questions o Not “What do you like about your service?” Nan-in served tea. He poured his visitor's cup full, and then kept on pouring. The professor watched the overflow o But “What, if anything, do you like…?” until he no longer could restrain himself. "It is overfull. No more will go in!" Write from the respondent’s perspective not your "Like this cup," Nan-in said, "you are full of your own opinions and speculations. perspective How can I show you Zen unless you first empty your cup?" 45
  • 46. The Fifth Factor: Right Speech Asking Objective Questions Remove ambiguity: “What is your favorite drink?” (drink = beverage or drink = alcoholic beverage) Ask one item at a time not: “How would you rate our price and service?” not: “How easy to reach someone to help?” Avoid industry jargon Specify how you use general terms Don’t make subtle distinctions Have others proofread your questions for clarity Pre-test survey with a segment of your audience 46
  • 47. The “Shoulds” of Question Wording Q. should be focused on a single issue or topic Q. should be interpreted the same way by all respondents Q. should use the respondent’s core vocabulary Q. should be a grammatically simple sentence if possible Q. should be brief 47
  • 48. The “Should Nots” of Question Wording Q. should not assume criteria that are not obvious Q. should not be beyond the respondent’s ability or experience Q. should not use a specific example to represent a general case Q. should not request recall of specifics when only generalities will be remembered Q. should not require the respondent to guess a generalization Q. should not ask for details that cannot be related Q. should not contain words that overstate the condition Q. should not have ambiguous wording Q. should not be “double-barreled” Q. should not lead the respondent to a particular answer Q. should not have “loaded” wording or phrasing. 48
  • 49. Test, Test, Test Self-Test Pre-test Pilot Test Publish Self-Test Question flow Answer validation Question wording Required answers Question types Skip patterns Scale consistency Errors of omission 49
  • 50. The Eightfold Path Right Choice: Prompting with Care 50
  • 51. What’s Your Favorite Color? Poll A – 6 Choices Poll H – 15 Choices Green Blue Purple Pink Green Orange Red Black White Blue Red Black Maroon Yellow Magenta Brown None of those! Gray Lime Green Sky Blue Hot Pink Whatever Hayley's favorite color is ;) 51
  • 52. Provide Exhaustive Lists of Choices Top 3 Favorite Colors As Determined by Questions with Different Numbers of Choices 15 10 5 0 A B C D E F G H Correct Ranking 52
  • 54. Impact of Showing Logos May increase brand awareness recall May be more accurate for assessing print, web and video campaigns May be less accurate for assessing radio campaigns 54
  • 55. To Label or Not Label Each Point of a Scale Many Variations Possible Best Practices Respondents prefer fully labeled scales Fully labeled scales have greater reliability and validity Numeric values alter the meaning of labels and should be avoided 5-point unipolar and 7-point bipolar scales have greatest reliability and validity Where possible use standard scales rather than write your own Source: Krosnick, J. A., & Fabrigar, L. R. (1997). “Designing rating scales for effective measurement in surveys.” 55
  • 56. Are Respondents Too Agreeable? Likert Scale Best Practices Completely disagree The traditional Likert scale is Disagree obsolete Somewhat disagree Over 100 studies have demonstrated acquiescence bias Neither agree or disagree Use “construct-specific response Somewhat agree options” instead – common rating Agree scales and custom scales Completely agree - Saris, Krosnick & Shaeffer (2005) 56
  • 57. Patterns to Use for Scales “He's embiggened that role with his Unipolar Scale (0..100) cromulent Bipolar Scale (-1..0..+1) performance.” Completely* disembiggened Not at all cromulent Slightly cromulent Mostly disembiggened Somewhat disembiggened Moderately cromulent Neither embiggened nor Very cromulent disembiggened Completely* cromulent Somewhat embiggened Mostly embiggened *or “Extremely” where appropriate Completely* embiggened 57
  • 58. Common Survey Rating Scales Acceptability Totally unacceptable, Unacceptable, Slightly unacceptable, Neutral, Slightly acceptable, Acceptable, Perfectly acceptable Agreement Completely disagree, Disagree, Somewhat disagree, Neither agree or disagree, Somewhat agree, Agree, Completely agree Amount of Use Never use, Almost never, Occasionally/Sometimes, Almost every time, Frequently use Appropriateness Absolutely inappropriate, Inappropriate, Slightly inappropriate, Neutral, Slightly appropriate, Appropriate, Absolutely appropriate Awareness Not at all aware, Slightly aware, Moderately aware, Very aware, Extremely aware Beliefs Very untrue of what I believe, Untrue of what I believe, Somewhat untrue of what I believe, Neutral, Somewhat true of what I believe, True of what I believe, Very true of what I believe Concern Not at all concerned, Slightly concerned, Moderately concerned, Very concerned, Extremely concerned Familiarity Not at all familiar, Slightly familiar, Moderately familiar, Very familiar, Extremely familiar Frequency Never, Rarely, Sometimes, Often, Always Importance Not at all important, Slightly important, Moderately important, Very important, Extremely important Influence Not at all influential, Slightly influential, Moderately influential, Very influential, Extremely influential Likelihood Not at all likely, Slightly likely, Moderately likely, Very likely, Completely likely Priority Not a priority, Low priority, Medium priority, High priority, Essential Probability Not at all probable, Slightly probable, Moderately probable, Very probable, Completely probable Quality Very poor, Poor, Fair, Good, Excellent Reflect Me Very untrue of me, Untrue of me, Somewhat untrue of me, Neutral, Somewhat true of me, True of me, Very true of me Satisfaction (bipolar) Completely dissatisfied, Mostly dissatisfied, Somewhat dissatisfied, Neither satisfied or dissatisfied, Somewhat satisfied, Mostly satisfied, Completely satisfied Satisfaction (unipolar) Not at all satisfied, Slightly satisfied, Moderately satisfied, Very satisfied, Completely satisfied 58
  • 59. Developing Customer Scales Labels Reflect Equal Intervals Common Scales 100 Liking Very Poor, Poor, Fair, Good, Excellent 80 Amount 60 None, Some, Half, Most, All Frequency 40 Never, Rarely, Sometimes, Often, Always 20 Likelihood Not at all likely, Slightly likely, Moderately likely, 0 Very likely, Completely likely Source: Krosnick, J. A., & Fabrigar, L. R. (1997). “Designing rating scales for effective measurement in surveys.” 59
  • 60. Use Dropdowns Only for Known Lists Great for letting respondents pick from a long list of known choices (e.g., states, provinces) Even casual users will use the keyboard effectively (e.g., click M three times to cycle from Maine to Maryland to Massachusetts) Make sure not to make any choice the default (e.g., having Alabama selected) Make the default choice an instruction like “Click here to choose” Don’t use when the choices have to be read to be understood, as in lists of industries or job titles 60
  • 61. “Don’t Know” or “No Opinion” Choices When “satisficing”, respondents will select a no-opinion choice if presented if one, sometimes even if they have an opinion. When no such choice is presented, most respondents will choose from the other choices. Omit a no-opinion choice when asking for attitude. Provide a “Don’t Know” choice when prompting to recall specifics. Source: Krosnick, J. A., & Fabrigar, L. R. (1997). “Designing rating scales for effective measurement in surveys.” 61
  • 62. Juxtaposing Next & Previous Buttons 1.20% Rate of Previous Button Use 1.00% 0.80% 0.60% 0.40% 0.20% 0.00% 9.4 9.5 9.6 9.7 9.8 9.9 10 10.1 10.2 10.3 Completion Time (minutes) Source: Couper, Baker, Mechling 62
  • 63. Juxtaposing Next & Previous Buttons 1.20% Rate of Previous Button Use 1.00% 0.80% 0.60% 0.40% 0.20% 0.00% 0 2 4 6 8 10 12 Completion Time (minutes) Source: Couper, Baker, Mechling 63
  • 64. The Eightfold Path Right Length: Not Too Short, Not Too Long 64
  • 65. Right Length: Not Too Short, Not Too Long 2-4 questions: Transactional Survey 5-10 questions: Event Evaluation 10-20 questions: Customer Satisfaction 20-30 questions: Planning 50-70 questions: Major Account Review 70-90 questions: Employee Satisfaction 65
  • 66. 66
  • 67. The Sixth Habit: Shorten the Survey 100% Abandonment Rate Decrease the number 90% Of 180-Question of matrix/grid 80% Survey questions 70% 60% Reduce the number of 50% open-ended questions 40% Put demographic 30% questions at the end 20% Shorten the survey! 10% 0% Intro 10 12 13 14 15 16 17 18 19 20 1 2 3 4 5 6 7 8 9 11 Average 6 questions per block Source: “Dropouts on the Web”, Galesic, 2006 67
  • 68. Causes of Survey Incompletes Primary Reason Respondent Abandoned Survey Subject matter 35% Media downloads 20% Survey length 20% Grids 15% Too many open-ends 5% 0% 20% 40% 60% 80% 100% Source: Lightspeed Research 68
  • 69. Interesting Questionnaires are Perceived as Shorter Optimal 1 length 0.9 0.8 0.7 } perception gap response rates”, 2002, Mirta Galešic 0.6 “Effects of questionnaire length on Somewhat 0.5 too long Higher interest in questionnaire 0.4 Lower interest in questionnaire 0.3 0.2 Absolutely 0.1 Source: too long 0 Fewer Average More questions number of questions questions 69
  • 70. Tips for Shortening the Survey Keep Your Focus – Remove questions that don’t directly address the goal of the survey Ask Only Most Important Questions – Common research tactic to have three similar questions on similar topic: use one Don’t Ask Esoteric Questions – Cut questions that make distinctions only apparent to those within your organization Don’t Set False Expectations – Remove questions that raise issues that can’t be addressed (for customers, free services; for employees, extended vacation time) 70
  • 71. Shorten the Survey from the Respondent’s Perspective Skip Respondents Past Inapplicable Sections – Don’t subject respondent to survey about products or services they don’t have or can’t have Import Answers – Use CRM data to pipe in answers to “hidden questions” Randomize Displayed Sections – For less important sections, randomly display only one section to each respondent Break into Multiple Questionnaires – Maybe questions around different target groups are so different that they are best served with different questionnaires Use Fewer Pages – Page submits add a burden, so the fewer pages the better for the most part. Keep the Questionnaire Interesting – Respondents perceive interesting surveys as shorter! 71
  • 72. The Eightfold Path Right Action: Sharing the Survey’s Karma 72
  • 73. Right Action Sharing the Survey’s Karma Honor the time and energy of your respondents Share the results Change and grow to serve them better Use survey triggers to immediately take action 73
  • 74. CRM Survey & System Panel Database Aggregate Responses Poor Rating Notification Manager Customer Survey with CRM Appended w/Customer Data Integration & Product Line Data or Customer Recovery 74
  • 75. Use Survey Alerts/Email Triggers to Improve Satisfaction Don’t Just Measure Satisfaction—Intervene to Improve It! Email appropriate department when a respondent provides a negative rating of service Reactive o Help-desk ticket satisfaction o General dissatisfaction o Hotel-stay satisfaction o Major-account satisfaction Proactive o Customer-service satisfaction o Literature fulfillment 75
  • 76. The Eightfold Path 1. Right Perspective: The Respondent as Human 2. Right Intention: Worthy Purpose for Survey 3. Right Order: Questions in the Right Order 4. Right Attention: How Each Question Listens 5. Right Speech: Asking Objective Questions 6. Right Choice: Prompting with Care 7. Right Length: Not Too Short, Not Too Long 8. Right Action: Fulfilling the Survey’s Karma 76
  • 77. Questions & Answers To request a demonstration contact your sales executive or: 1-800-787-8755 sales@vovici.com I welcome your questions: Jeffrey Henning, jhenning@vovici.com Free e-book available from http://blog.vovici.com 77

Editor's Notes

  1. http://www.imdb.com/title/tt0095159/quotes
  2. http://blog.vovici.com/blog/bid/18156/Satisficing-and-Survey-Respondent-Behavior
  3. http://blog.vovici.com/blog/bid/18156/Satisficing-and-Survey-Respondent-Behavior
  4. http://blog.vovici.com/blog/bid/18156/Satisficing-and-Survey-Respondent-Behavior
  5. http://blog.vovici.com/blog/bid/18225/Long-Surveys-Turn-Respondents-into-Liars
  6. http://blog.vovici.com/blog/bid/18156/Satisficing-and-Survey-Respondent-Behavior
  7. http://blog.vovici.com/blog/bid/57499/Understand-Respondent-Behavior-to-Write-Better-Surveys
  8. Egocentric bias: recalling the past in a self-serving manner, e.g., remembering one's exam grades as being better than they were, or remembering a caught fish as bigger than it really wasLeveling and Sharpening >: memory distortions introduced by the loss of details in a recollection over time, often concurrent with sharpening or selective recollection of certain details that take on exaggerated significance in relation to the details or aspects of the experience lost through leveling. Both biases may be reinforced over time, and by repeated recollection or re-telling of a memory.Telescoping effect > >: the tendency to displace recent events backward in time and remote events forward in time, so that recent events appear more remote, and remote events, more recent.</li>
  9. http://blog.vovici.com/blog/bid/21978/Acquiescence-Bias-Agree-Disagree-Scale-Best-Practices
  10. http://blog.vovici.com/blog/bid/25082/Social-Desirability-Bias-Sex-Drugs-Rocking-Surveys
  11. http://blog.vovici.com/blog/bid/52933/Mode-Effects-Same-Survey-Different-Medium-Different-Results
  12. http://blog.vovici.com/blog/bid/28597/Halo-Error-Lose-the-Halo-to-Gain-More-Accurate-Insightshttp://blog.vovici.com/blog/bid/31148/Response-Substitution-Answering-the-Unasked-Question
  13. http://blog.vovici.com/blog/bid/18186/Good-Surveys-start-with-Good-Goals
  14. http://blog.vovici.com/blog/bid/18186/Good-Surveys-start-with-Good-Goals
  15. http://blog.vovici.com/blog/bid/18285/Order-Questions-Logically
  16. http://blog.vovici.com/Blog/bid/18203/Screening-Questions-May-Indicate-Need-for-Better-Survey-Profiles
  17. http://blog.vovici.com/Blog/bid/18095/Open-Ended-Questions
  18. http://blog.vovici.com/blog/bid/21917/How-to-Maximize-the-Number-of-Open-Ended-Responses
  19. http://blog.vovici.com/Blog/bid/18190/Skip-Logic-Conditional-Branches-in-Surveys
  20. http://blog.vovici.com/blog/bid/28235/Sequential-vs-Grouped-Placement-of-Filter-Questions
  21. http://blog.vovici.com/Blog/bid/18176/Demographic-Questions-Sample-Survey-Templatehttp://blog.vovici.com/Blog/bid/18047/Firmographic-Definition-Template
  22. http://blog.vovici.com/blog/bid/27344/Respondent-Satisfaction-Post-Survey-Assessments
  23. http://blog.vovici.com/blog/bid/18197/Types-of-Questions-Four-Building-Blocks-for-Constructing-Questionnaires
  24. http://blog.vovici.com/blog/bid/18088/Open-Ended-Questions-vs-Closed-Ended-Questions
  25. http://blog.vovici.com/blog/bid/18088/Open-Ended-Questions-vs-Closed-Ended-Questions
  26. http://blog.vovici.com/blog/bid/18123/Yes-No-Questions-Common-Pitfalls-to-Avoid-When-Writing-Questionnaires
  27. http://blog.vovici.com/blog/bid/18170/Multiple-Answer-Questions-Select-All-That-Apply-Best-Practices
  28. http://blog.vovici.com/blog/bid/18201/Matrix-Questions-Powerful-but-Perilous-Technique
  29. http://blog.vovici.com/blog/bid/26649/Bullet-Time-for-Matrix-Questions
  30. http://blog.vovici.com/blog/bid/18228/Ranking-Questions-vs-Rating-Questions
  31. http://blog.vovici.com/blog/bid/18228/Ranking-Questions-vs-Rating-Questions
  32. http://blog.vovici.com/blog/bid/57326/Constant-Sum-Allocation-Questions-Best-Practices
  33. http://blog.vovici.com/blog/bid/18230/Writing-Objective-Survey-Questions
  34. http://blog.vovici.com/blog/bid/18230/Writing-Objective-Survey-Questions
  35. http://blog.vovici.com/blog/bid/56281/The-Shoulds-Should-Nots-of-Question-Wording
  36. http://blog.vovici.com/blog/bid/56281/The-Shoulds-Should-Nots-of-Question-Wording
  37. http://blog.vovici.com/blog/bid/18842/Survey-Test-Mode
  38. http://blog.vovici.com/blog/bid/18101/Favorite-Color-Survey-Results-How-Short-Choice-Lists-Lead-to-Wrong-Answers
  39. http://blog.vovici.com/blog/bid/18101/Favorite-Color-Survey-Results-How-Short-Choice-Lists-Lead-to-Wrong-Answers
  40. http://blog.vovici.com/blog/bid/18076/Rating-Scale-Labels-Label-End-Points-or-Every-Point
  41. http://blog.vovici.com/blog/bid/21978/Acquiescence-Bias-Agree-Disagree-Scale-Best-Practices
  42. http://blog.vovici.com/blog/bid/18620/Custom-Scale-Development
  43. http://blog.vovici.com/blog/bid/18261/Common-Scales-to-Use-when-Writing-Questions
  44. http://blog.vovici.com/blog/bid/18620/Custom-Scale-Development
  45. http://blog.vovici.com/blog/bid/18083/When-to-Use-Drop-down-Lists-and-When-Not-To
  46. http://blog.vovici.com/blog/bid/20397/No-Opinion-as-a-Question-Choice
  47. http://blog.vovici.com/blog/bid/18133/Recommended-Survey-Length
  48. http://relevantinsights.com/long-surveys
  49. http://blog.vovici.com/blog/bid/17976/The-Long-and-the-Short-of-Questionnaire-Length
  50. http://blog.vovici.com/blog/bid/50909/How-To-Shorten-a-Questionnaire
  51. http://blog.vovici.com/blog/bid/17976/The-Long-and-the-Short-of-Questionnaire-Lengthhttp://blog.vovici.com/blog/bid/23110/Perceived-Questionnaire-Length
  52. http://blog.vovici.com/blog/bid/18174/Closing-the-Feedback-Loop-Sharing-Results-with-Online-Community-Members-Respondents
  53. http://blog.vovici.com/blog/bid/18667/Email-Trigger-a-Key-Aspect-of-EFM
  54. http://blog.vovici.com/blog/bid/18053/Survey-Alerts-Trigger-Emails-Improve-Satisfaction