The document discusses improving data quality in surveys. It covers topics like non-response bias, questionnaire design, question construction, and pre-testing. To address non-response bias, the document recommends documenting non-responses, looking for correlations with response rates, using incentives thoughtfully, and gathering information from interviewers about non-respondents. For best questionnaires, it suggests starting with clear objectives, following good design practices, crafting clear questions informed by research, and pre-testing surveys. The overall goal is to reduce errors and biases to improve the reliability and validity of survey data.

2. John Barrett Rudly Raphael
President, QuestionPro AudienceCEO, Priority Metrics Group
www.pmgco.com

3. QuestionPro Audience
20 Million
Respondents
B2B and B2C
Solutions
Mobile Research
Panels
10 Specialty
Panels
• Global Scalability
• Quality
Recruitment
• Ongoing quality
control
• Panel
Respondents
• Business Panels
• Customer Panels
• Real time results
• Mobile
community
• Robust platform
• Veterinarians
• Building
Contractors
• Pet Owners
• Registered Voters
• And more!

4. #DataQualitytips

5. Agenda
1. Data quality
2. Data quality problems
3. Conversion
4. Interviewer training
5. Reducing non-response bias
6. Questionnaire design
7. Question Construction

6. Improving Data Quality

7. Importance of Data Quality

8. Quality Problems – Total Survey Error
Groves (1989) identifies three categories of error:
1. Coverage – some members of the population under study do not have a known nonzero
chance of being included in the sample.
2. Measurement effect – the instrument or items on the instrument are constructed in such
a way to produce unreliable or invalid data.
3. Non-Response effect – nonrespondents in the survey sample differ from respondents in
ways that are germane to the objectives of the survey.

9. Total Survey Error … for the rest of us
Measurement Effect Non-Response Effect

10. Data Quality Problems
1.Sampling
2.Non-Response
3.Questionnaire Design
4.Execution/Methodology
5.Quality Control
6.Analysis/Reporting
7.Costs
Poll: Which of the following has a direct effect on
data quality, as it relates to your research?
Please select the top 3.
a. Poor sample
b. Poor questionnaire design
c. Poor execution/methodology
d. Poor quality control procedures
e. Poor survey incentives
f. Poor open-end responses
g. Limited analysis/reporting tools
h. Limited research budget http://buff.ly/2n7Lk30

11. Non-Response

12. Non-Response Documentation
• Document non-response using AAPOR categories:
❏Interview
❏Eligible, non-interview
❏Unknown eligibility, non-interview
❏Not eligible
• Identify correlates of non-response

13. Non-Response vs. Maximizing Response Rates
Survey of healthcare leaders: web-based,
self-report, four follow-ups
• Overall 95% response rate, examined
results in each of five waves
Response Wave and Evaluative Attitudes Assessed Using a Multi-Item Scale:
Mean and 95 Percent Confidence Intervals for the Alignment and Commitment
Scale (ACS).
Conclusion: Although high response rates
are desirable because of their effect on
precision and power, absolute thresholds
representing “adequate” survey response
rates may not be accurate.
Source: “Response Rates, Nonresponse Bias, and Data Quality: Results from a National Survey of Senior Healthcare
Leaders, “Meterko et al, Public Opinion Quarterly, January 27, 2015

14. Conversion: Non-Respondent Sees the Light!
✓ Different versions of survey
introduction
✓ Study contact rules
✓ Incentives
✓ Interviewer style and training
Research literature reports telephone
conversion rates of 5% to 40%.

15. Interviewer Training – The Often Overlooked Nugget
✓ Interviewer hiring practice
✓ Thorough interview training
❏Survey background
❏Questionnaire familiarity
❏5-second rule: Introduction, survey purpose
❏Timing of contact
❏Personality of interviewer
✓ Multiple contacts
❏May increase response rates dramatically The right person makes a
difference.

16. Reducing Non-Response Bias
• Good survey design increases response rates
• Good survey communication increases response rates
Incentives can increase response rates
Clever
Foreshadow
✓ Do not cross “coercive threshold”
✓ Donation to charity (business)
✓ Prepaid (vs. promised) incentive
(consumer)
✓ Differences decrease with follow-up
✓ May lead to lengthier open-end
answers
Source: “National Survey of Physicians to Determine the Effect of Unconditional Incentives on Response Rates of Physician Postal
Surveys,”Abdulaziz et al, BMJ Open, Volume 5, Issue 2

17. Reducing Non-Response Bias
Interviewers can gather information about non-respondents
• Observable
characteristics
• Evaluations of
engagement,
honesty, ability
• Visual and verbal
clues
• Observable
characteristics
• Evaluations of
engagement,
honesty, ability
• Verbal clues

18. Questionnaire Design

19. Questionnaire Design Starts with Clear
Objectives
Start End

20. Good Design Practices
✓ Promise (or at least offer) anonymity or confidentiality
✓ Some form of relationship (known brand or existing customer) increases response rate
✓ Simplify the perceived task
• Preserve white space
• Design a logical flow
• Group common items together – similarity and proximity
• Don’t ask unnecessary questions
• Minimize task difficulty
✓ Avoid use of jargon and notation
✓ Include relevant questions using question and page logic based on previous answers

21. Good Design Practices
Improvements
Explanation of task
Question wording
Confidentiality
Positive Elements
Clean, simple
Logical flow
Comment space

22. Visual Display
✓ No more than four visual elements
✓ Identify new information with distinct colors and/or
sizes
✓ Bold type used in favor of italics, underline, or upper
case
✓ Use highlights and graphics sparingly
✓ Differentiate important elements with large, bright or
distinctive colors

23. Visual Display
Positive Elements
Logical flow
Improvements
Jargon throughout
• Contract
• Self-discover
• Influencing style
37 visual elements
Everything is bright
Everything is new
Everything is important

24. Question Construction
✓ Use conversational norms where possible
✓ Use common words with single primary
definition
• Few letters and syllables
• Easy to pronounce
• Avoid abbreviations
✓ Avoid asking certain types of questions:
• Opinions held at prior times
• Explain prior behavior or thoughts
✓ Balance use of closed- and open-end
questions appropriately
• Use open-end for numeric answers
and categorical questions with
unknown breadth of possible
answers

25. Question Construction – Grice’s Maxims
The maxim of quantity, where one tries to be as informative as one possibly
can, and gives as much information as is needed, and no more.
The maxim of quality, where one tries to be truthful, and does not give
information that is false or that is not supported by evidence.
The maxim of relation, where one tries to be relevant, and says things that
are pertinent to the discussion.
The maxim of manner, when one tries to be as clear, as brief, and as orderly
as one can in what one says, and where one avoids obscurity and ambiguity.

26. Question Construction – Response Process
(Tourangeau Model)
Response
based on recall or educated guess from
cues or inferences, often requires choice
of answers to report and how to report
(agree or strongly agree)
Comprehension
how respondents understand the
questions and infer the question’s point
Judgment
assessment of completeness or sufficiency of
information/opinion, appropriateness and
manner of using inferences, and how to
transform retrieved information into appropriate
answer
Retrieval
recalling information from long-term
memory (behavioral questions) or a
preformed opinion (attitude questions),
but retrieval is rarely complete

27. Question Construction – Bad Examples
1. What is your frequency of utilization of retail travel agents?
2. Are you against the restrictive House Bill 935?
3. Have you visited ___ in the past and did you enjoy your visit?
4. Please rank order the following with 1 being the most important and 7 being the
least important
5. When you purchased ____ what other options did you consider?
6. Did you use the features on our website?
7. In order to speed up your shopping experience, have you used the self check-out
lanes?
8. How likely are you willing to pay $50 for a ____?

28. Rating Scales
✓ Label options with words, not numbers
✓ Ensure range of options covers all points on
continuum
✓ For bipolar constructs use 7-point scale
✓ For single constructs use 5-point scale
✓ Don’t offer “Don’t Know” response
✓ Rotate order of response on categorical
questions and wherever else it is practical
How satisfied were you with the speed of check-out?

29. Communication Plan
✓ Invitations and advance/introductory letters
✓ Self-administered questionnaires
✓ Reminder letters/emails

30. Self-
Administered
Interviewer-
Administered
Computer-
Administered
Importance of Pre-Test
Traditional pre-test – match survey method to complexity
Cognitive pre-test – think-aloud

31. Conclusions – Part 1
To address non-response bias:
1. Document and categorize non-responses
2. Look for correlations
3. Don’t be fooled by high response rate
4. Convert!
5. Hire the best interviewers and train them well
6. Use well-designed questionnaires
7. Use incentives thoughtfully
8. Gather information from interviewers about non-respondents (back to #2)

32. Conclusions – Part 2
To build the best questionnaires:
1. Start with the end in mind
2. Follow GDPs
3. Establish relationship
4. Use visual elements in limits
5. Craft questions that are clear and direct
6. Test against Grice’s Maxims
7. Think like a respondent (Tourangeau Model)
8. Benefit from research learnings about rating scales
9. Develop and implement a communication plan
10. Pretest, and if necessary, pre-test again

33. Poll Results
Poll: Which of the following has a direct effect on data quality, as it relates to your research?

34. Thank You!!!
Questions?
questionpro.com
@questionpro
sampleprojects@questionpro.com
sales-team@qsample.com