How to design surveys; describes differences between approaches to measuring awareness, opinions, perceptions, behaviors, needs and attitudes; describes roles of survey sponsor and researcher.
2. Learning outcomes
By the end of this session participants will acquire
understanding of the following aspect of survey
design process
• What be measured with a survey?
• Roles (sponsor vs. researcher)
• Survey process
• Design (formulating research question)
• Question development
• Testing (and train)
• Data collection
• Analysis
3. Reasons for conducting
surveys
• To uncover the answers. Learn about what motivates survey respondents and
what is important to them, gather meaningful opinions, comments, and feedback.
• To evoke discussion. Give your survey respondents an opportunity to discuss
important key topics. Communicate with your respondents about your survey
topic. This allows you to dig deeper into your survey, and can incite topics
related to your survey within a broader perspective.
• To provide objective evidence for decisions. Conducting surveys is an
unbiased approach to decision-making. Don’t rely on “gut feelings” to make
important business decisions. You can collect unbiased survey data and develop
sensible decisions based on analysed results. By analysing results, you can
immediately address topics of importance, rather than waste time and resources
on areas of little concern.
• Compare results. Survey results provide a snapshot of the attitudes and
behaviours of your target populations. This feedback is your baseline to
measure and establish a benchmark from which to compare results over time.
4. When NOT to conduct
surveys…
• Your reason for doing the survey starts with, “It would be nice to know….”
• You don’t know much about your target audience and you want to use the
survey to learn about them (focus groups or personal interviews would be a
better first step).
• You have already made your decision; you just want to see how it is going to be
received.
• You don’t have the enough resources to act on the information you get back.
• You don’t have enough money to do your survey right without cutting corners.
• You’ve never done a survey before and you aren’t getting help from someone
who has.
5. Before you design your
survey…
• You should be clear about…
• The main goal of the survey
• How you going to apply the survey data
• The decisions you will make as a result of the survey
data
• Who are your target population(s)
• Write outline of your final report…
6. Measuring awareness
• Awareness questions measure consciousness of
particular issues.
• Asking how familiar people are or how much
people think they know is acceptable when
measuring public awareness.
• An example is, “Before this survey, were you very,
somewhat, or not at all familiar with the Liaison
Librarian services at ULS?”
7. Measuring opinions
• Opinion questions measure beliefs for ideas held with
confidence but not substantiated by direct proof or
knowledge.
Examples:
“In general, how effective would you say freshman
composition library instruction is in preparing students for
their research assignments? Would you say they are very
effective, somewhat effective, or not at all effective?
“In the past three years, would you say the WIFI
performance in Hillman has improved, stayed the same,
or gotten worse?”
8. Measuring perceptions
• Measures the distance (gap) between the ideal and the
image(perception) profile of the respondent
• Can use semantic differential or distance scales
My pizza was: hot _ _ _ _ _ _ _ _ cold
fresh_ _ _ _ _ _ _ _ stale
My job can be described as 1(Not at all) – 7 (Perfectly)
__ Boring __Demanding
__ Easy __ Satisfying
__Rewarding __ Degrading
9. Measuring behaviours
• Behavior and activity questions inquire about what
people actually do: What? Where? When? and how
often?
My day at Hillman
Adding context by measuring
• lifestyle patters
• demographics
• affiliations
10. Measuring needs
• Involves identification of relevant needs and asking respondents to
rank them (e.g. forced comparison or paired comparison)
Rank in order of importance to you (from1 to 3)
More outlets ___
Better WIFI ____
Better lighting ___
For each pair of choices indicate that which is more important to you
More outlets ______ Better lighting ____ Better WIFI _____
Better lighting ______ Better WIFI ______ More outlets ______
11. Measuring attitudes
• Attitude questions measure states of mind or feelings.
• Attitudes cannot be measured with just one question
Generally, a battery of questions is needed to identify
and assess attitudes.
• Attitudes measurement = knowledge, feeling and
action
• Knowledge: Are you aware? Have you heard? List or
name ….
• Feelings: scale to show direction and distance from
neutral
• Action: ask about past, present of future behaviour or
create a hypothetical scenario
12. Measuring decisions
• …or the process of decision making
• Information sources and evaluation criteria are
two components of decision making process that
can be measured through survey
• Sources of information: personal knowledge,
social influence and media sources
• Evaluation criteria: respondents may rank
attributes in order of importance
13. Roles: Survey sponsor
• Explains WHAT information is needed, WHY it is
needed and HOW it may be used (frankly!)
Alreck and Settle, 3rd ed.
2000
14. Roles: Researcher
• Advises sponsor on capabilities and limitations of
research tools
• Understands sponsor’s questions, motivations
and intentions
15. Survey design 1
• What do I want to measure in my survey?
• Attitudes, perceptions, needs, behaviours
• What will I ask to measure it?
• What are characteristics of the variable I want to measure and what questions
should I use to directly/indirectly measure that variable
• Are there any valid pre-existing scales/questions I could use?
• How will I measure it?
• Dichotomous scale (yes/no; true/false),
• Nominal scale (allow for several choices that are not ordered),
• Ordinal scale (can be ordered on continuum),
• Rank ordered (from most xxx to least xxx)
• Interval scale (allows for calculating mean)
• Numeric or semantic Likert-type scale
• Continuous (answers placed anywhere on a scale)
16. Survey design 2
• How will I check for validity?
• Valid means that survey questions measure what they are intended to measure
(and not some other concept)
• Confirm with people knowledgeable in the field and pilot
• How will I administer the test?
• Timing (work backwards from the date results are needed by; consider best times to
contact respondents; need approvals, consider workloads)
• Mode (how do I contact respondents; email; paper; f-2-f or phone interview)
• How will I check for reliability?
• Reliable survey questions measure consistently and accurately what is was
designed to measure
• What will I do with the results?
• What is “good” or “bad” result?
• How will I use this to make decisions?
• What are the results NOT telling me?
• Have outline of final report prepared BEFORE you start designing your survey
17. Testing for reliability
• Test and re-test: administer test to the same group
multiple times and check for consistency of
responses
• Conduct factor analysis: see if there are
correlations in answers to similar questions (e.g.
Cronbach coefficient alpha when want to test
reliability of Likert-type scales)
18. Open-ended questions
• Allow respondents to write answers “in their own
words”
• Potential for rich source of information and can reveal
what respondents value/care for the most
But
• Need coding and scoring by multiple “experts”
• High levels of agreement on the above shows validity
• Greater risk of respondents misunderstanding the
question
19. Closed-ended questions
• Easy in respondents – no need to articulate answers
• Less potential for ambiguity (in how responded interprets
question and how responses are analysed)
• Easy to classify (code) making analysis straightforward
But
• “Suggests” answers that respondents did not consider
before
• Gives respondents no opportunity to give a response
different from that listed
Another option: “open response” question
20. Open response questions
• Eliminates disadvantages of closed and open
questions
• Allows for both specific and open responses
21. Attributes of a good survey
question
• Focusing directly on a single, specific issue
• Which brand do you like the best?
• Which of these brands are you most likely to buy?
• Brevity – long questions risk of lacking focus or clarity
• Can you tell me how many children you have, are they boys or girls
and how old are they?
• What is the age and sex of each of your children?
• Clarity – everyone must interpret the question in the same
way
• How much storage space in your house do you and your
spouse use?
• What proportion of the storage space in your house is
used for your things and how much for your spouse’s?
22. Putting survey questions in a
meaningful order and format
• Opening questions need to be easy to answer and be
non-threatening (“hook them”)
• Follow a logical order (e.g. questions on one topic
grouped together)
• Use a variety of question types (to keep in interesting)
but do so in moderation (do not confuse with different
scales or types of required response)
• Include sensitive questions towards the end (so you
don’t lose them early on); do not save important
questions to the end of a long survey (respondents will
be bored by them and will want to finish asap)
