1. The Good, the Bad, and the Ugly:
How to design an effective survey
Workshop 26.02.2013
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J

2. The Good, the Bad, and the Ugly:
How to design an effective survey
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

3. Workshop overview
10.00-10.15 Introduction to survey design
10.15-11.15 How to design an effective survey
Writing good questions
11.15-11.30 Morning tea
11.30-12.30 Tips and tricks
How to maximise response rate
Getting the most out of your data
12.30-1.00 Q&A and Close
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

4. Introduction
So you want to create a survey...
 What is the purpose of the survey?
• Needs assessment
• Impact assessment
• Knowledge assessment
• Client satisfaction / program feedback
• Program evaluation
 What survey mode will you use?
• Telephone, online, paper, interview
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

5. When to use a survey
 To collect information from a large number of
people or records
 To collect this information relatively inexpensively
 To say something about the population from which
the sample was drawn
 When you do not need very in-depth information
 When you want the data collection procedure to
be as standardised as possible
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

6. Some advantages of surveys
 Anonymity
 Asking sensitive questions
 Standardised data to monitor trends
 Quick to administer
 Simple data analysis (%, frequencies)
 Data can be presented as graphs/charts
 Multiple distribution channels (online,
paper, social networking sites)
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

7. Some limitations of surveys
 Limited explanations for the data
 Comments open to a range of
interpretations
 Don‟t know why questions were skipped
or why survey was partially completed
 No opportunity to explain questions that
people don‟t understand
 Can‟t guarantee the survey was even
received or completed by the right person
 Basic literacy levels needed
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

8. What makes a ‘good’ survey?
Have you ever created a survey and felt
disappointed with the data you collected?
Have you ever been asked to answer a survey and
felt irritated with the questions asked?
 A good survey considers both your needs for data
and the capabilities and experience of the
respondent.
 Collects complete, accurate, and reliable data.
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

9. What is a ‘bad’ survey?
 Too wordy
 Language is hard to understand
 Bad timing - e.g. respondents tired at the end of a session
 No „Other‟ response category
 Questions that seem irrelevant to the topic
 Repetitive
 Boring – e.g. same structure throughout, no images
 Does not consider the capabilities of the respondents
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

10. Is a long survey a bad survey?
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

11. HOW TO DESIGN AN
EFFECTIVE SURVEY
IN 3 EASY STEPS!
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J

12. Step 1.
Establish your survey goals!
Ask yourself:
 Why am I creating this survey?
 What do I hope to learn or
accomplish?
 How do I plan on using the data
that I collect?
 How will the data influence my
decisions?
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

13. Step 2.
Create a ‘dummy table’
To ensure all objectives are being addressed and all
questions address objectives. Consider data analysis
before, during and after survey development.
Areas of inquiry Questions
Demographics
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

14. A dummy table …
• Ensures you can interpret data
• Ensures each question will collect the
information you need
• Ensures you leave nothing of importance out
• Ensures you don‟t collect more information
than you need
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

15. Example: Health Care Experience of the
LGBTI* Community in WA
Activity – Find questions related to:
Area of inquiry Questions
LGBTI health issues
Demographics and lifestyle
Current experience of healthcare
services
How and where to reorient current
health services
*LGBTI – lesbian, gay, bisexual, transgender, intersex
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

16. Answers
Area of inquiry Questions
LGBTI health issues 10,11,24
Demographics and lifestyle 1,2,3,4,5,6,7,8,9,15,16
Current experience of healthcare 12,13,14,20,21,22,23
services
How and where to reorient current 17,18,19,25
health services
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

17. Step 3.
Write good questions
Try to use a range of question types:
 Open-ended questions
• To explore issues
• Open to a variety of responses (comment boxes)
 Closed-ended questions
• Dichotomous
• Likert scale
• Guttman scale
• Multiple choice (one response, multiple responses)
• Rank order
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

18. Example - Dichotomous
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

19. Example – Likert Scale
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

20. Example - Guttman Scale
 Agreement with any one item implies agreement with the lower-order
items.
 E.g. are you willing to permit _______ to...
- live in your country?
- live in your community?
- live next door to you?
- marry your child?
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

21. Example – Multiple Choice (one response)
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

22. Example – Multiple Choice
(multi-response)
Evaluation of Health Promotion to design an effective survey
How
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J
382

23. Example – Rank Order
 Rank the following methods of receiving a positive STI
diagnosis in order of preference from highest to lowest
(1=your most preferred method; 5=your least preferred
method)
• SMS______
• Phone call ______
• In person ______
• Email ______
• Letter ______
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

24. Developing a Client Satisfaction
Survey
Example: The Wheatbelt Resilience Program
 Included a series of community workshops
 A survey was developed to find out how useful the
workshops were in terms of increasing awareness of the
Wheatbelt Resilience Program; understanding of suicide
and suicidal behaviours; and opportunities to help with this
issue in the community
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

25. Can you identify
any ways in which
this feedback sheet
could be improved?
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

26. The questionnaire
was adapted to
capture more
specific information
and to also gauge
satisfaction with the
workshop. The
pre-questionnaire
was on a separate
document to the
post-workshop
evaluation.
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

27. Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

28. WRITING GOOD QUESTIONS
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J

29. What do you want to know?
What do you need to know?
 Don‟t waste the opportunity to collect good quality
data by asking vague questions
 Be economical with the number of questions but
make sure you collect all the information required
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

30. Activity: Work in small groups
Re-write these questions and include closed
ended responses:
 (a) “How often do you participate in physical
activities on average?”
 (b) “When were you last physically active?”
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

31. What to avoid
 Double-barreled questions
 Biased/leading questions
 Questions with a double negative
 Loaded questions
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

32. Example: double-barreled question
Please indicate how much you agree or
disagree with each of the following statements
about our service:
I feel welcomed by staff and other youth at the
centre
Example: leading question
Negotiating condom use with a sexual
partner is difficult. Did this session help
you feel more confident to negotiate
condom use successfully?
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

33. Example: double negative
How do you feel about the following statement?
We should not reduce investment in sexual health
services.
Strongly Agree Agree
Disagree Strongly Disagree
Example: Loaded questions
Which of the following services most
urgently requires improvement?
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

34. What to avoid
 Questions which could create a response set
i.e. the tendency for a respondent to answer later questions
the same way they answered earlier questions
 Technical language/jargon
Use simple, common words
 Emotive language
Do you approve of the government‟s oppressive immigration
policy?
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

35. What to avoid
 Using “and”, “or”, “but” in the same question
 Long, complex, multi-part questions
 Questions which contain a negative and a positive
If you were in a foreign country and did not speak the language but knew
someone who did would you…
a. Let the person that spoke the language speak for you Huh?!
b. Try to learn the language on your own
c. Not try to learn the language or let the person that you knew that spoke
the language speak for you.
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

36. What to avoid
Response categories that are not exhaustive
What is your current age?
10 to 20
21 to 30 I‟m 41!
31 to 40
Response categories that are not mutually exclusive
What is your current age?
10 or less
10 to 20 I‟m 30!
20 to 30
30 or greater
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

37. Avoid fuzzy words in the question
 Those words which are open to different
interpretation by the respondents.
• Have you ever had any major injuries during the past year?
 Avoid words like: important, major, better, regular,
often, occasional (subjective interpretation).
 Best to give specific time & details
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

38. Irrelevant questions
 Questions which are irrelevant to the survey
objectives
• E.g. do you really need to know someone‟s income?
 Questions which are irrelevant to the respondent
• e.g. Don‟t ask nurses to compare the effectiveness of different
patient management techniques if they have no experience of the
alternatives.
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

39. Presuming knowledge
 Don‟t assume people either know or have
appropriate experience to answer the
question. What is
bullying?
• Have you been bullied this year? What is a
• How many standard drinks have you drunk in the standard
drink?
last week?
• How many cigarettes have you smoked in the
I don‟t
last week?
smoke!
 Use filter questions or provide more
information to avoid these problems
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

40. Presuming knowledge
 Hypothetical or predictive questions
• Would you like to try marijuana?
• Avoid - many people might like to try anything once.
 Generally avoid:
• Would you like to...
• What would you do if..?
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

41. Personal questions
 Be sensitive!
 May need to preface personal questions to give
respondent „permission‟ to respond truthfully
 Consider the structure of response choices
• people don‟t like to consider themselves at extreme ends of a
scale
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

42. Activity: Design a brief survey to
evaluate this workshop
Workshop objectives
• Increase participants’ knowledge and skills in effective survey design
• Assess participants’ satisfaction with the SiREN training workshop
Areas of inquiry Questions
Demographics
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

43. MORNING
TEA
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

44. TIPS AND TRICKS
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J

45. Critically assess your survey
questions
 Is the respondent‟s understanding of response
categories likely to be influenced by more than
words?
 The vaguer the question and answer categories,
the greater potential for error / poor data
 Is the survey information being collected in more
than one mode? E.g. phone, interview, email
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

46. Critically assess your survey
questions
 Is changing a question acceptable to the survey
sponsor?
 Is the question necessary?
 Will it improve the validity of your study if you do
ask more than one question?
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

47. Critically assess your survey
questions
 How easy is it to get an accurate answer?
 Can people accurately recall and report past
behaviours?
 Will the respondent be willing to reveal the
requested information?
 Will the respondent feel threatened or defensive
by the question?
 Will the respondent feel motivated to answer each
question?
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

48. Tips and tricks
 Insert dummy questions
• These may be irrelevant to the research to avoid respondents
guessing desirable answers
 Reverse the wording of some survey items
• This technique can avoid creating response sets
I always enjoy going to work
I dread going to work
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

49. Tips and tricks
 The order of questions may influence responses
• Put interesting questions first
• Sensitive questions towards end
• Demographic information – location of these questions depends
on importance
• Similar themes together
 Put instructions for questions before the question
not at the beginning of the survey
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

50. Tips and tricks
 Be specific
Example: How can we improve our service?
This is too vague and may elicit single responses about many
different aspects of the service without much guidance to help you
decide which ones are important.
Instead, consider asking respondents to rank from 1-5 the quality of
different aspects of the service
 What claims will you use the data to support?
Be honest and consider if this is realistic or credible. This technique
helps you to clarify the wording of questions
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

51. HOW TO MAXIMISE RESPONSE
RATE
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J

52. Aim to get a 65% response rate
 That is: 65% of your sample complete and
return a questionnaire
 Non-respondents may bias your responses
 Need to consider response rates when selecting
sample size
 Need to consider attrition
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

53. How to maximise response rate
 Involve your target group in design if possible
 Pre-test!
 Ask questions that people can and will answer
 Uplift, motivate and encourage the respondents to
give information
 Avoid bad survey questions
 Allow sufficient time to complete survey
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

54. How to increase responses
 Plan multiple contacts
• Pre-notification (tell people you are planning to survey)
• Reminders for non-responders
 Consider incentives
• Individual incentives more effective than prize draws e.g. incentive
given for getting others to complete the survey
• Dependent on resources
 Obtain ethics approval for your survey
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

55. A note about survey ethics…
Considerations may include:
 Information and consent forms
 Confidentiality and anonymity
 Samples which include minors
 How to ask questions about potentially illegal activities
 Length of survey / burden on respondents
 Format of survey - e.g. acceptability, culturally appropriate
Please contact siren@curtin.edu.au for more advice about survey ethics
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

56. GETTING THE MOST OUT OF
YOUR SURVEY DATA
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J

57. Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

58. Planning and evaluation cycle
Needs
assessment Survey
data
Use different
Re-planning Program types of survey
planning data as part of a
continuous
Survey
data
program
Survey
data
planning and
evaluation cycle
Evaluation
Implementation
Survey
data
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

59. How will you use the data?
 Use survey data to inform the program planning cycle
 Adopt a mindset of continuous program improvement
 After implementing the survey, critically evaluate the
quality and completeness of the data collected
• Were any questions ambiguous?
• Were all the questions necessary?
• Did any questions appear to cause confusion? Ask yourself what
• Did you get some unexpected responses? will you change?
• Were the data useful?
• What decisions could the data inform?
• Are there still gaps in the data?
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

60. Getting the most out of your data
 Don‟t just file the survey data away until the next activity
report for your funding provider is due!
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

61. Health promotion short course –
example survey data
How confident do you Pre course Post course
11 11
feel to undertake a needs
assessment to plan a 3
5
2
health promotion 0
program? Not Confident Very Not Confident Very
confident confident confident Confident
How confident do you feel to Pre course Post course
10 9
contribute to the design and 7
implementation of an 3 3
evaluation of a health 0
promotion initiative? Not Confident Very Not Confident Very
confident Confident confident confident
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

62. Health promotion short course –
example survey data
Pre course Post course
How confident do you feel to 13
12
work collaboratively with
your community and with 4
3
other organisations?
0 0
Not Confident Very Not Confident Very
confident Confident confident confident
 What could you conclude from these data?
 What decisions about the course would you make as a
result of these data?
 Are these decisions valid? What other data might you want
to collect?
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

63. Any questions?
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

64. FOLLOW UP SESSION
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J

65. Follow up session
SURVEY CLINIC
Tuesday 12th March
2013, 10am – 12 noon
Curtin University (Room TBA)
 In two weeks‟ time!
 Aim to put what you have learned into practice
 Either:
1. Critically review a survey you have used previously OR
2. Design a new survey using what you have learned.
 Bring your surveys and any questions to the survey clinic
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

66. Workshop evaluation
Please draw around
your hand (or both
hands) and provide
some feedback on
today‟s workshop.
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J

67. THANKS FOR COMING!
Please don’t forget to complete the workshop evaluation!
Thank you 
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J