Accompanying deck for my 30-minute presentation on survey. Survey is quite a lengthy topic so had to focus on the practicalities of choosing a survey and the rules of thumb around developing questions and the importance of sampling. There is a also a study of the Gallup Poll during the 1948 elections.

1. Dada Veloso-Beltran
dadaspeaks.com
survey

2. primer
nuts & bolts
case study

3. By a small sample we may
judge the whole piece.
Miguel de Cervantes (1547-1616)

4. Evolutionary
Lack sharp corners & sudden turns.

5. primer
brass tacks
case study

6. Answers reveal relationships
to something they measure.
designing questions
to be good measures

7. Qualitative
Intent Bubbles
I want to see the world
through the eyes of my
respondents.
I want to describe the
context in a lot of detail.I want to show
how social
change occurs.
I’m interested
in how things
come to be.
I want to makesure
others can repeat
my findings.
I really spend a lot of
time wondering how
to measure things.
I wonder how
smallpatterns
generalize to
big patterns.
I want to know what
causes something else.
Quantitative
I really want my research approach to
be flexible and ableto change.

8. A standardized stimulus designed to convert
fuzzy phenomenon into data. (Neill. 2014)
Study of a population through
observation of its members.
"The survey is a systematic method for gathering
information from entities for the purpose of constructing
quantitative descriptors of the attributes of the larger
population of which the entities are members.”
(Groves et al. 2004, p.4)

9. Research Design Matrix
MixedMethod
Quantitative
Qualitative
Explore
Theorize
Validate
Optimize
Constructivist
Interpretive
Big Picture
Objectivist
Positivist
Granular
Depth of Insight, Small Sample,
Longer Engagement, Personal
Broadness of Phenomena, Large
Sample, Shorter Engagement, General

10. No
Yes
I know a lot about the topic
I need to summarize thefindings in numerical format
I don’t need a lot of detail and nuance
I need to generalize the resultsto the population
I need to predict the likelihood of a certainthing happening
I wantto measureincremental change
I don’t reallyknow much about the topic
I don’t need to summarize thefindings in numerical format
I need to communicate a lot of detail and nuance
I’m interested in casestudy information
I’m not planning onpredicting the likelihood of anything
I don’t need to measure delta
Is survey
the right
research
method?

11. Mutually exclusive and exhaustive
§ No overlap between answers
§ All possible answers
No double-barreled questions
§ Wrong: “Is your work permanent or casual? “
§ Right: “What is your opinion of our response time?”
Focus on opinions or beliefs of participants
§ Wrong: “How usable is our Web site?”
§ Right: “What are users’ opinions of the site’s look and feel?”
Close-ended is best
§ Cuts down on analysis time
Demographic questions at the end
Survey
Question
Do’s

12. Survey
Question
Don’ts
Inapplicable: does not apply to all respondents
§ How many years can a worker of average health be employed in your trade?
Over-demanding: e.g., recall of time-consuming details
§ What are the general physical, intellectual and moral conditions of life of
the working men and women employed in your trade?
Ambiguous: meaning must be clear to all respondents
§ Describe wage increases during so-called prosperity periods.
Leading
§ Does your employer or his representative resort to trickery in order to
defraud you of your earnings?
Loaded
§ Have you ever known any rank and file workers who could retire from
employment at the age of 50 and live on the money earned by them as wage
workers.

13. Probability
Sampling
Simple random sampling: Choose the people to survey randomly without any
order or preference, like taking an email list and choosing 10% of them in no
specific order and without any preference.
More scientific.Randomnessensureshigher chances of
representationinthetargetpopulation.
Systematic Random Sampling: More organized version of random sampling,
where you take an email list and choose, say, every 3rd on the list.
Stratified Sampling: this is when you first divide the target population into
smaller groups, like separating email addresses according to how much each one
of them earns or their gender or education level. Then in each group you
systematically choose whom to talk to, for example every 5th person on the list.
Cluster Sampling: here you divide the target population into groups that seem
natural, which almost always means based on geographical spread. After having
done that, you can perform a simple random sampling within each cluster.

14. Non-
Probability
Sampling
Judgment: this is when the person conducting the survey uses personal
judgment or experience to choose the people to interview, like standing outside a
supermarket and choosing to talk only to the people that look respectable.
Often done for the convenience, subjects who are easily accessible or require the
least expense.Use this sampling when you not looking for the exact truth, but just
a decent approximation of it. Useful asa way of testing or gauging the marketin
a very simple way.
Quota: Cut the group you want to talk to into smaller groups according to gender,
income, or some other criteria, just like in stratified sampling. Then you choose
whom to talk to according to your judgment of which ones are more
representative, or simply who is easier and least costly to reach.
Snowball: Asking people whom you have just surveyed to refer you to other
related people. Economical way of sampling a group that is difficult to locate, like
people with a certain illness or uncommon hobby, preference, or interest, as those
people will usually have a network.

15. Is survey a
numbers game?
Your subject population informs the game.
If the population is large but homogenous, your sample can be small. If the population
is very diverse, you need a larger sample.
So it’s not just about the absolute number of people surveyed. What's more important
is the mapping of the target population correctly, and then sampling accordingly.

16. primer
brass tacks
case study

17. #EpicFail
“That’s not what I heard.”

18. 1948 US
Election
Survey George Gallup introduced the world to survey sampling by predicting
the result of a previous election.
Gallup was a respected figure in the field. He not only pioneered polling
firms but advanced social innovation – feeling the pulse of the public
on issues that affected them.
Telephone polls held before the elections consistently showed Dewey
as the leading candidate. Other polling firms such as Crossley and
Roper predicted the same result.
Insteadof facilitatinga transitionin government,the polls had
misledthe presidentialcandidatesandall other politicians,the
Washingtonbureaucrats,themedia,and the public.

19. Critical
Errors
Polling firms ended their polling two weeks before Election Day. Therefore,
they missed the importance of measuring preferences just before the
election.
Pollsters had no certain way of deciding who would stay home on Election
Day and who would go and cast a vote.
The polling firms did not put much premium on the undecided voters, and
therefore miscalculated their margins of error.
The pollsters’ use of quota sampling that biased responses towards
Truman.
The Social Science ResearchCouncil committeeconcludedthat the
1948 polls werenot up to the standardsofscience.

20. Legacy
It brought the academe and social scientists into a field they previously did
not pay attention to. This brought rigor into survey as a method for
gathering information on public opinion.
Survey design became more sensitive to errors of sampling and brought
attention to the importance of designing questions that elicit more
accurate indicators of their target population.
The academe developed a more strategic method for survey sampling –
probability sampling, to lessen errors of underrepresentation or
misrepresentation.
The Gallup Poll of the 1948 US PresidentialElectionsprovedto be a
watershedeventthat revolutionizedthescience of survey.

21. Up to this point, we in the field of public
opinion research have had tocarry the ball
ourselves with little help,but withplenty of
criticism, from the social scientists.
(Gallup 1948)
The moreI worked on this report, the moreI felt the debt that weowe to these men who have been
willing to risk their own money in trying to learnsomething about American political behavior…
And wehave a responsibility, wein the universities, to do our best to help improve these
techniques…. BecauseI believe in this work as aninstrument of democracy. (Stouffer 1948.p 214.)

22. A famous sampling mistake
#TurningPoint
“…predicting behavior on the basisof knowledge
of attitudeis a very hazardousventure.”
(Dinerman.1948)

23. If weknew what it was we were
doing, it would not be called
research,would it?

24. Babbie, Earl. (2011) The Practice of Social Research. Belmont CA: Wadsworth.
Creative Research Systems (2009). Survey design: How to beginyour survey project.
Fowler, F. J.,Jr. (2002). Designing questions to be good measures. In InF. J. Fowler, Survey research methods
(3rd ed.) (pp. 76-103). Thousand Oaks, CA: Sage.
Groves, Robert M.; Fowler, Floyd J.;Couper, Mick P.; Lepkowski, James M.; Singer,Eleanor & Tourangeau,
Roger (2004). Survey methodology. Hoboken,NJ: JohnWiley & Sons.
Howitt & Cramer (2011):
Chapter 33 - The analysis of a questionnaire / survey project (pp. 407-415)
Lindloff, Thomas & Taylor, Bryan. (2011). Qualitative Communication Research Methods. Thousand Oaks,
CA: Sage.
Nardi, P. (2006). Developing a questionnaire (Ch. 4). In Doing surveyresearch: Aguide to quantitative
methods (2nd. ed.) (pp. 66-106). Boston,MA: Pearson.
Trochim, W. M. K. (2006). Survey research.
Zetterberg, Hans (2004). US Election 1948: The First Great Controversy aboutPolls, Media, and Social Science.
Paper presented at the WAPOR regional conference on"Elections, News Media and Public Opinion"in
Pamplona, Spain,November, 24-26, 2004. http://www.zetterberg.org/Lectures/l041115.htm
References