Reducing Bias in Public Opinion Polls - By Autumn Carter Presentation Delivered at Stanford University, Summer Research College, August 2009 Presentation of some of my results from my research. Over the course of the project, I analyzed 7 survey experiments across 9 unique vendors, who conducted the survey via phone or the internet and using either a probability sample or a non-probability sample. In all, about 10000 unique individuals were surveyed. This presentation highlights the question bias of Agree/Disagree type questions and the response bias associated with the Acquiescence Effect. I conducted my research through Stanford\'s Political Psychology Research Group. pprg.stanford.edu

1. Reducing Bias in Public Opinion Polls Autumn Carter Professor Jon Krosnick Summer Research College 2009

2. Satisfy Your Constituents... Or Suffer the Consequences.

3. So, how do we effectively determine what Americans are feeling?

4. Public Opinion Polling

5. But Are the Polls Accurate? ? =

6. Critics Don't Think So... “ The dirty little secret of the polling industry is that, all too often, its findings are based on flawed methodology and dubious assumptions.” -Arianna Huffington, Huffington Post Founder

7. Sample Public Opinion Question Form 1 Individuals are more to blame than social conditions for crime and lawlessness in this country. Agree Disagree

8. Sample Public Opinion Question Form 1 Individuals are more to blame than social conditions for crime and lawlessness in this country. Agree: 59.6% Disagree: 40.4%

9. Sample Public Opinion Question Form 1 Individuals are more to blame than social conditions for crime and lawlessness in this country. Agree: 59.6% Disagree: 40.4% Form 2 Social conditions are more to blame than individuals for crime and lawlessness in this country. Agree Disagree

10. Sample Public Opinion Question Form 1 Individuals are more to blame than social conditions for crime and lawlessness in this country. Agree: 59.6% Disagree: 40.4% Form 2 Social conditions are more to blame than individuals for crime and lawlessness in this country. Agree: 56.8% Disagree: 43.2% Acquiescence Effect: 16.4%

11. What is Acquiescence? “ ...a presumed tendency for respondents to agree with attitude statements presented to them.” -Schuman and Presser (1981)

13. Can we reduce acquiescence by administering the survey via the Internet instead of by telephone?

17. A Classic Acquiescence Experiment Form 1 Individuals are more to blame than social conditions for crime and lawlessness in this country. Agree Disagree Form 2 Social conditions are more to blame than individuals for crime and lawlessness in this country. Agree Disagree

18. Replication in a National Survey Telephone survey of a representative sample: Internet survey of a representative sample: * p < 0.05 + p <0.10 11.5%* 7.1%+ TelephoneXInternet Interaction: n.s.

19. What about Internet surveys of people who are not representative of the national population?

21. You've all seen these...

23. Acquiescence Bias * * * * * † * † † *

24. If anything, non-probability samples increase acquiescence. So we should be cautious when interpreting these types of surveys.

25. What if we just didn't ask questions that ask respondents to agree or disagree with a statement?

26. Balanced Forced-Choice Questions Form 1 Which do you think is more responsible for crime and lawlessness in this country: individuals or social conditions ? Individuals Social Conditions Form 2 Which do you think is more responsible for crime and lawlessness in this country: social conditions or individuals ? Social Conditions Individuals

27. Response Order Effects ns ns ns ns ns ns ns ns ns