Here is my presentation from the 2020 AAPOR 75th Annual Conference

1. Evolution of the
American Trends Panel
Nick Bertoni Courtney Kennedy
Panel Manager Director of Survey Research
AAPOR 75th Annual Conference

2. 2
Pew Research Center’s path to an online panel
• For 20 years, we were an RDD telephone polling organization
• We created our online “American Trends Panel” in 2014 and just
finished fielding our 68th panel survey
• This talk will draw on our experiences with the ATP and highlight
some important changes over the years

3. 3
• National probability-based sample of adults who take surveys for us
• Created in 2014
• 100% online survey administration
• Non-internet households are provided tablets and data plan
• We conduct 1 to 2 surveys per month, either 8- or 15-min long
• $5 to $20 for each survey
• Primary platform for Center domestic data collection
American Trends Panel Overview

4. BUILDING THE ATP
June 11, 2020 4

5. 5
ATP Establishment, Enhancement and
Expansion

6. 6
ATP Establishment, Enhancement and
Expansion

7. 7
ATP Establishment, Enhancement and
Expansion

8. 8
ATP Establishment, Enhancement and
Expansion

9. 9
ATP Establishment, Enhancement and
Expansion

10. 10
ATP Establishment, Enhancement and
Expansion

11. RECRUITMENT MODE – FROM RDD TO ABS
June 11, 2020 11

12. 12
Changing recruitment from RDD to ABS
• ATP was initially recruited at the end of RDD polls in 2014 and 2015
• One limitation of this approach is that the cumulative response rate to
panel surveys was 2%
• In 2017 we had a dedicated RDD recruitment survey
• Starting in 2018 we switched to ABS recruitment
10% Response rate to recruitment surveys
X 50% Agreement to join panel among recruitment respondents
X 55% Panelists active at the start of the wave
X 77% Wave-level response rate
2% Cumulative response rate for ATP wave

13. 13
Changing recruitment from RDD to ABS
• When we switched to recruiting via ABS, we saw notable
improvement
Recruitment
Recruitmen
t survey
AAPOR
RR3
% of recruitment
survey respondents
agreeing to join the
panel
2017 RDD
Design
10% 50%

14. 14
Changing recruitment from RDD to ABS
• When we switched to recruiting via ABS, we saw notable
improvement
Recruitment
Recruitmen
t survey
AAPOR
RR3
% of recruitment
survey respondents
agreeing to join the
panel
2017 RDD
Design
10% 50%
2018 ABS Design 12% 94%

15. 15
Changing recruitment from RDD to ABS
• When we switched to recruiting via ABS, we saw notable
improvement
Lower response rate and panel agreement rate in 2019 attributed to targeted recruitment of
Hispanic and non-internet households
Recruitment
Recruitmen
t survey
AAPOR
RR3
% of recruitment
survey respondents
agreeing to join the
panel
2017 RDD
Design
10% 50%
2018 ABS Design 12% 94%
2019 ABS Design 11% 80%

16. 16
Changing recruitment from RDD to ABS
• Now our cumulative response rate looks like this
10% 11% Response rate to recruitment surveys
X 50% 79% Agreement to join panel among recruitment respondents
X 55% 85% Panelists active at the start of the wave
X 77% 77% Wave-level response rate
2% 6% Cumulative response rate for ATP wave

17. 17
Changing recruitment from RDD to ABS
• Now our cumulative response rate looks like this
• To be sure, it’s still low.
• And, yes, Montaquila and Brick saw this pattern 10 years ago in
NHES!
• But as the ABS-recruits become a larger and larger share of the
panel and as we refine the ABS protocol and improve the recruitment
response rate…
10% 11% Response rate to recruitment surveys
X 50% 79% Agreement to join panel among recruitment respondents
X 55% 85% Panelists active at the start of the wave
X 77% 77% Wave-level response rate
2% 6% Cumulative response rate for ATP wave

18. 18
Changing recruitment from RDD to ABS
• In time it may be possible to get a cumulative response rate more
likely 15% even without an in-person recruitment stage
• This would provide a clearer, more compelling distinction between
probability-based versus opt-in online polls
Even more importantly, ABS yielded a more representative sample
10% 11% 25% Response rate to recruitment surveys
X 50% 79% 94% Agreement to join panel among recruitment
respondents
X 55% 85% 85% Panelists active at the start of the wave
X 77% 77% 77% Wave-level response rate
2% 6% 15% Aspirational cumulative response rate

19. 19
Changing recruitment from RDD to ABS
• The ABS recruited sample was closer to population benchmarks on
virtually every dimension we measured
• It’s important to note that ABS oversampled based on commercial
flags for black, Hispanic, and age 18-24
(%) unweighted profile of ABS-recruited versus existing, RDD-recruited panelists

20. INTERVIEW MODE – FROM MAIL TO WEB
June 11, 2020 20

21. WHY USE MAIL MODE TO BEGIN WITH?
June 11, 2020 21

22. June 11, 2020 22
Coverage
• Not everyone uses the internet
• Those who do not use the internet are different
than those who do
• Prefer to include non-internet households

23. 23
Covering those without internet access
• There are three main approaches:
 Provide the technology and interview online
 Interview by telephone
 Interview by mail
• Selecting the optimal approach is a multi-dimensional problem
• We see at least six important dimensions

24. 24
Covering those without internet access
Dimension
Provide
technology
Interview by
telephone
Interview by
mail
Cost Very expensive
Less
expensive(?)
Relatively
inexpensive

25. 25
Covering those without internet access
Dimension
Provide
technology
Interview by
telephone
Interview by
mail
Cost Very expensive
Less
expensive(?)
Relatively
inexpensive
Representation of
non-internet adults
Weaker with
potential to
change panelists
Strong Strong

26. 26
Covering those without internet access
Dimension
Provide
technology
Interview by
telephone
Interview by
mail
Cost Very expensive
Less
expensive(?)
Relatively
inexpensive
Representation of
non-internet adults
Weaker with
potential to
change panelists
Strong Strong
Mode effects risk? None Yes Minimal

27. 27
Covering those without internet access
Dimension
Provide
technology
Interview by
telephone
Interview by
mail
Cost Very expensive
Less
expensive(?)
Relatively
inexpensive
Representation of
non-internet adults
Weaker with
potential to
change panelists
Strong Strong
Mode effects risk? None Yes Minimal
Possible to do
timely surveys?
Yes Yes No

28. 28
Covering those without internet access
Dimension
Provide
technology
Interview by
telephone
Interview by
mail
Cost Very expensive
Less
expensive(?)
Relatively
inexpensive
Representation of
non-internet adults
Weaker with
potential to
change panelists
Strong Strong
Mode effects risk? None Yes Minimal
Possible to do
timely surveys?
Yes Yes No
Can do extensive
skips, fills,
rotations?
Yes Yes No

29. 29
Covering those without internet access
Dimension
Provide
technology
Interview by
telephone
Interview by
mail
Cost Very expensive
Less
expensive(?)
Relatively
inexpensive
Representation of
non-internet adults
Weaker with
potential to
change panelists
Strong Strong
Mode effects risk? None Yes Minimal
Possible to do
timely surveys?
Yes Yes No
Can do extensive
skips, fills,
rotations?
Yes Yes No
Added risk of
programming error
No Yes Yes

30. 30
Covering those without internet access
• Each solution has its limitations
• On the ATP, we started with the mail approach but switched in 2016
to providing the technology
• For us concerns about timeliness and mode effects are greater than
concerns about cost and somewhat reduced representation of the
non-internet population
• But this remains a difficult issue

31. CONVERTING THE PANELISTS – LOGISTICS AND
RESULTS
June 11, 2020 31

32. June 11, 2020 32
Conversion Protocol
• Advance letter to current mail mode panelists (N=574)
• Conversion call
• Inform panelists of the conversion from mail to web
• Determine internet access and/or device needs
• Ask to convert
• Obtain email if available, provide tablet with internet connection if
necessary
• Tablet gets configured and shipped by vendor
• Follow up call after panelists receive tablet

33. June 11, 2020 33
Conversion Stats
• Initial results of the Conversion Call were encouraging
Mail to Web Conversion Call
% of mail panelists that…
Percent
Agreed to convert 66
Refused to convert 25
Non-contact 8
N 574
Source: Mail to Web Conversion Call survey conducted 4/11/2016-
1/4/2017. Results do not add up to 100 due to rounding.
PEW RESEARCH CENTER

34. June 11, 2020 34
Conversion Stats
• If only it ended at the Conversion Call
Mail to Web Conversion
% of mail panelists that…
Percent
Refused after initially
agreeing to convert 3
Refused after receiving the
tablet 12
Received the tablet but never
participated 9
Participated at first but then
disappeared 2
Note: Percent shown is of total attempted to convert (N=574)
PEW RESEARCH CENTER

35. June 11, 2020 35
Conversion Stats
• When all of the dust settled
Mail to Web Conversion
% of mail panelists that…
Percent
Agreed to convert 41
Refused to convert 59
N 574
Source: Mail to Web Conversion.
PEW RESEARCH CENTER

36. June 11, 2020 36
Composition of the Panel:
No Observed Differences
• We observed no statistically significant difference in sample
composition of the panel pre- and post-conversion for several key
areas
Sex Political ideology
Race/Ethnicity Follow politics
Party ID Follow news
Voter registration Religious attendance

37. June 11, 2020 37
Composition of the Panel:
Internet Use
• A smaller proportion of the panel are non-internet users after the
conversion
6
2
0
1
2
3
4
5
6
7
8
9
10
Non-internet user
Before conversion After conversion
%
(N=4,563) (N=4,248)

38. June 11, 2020 38
Composition of the Panel:
Education
• A smaller proportion of the panel has HS or less education after the
conversion
%
18
33
49
15
32
52
0
10
20
30
40
50
60
70
80
HS or less Some college College+
Before conversion After conversion
(N=4,563) (N=4,248)

39. June 11, 2020 39
Contact Information
Nick Bertoni
Panel Manager
nbertoni@pewresearch.org