Dina Neiger's presentation on SRC's panel, Life in Australia.

1. A subsidiary of:
Weighting a probability online panel with
multiple waves of recruitment
JSM
August 2020

2. www.srcentre.com.au
Authors
Dina Neiger
Chief Statistician, the Social Research Centre
Campus Visitor, ANU Centre for Social Research and Methods
Andrew Ward
Principal Statistician, the Social Research Centre
Jack Barton
Analyst, Statistical and Survey Methods, the Social Research Centre
The Social Research Centre is a for-profit wholly owned subsidiary of the
Australian National University
2

3. www.srcentre.com.au
Acknowledgements
Darren Pennay
Founder & Senior Adviser, the Social Research Centre
Campus Visitor, ANU Centre for Social Research and Methods
Adjunct Professor, Institute for Social Science Research, the University of Queensland
Ben Phillips
Chief Survey Methodologist, the Social Research Centre
Campus Visitor, ANU Centre for Social Research and Methods
Paul J. Lavrakas
Senior Methodological Adviser, the Social Research Centre
Charles Dove
Research Consultant, the Social Research Centre
Panel Manager, Life in Australia™
3

4. www.srcentre.com.au
Introducing Life in Australia™
4

5. www.srcentre.com.au
Australia’s only probability-based online panel
Created by the Social Research Centre in 2016
Ø Non-probability panels in Australia similar to US/Europe, relative to
probability samples, they are
o more biased
o more variable; and
o bias may be increased via traditional weighting approaches.
Ø Quality online social research
Self-funded
Ø Designed for 2,000 completed surveys per wave
Ø From 2019, expanded to provide 3,000 or more completes for a full wave
Ø Primarily split-panel waves with different clients
5

6. www.srcentre.com.au
Substantive areas measured
6
Australia’s place in
the world
Crime and justice Attitudes to
disability, autism
Political views Sun protection
attitudes /
behaviours
Social cohesion Communications
use
Awareness of
alcohol harms
Image-based abuse Cyber crime
Data privacy and
linkage
Attitudes about
immigration
Fertility Gambling Elections
Clients to date: government, academic and not-for-profit

7. www.srcentre.com.au
Operating model
Waves most months
Ø Typically whole-of-panel with occasional sub-panel selections
Ø Main client for most waves
Ø Add-on omnibus modules for secondary clients
Incentives
Ø 10 AUD base incentive for waves under 20 minutes
Ø 5 AUD additional for every extra 5 minutes
Use for primary data collection and calibration
Ø Rich profile for all new recruits including: range of primary and secondary
demographic variables, risk behaviours, health outcomes, Big5
personality inventory and social desirability indicators
7

8. www.srcentre.com.au
Evolution of recruitment
8

9. www.srcentre.com.au
Panel establishment
• Single-frame design weights where
probability of selection is
•𝑝 =
!!!""
#!!$%!!
+
!"#&'
#"#
• where 𝑆"" is number of landline
respondents, 𝐿𝐿 is 0,1 indicator of having
landline, 𝑈"" is size of landline universe,
𝐴𝐷"" is number of adults in HH, 𝑆&' is
number of mobile respondents, 𝑀𝑃 is 0,1
indicator of having mobile and 𝑈&' is size of
mobile universe
2016recruitment
• Dual-frame RDD
• Landline (n=911)
• Mobile (n=2,411)
2016designweights
9

10. www.srcentre.com.au
Evolution of Life in Australia™ recruitment 2018 refresh
2016recruitment
• Dual-frame RDD
• Landline (n=911)
• Mobile (n=2,411)
• Panel size n=3,322
2018
replenishment
• Mobile RDD (n=267)
• Only age 54 and
under
• Quotas on university
education
• Panel size n=2,839
10

11. www.srcentre.com.au
Evolution of Life in Australia™ recruitment - 2019 refresh
2016recruitment
• Dual-frame RDD
• Landline
(n=911)
• Mobile
(n=2,411)
• Panel size
n=3,322 2018replenishment
• Mobile RDD
(n=267)
• Only age 54 and
under
• Quotas on
university
education
• Panel size
n=2,839
2019expansion
• ABS (n=1,810)
• Area-level
stratification by
SES and age
• Expand to 3,000
completes/ wave
• Panel size
n=4,025
11

12. www.srcentre.com.au
Evolution of Life in Australia™ recruitment
2016recruitment
• Dual-frame RDD
• Landline
(n=911)
• Mobile
(n=2,411)
• Panel size
n=3,322
2018replenishment
• Mobile RDD
(n=267)
• Only age 54 and
under
• Quotas on
university
education
• Panel size
n=2,839
2019expansion
• ABS (n=1,810)
• Area-level
stratification by
SES and age
• Expand to 3,000
completes/ wave
• Panel size
n=4,025
12
non-random attrition, increasingly unknowable probabilities of selection
decreasing weighting efficiency, increasing weighting complexity

13. www.srcentre.com.au
Weighting approaches
13

14. www.srcentre.com.au
Derive exact probabilities of selection
Ø Potential to derive exact probabilities of selection either
o Assuming single frame or
o Treating as composite frames
Ø Challenges
o Future-proofing, increasingly complicated
o Large weight differentials leading to inefficiencies
o Attempted for 2018 – very low efficiency
o Not feasible for 2019 – too many non random elements
Ø References
o Kish, Leslie. 1965. Survey Sampling. New York: Wiley
o Lohr, Sharon L. 2011. ‘Alternative Survey Sample Designs: Sampling with Multiple
Overlapping Frames.’ Survey Methodology 37(2):197–213.
14

15. www.srcentre.com.au
Approximate probabilities of selection
Ø Propensity response model to match new cases (full coverage and
design weight applied) and continuing cases to approximate selection
weight
Ø Challenges
o Disproportionate selection of new cases (reference sample)
o Lack of common support for parts of the population, large weight differentials leading to
inefficiency
Ø References
o Rosenbaum, P.R. and D.B. Rubin (1983) The central role of the propensity score in
observational studies for causal effects. Biometrika 70(1), 41-55.
o Valliant, R., J. Dever, and F. Kreuter (2013). Practical Tools for Designing and Weighting
Survey Samples. New York: Springer.
o Bai, H., & Clark, M. (2019). Propensity score methods and Applications. Thousand Oaks,
CA: SAGE Publications, Inc.
15

16. www.srcentre.com.au
Propensity response model
16
A range of primary and secondary demographic variables, risk behaviours,
health outcomes, Big5 personality inventory and social desirability indicators
Propensity classes using stepwise logistic regression
Bias is measured as an absolute average error (difference) from the high
quality (official statistics) benchmarks for the modal response.
• Unweighted bias = 4.2
With propensity selection weights:
• Bias = 3.6
• Weighting Efficiency = 68.9%

17. www.srcentre.com.au
Doubly-robust approach
Ø Combine
o Approximate probabilities of selection
o Post-stratification (GREG)
Ø Challenges
o Calculation of probabilities of selection
o Model choice
o Further reduction in efficiency
Ø References
o Valliant, R. (2020) Comparing Alternatives for Estimation from Nonprobability Samples.
Journal of Survey Statistics and Methodology (2020) 8, 231–263
17

18. www.srcentre.com.au
Doubly-robust approach
Many combinations of variables (2,000 solutions for illustration):
Max weighting efficiency 69.1% for Bias < Unweighted Bias (4.1)
Minimum bias* 2.3 (Weighting efficiency 37.0%)
Ave weighting efficiency of 55.2% for solutions with bias btw 3. 5 & 3.7
*Variables used in calibration are excluded from bias measurement
18

19. www.srcentre.com.au
Super-population model
Ø Observed values are treated as if they have been generated by a model
o Combine all samples and
o Predict values for those units of the population that have not been sampled through the
whole panel using auxiliary information from external sources
Ø Challenges
o Model failure
o Tension between including covariates and efficiency
Ø References
o Valliant, R. and Dever, J. A. (2018) Survey Weights: A Step-by-Step Guide to Calculation.
Stata Press, 2018
19

20. www.srcentre.com.au
Comparison results – 2019 refresh panel weights
Approach Weighting
efficiency
Bias Measure
(unweighted
4.2)
Final variables used for
selection weight
Final variables used for
calibration
Approximate
selection weights
68.9% 3.6 Profile variables None
Doubly-robust
approach
Max 69.1%
37.0%
Ave 55.2%
4.1
Min 2.3
Btw 3.5 and 3.7
As above >2000 solutions
Super-population
model
Max 97.4%
53.9%
Ave 77.0%
Goldilocks
Solution:
75.8
4.1
Min 2.6
Btw 3.5 and 3.7
Goldilocks
Solution:
2.9
None >2000 solutions
Goldilocks Solution:
Age, Country of Birth,
Gender, Phone status,
SEIFA, State
20

21. www.srcentre.com.au
Lessons learnt
• Design-based weights are not suited for a long-term panel with multiple
refreshes
• Model-based approach is
• Simpler to implement
• Simpler to explain
• More efficient and less biased
• Universally applicable independent of the sample selection
mechanism
21

22. * PO Box 13328
Law Courts Victoria 8010
Thank you
( 03 9236 8500
A subsidiary of:
dina.neiger@srcentre.com.au
22