This document summarizes challenges in conducting natural experiments to measure the causal impact of online advertising at scale. It discusses 5 key challenges: 1) the longitudinal nature of online advertising data, 2) user fragmentation across cookies, 3) lack of predetermined study periods, 4) validating causal models, and 5) dealing with billions of events and terabytes of raw data. Overcoming these challenges requires both engineering and modeling approaches. The document provides examples of how to address issues like seasonality, define effect windows, and implement negative control tests when analyzing large-scale online advertising data.
1. CODE@MIT – OCTOBER 16 2015
Lessons learned in display advertising
Natural experiments at scale
Robert Moakler – (rmoakler@stern.nyu.edu)
Ekaterina Eliseeva – (keliseeva@integralads.com)
Kiril Tsemekhman – (kiril@integralads.com)
CODE@MIT 2015
2. CODE@MIT – OCTOBER 16 2015
The $100+ billion question!
Does online advertising really work?
$104.57
$120.05
$140.15
$160.18
$178.45
$196.05
$213.89Digital ad spending!
% change!
2012 2013 2014 2015 2016 2017 2018!
Source: www.emarketer.com, “Global Ad Spending Growth to Double This Year”
20.4%
14.8% 16.7% 14.3% 11.4% 9.9% 9.1%
3. CODE@MIT – OCTOBER 16 2015
The $100+ billion question!
Does online advertising really work?
Do online ads cause you to take some action?
4. CODE@MIT – OCTOBER 16 2015
The usual approach!
Randomized experiments and A/B tests are great!
Campaign AdPSA
5. CODE@MIT – OCTOBER 16 2015
The usual approach!
But sometimes …
RIGHT
WRONG
Randomized experiments and A/B tests are great!
Campaign AdPSA
6. CODE@MIT – OCTOBER 16 2015
Natural experiments!
• Consider the typical setup for the ad serving process
Confounding!
W
User
features
A
Served
ads
Y
Convert
7. CODE@MIT – OCTOBER 16 2015
Natural experiments!
• Consider the typical setup for the ad serving process
• Introduce a mediating variable
W
User
features
A
Served
ads
Y
Convert
M
Mediator
W’
Residual
Confounders
8. CODE@MIT – OCTOBER 16 2015
Natural experiments!
• Consider the typical setup for the ad serving process
• Introduce a mediating variable
– Viewability
W
User
features
A
Served
ads
Y
Convert
V
Viewable
ad
W’
Residual
Confounders
9. CODE@MIT – OCTOBER 16 2015
Ad viewability!
Horizontal location (px) Proportion of ads
Verticallocation(px)
Addensity
10. CODE@MIT – OCTOBER 16 2015
Running in the wild!
• Natural experiments aren’t always clean or easy
• We will discuss five problems that we have run into and some
solutions for dealing with them
11. CODE@MIT – OCTOBER 16 2015
An online advertising campaign!
• Our data structure
Analysis window
Viewable ad
Unviewable ad
Conversion
Web activity
Our users
12. CODE@MIT – OCTOBER 16 2015
Longitudinal data!
• Monitoring
– Most online advertising campaigns run continually
– We are constantly monitoring many campaigns at the event level
• Running an intermediary analysis
– Data is subject to left truncation and right censoring
– We need to account for our residual confounders, W’
• Use survival analysis
– Cox Proportional Hazards (CPH) model
13. CODE@MIT – OCTOBER 16 2015
User fragmentation and study period!
• In reality, our users are defined by cookies.
– However, people do not just have one cookie!
Viewable ad
Unviewable ad
Conversion
Web activity
Sarah
Cookie 1 Cookie 2 Cookie 3
Bob
Cookie 1 Cookie 2
Analysis window
Cookie 3
14. CODE@MIT – OCTOBER 16 2015
User fragmentation and study period!
• In reality, our users are defined by cookies.
– However, people do not just have one cookie!
• Some methods we use to account for this
– We define an effect period of 1 week
• Seasonality has a major impact
• Users are selected through iterative simulation and research
• Incremental causal estimates level off after a single week
15. CODE@MIT – OCTOBER 16 2015
Validation!
• How do we know our causal models give reasonable estimates?
• Use an array of negative control tests
– Use the impressions of one campaign to predict an unrelated conversion
W
User
features
A
Served
ads
Y
Convert
W’
Residual
Confounders
Y
Unrelated
Event
-
V
Viewable
ad
16. CODE@MIT – OCTOBER 16 2015
Running at scale!
• Converting our data into something analyzable is a challenge
…
Raw daily logs
Billions of events
HDFS scalable
cluster storage
Hadoop
People browse
the web.
Advertising events
turn into billions of
daily events.
Raw data is moved
to scalable storage
optimized for our
experimental setup.
Users are subsampled
and negative controls
are chosen in parallel.
Reports are run in
parallel using
stripped down R
libraries.
Iterative process of
simulation and
research.
17. CODE@MIT – OCTOBER 16 2015
Summary!
• Mediators and natural experiments may already exist in your data
• Running a natural experiment at scale is not straight forward, because
1. The longitudinal nature of the data
2. Users can become highly fragmented
3. No predetermined start and end dates
4. Validation of causal models
5. Billions of events and terabytes of raw data
• Equal parts engineering and modeling
• We explored online advertising, but this setup can apply to a wide
variety of industries
18. CODE@MIT – OCTOBER 16 2015
Thanks!
Robert Moakler – (rmoakler@stern.nyu.edu)
Ekaterina Eliseeva – (keliseeva@integralads.com)
Kiril Tsemekhman – (kiril@integralads.com)
Grab this deck @ bit.ly/natural-experiments-at-scale
19. CODE@MIT – OCTOBER 16 2015
Acknowledgments!
Integral Ad Science
Ekaterina Eliseeva
Kiril Tsemekhman
Ana Calabrese
Gijs Joost Brouwer
Sergei Izrailev
NYU Stern
Foster Provost
Amazon, Inc.
Daniel Hill
20. CODE@MIT – OCTOBER 16 2015
References!
Chan, D., Ge, R., Gershony, O., Hesterberg, T., & Lambert, D. (2010, July). Evaluating online ad
campaigns in a pipeline: causal models at scale. In Proceedings of the 16th ACM SIGKDD
international conference on Knowledge discovery and data mining (pp. 7-16). ACM.
Dalessandro, B., Perlich, C., Stitelman, O., & Provost, F. (2012, August). Causally motivated attribution
for online advertising. In Proceedings of the Sixth International Workshop on Data Mining for Online
Advertising and Internet Economy (p. 7). ACM.
Hill, D. N., Moakler, R., Hubbard, A. E., Tsemekhman, V., Provost, F., & Tsemekhman, K. (2015, August).
Measuring Causal Impact of Online Actions via Natural Experiments: Application to Display
Advertising. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge
Discovery and Data Mining (pp. 1839-1847). ACM.
Johnson, G. A., Lewis, R. A., Nubbemeyer, E. I. (2015, October). Ghost Ads: Improving the Economics
of Measuring Ad Effectiveness. Available on SSRN: ssrn.com/abstract=2620078
Klein, J. P., & Moeschberger, M. L. (2003). Survival analysis: techniques for censored and truncated
data. Springer Science & Business Media.
Pearl, J. (2009). Causality. Cambridge university press.