SlideShare utilise les cookies pour améliorer les fonctionnalités et les performances, et également pour vous montrer des publicités pertinentes. Si vous continuez à naviguer sur ce site, vous acceptez l’utilisation de cookies. Consultez nos Conditions d’utilisation et notre Politique de confidentialité.
SlideShare utilise les cookies pour améliorer les fonctionnalités et les performances, et également pour vous montrer des publicités pertinentes. Si vous continuez à naviguer sur ce site, vous acceptez l’utilisation de cookies. Consultez notre Politique de confidentialité et nos Conditions d’utilisation pour en savoir plus.
+ 3,000 Photographers worldwide
+ Over 100k listings photographed
+ Almost 2 million professional photos
+ Many companies have ofﬂine operations
+ Can optimize these using experiments
We run these all the time too.
If you are curious about on our online experimentation see Jan Overgoor’s tech talk
Before and after won’t work
• Often very little data before professional photos are added
• Seasonality and other confounding factors bias results
Selection bias often impacts analysis
• Listings that opt to get professional photography are not the
same as listings that do not get photography
Without an experiment, we don’t know the causal effect
This is the same reason we need online experiments
01−01 01−15 02−01 02−15 03−01 03−15
e.g. Offered Free Professional Photography
Traditional A/B Testing Online
Let’s run an experiment!
Beware of Cannibalization
The unit of randomization depends on the effect we want to estimate
Local Operations: Market Level Experiment
+ Smaller “long tail” markets < 100 reviewed listings
93 Treatment / 92 Control
Assess impact of operational strategy on market growth
+ Statistically measure the lift due to local ops teams
+ Measuring active listings, hosts, reviewed listings, and
Finding: Local Ops Efforts Have Positive Impact on Growth
Local Ops Kickoﬀ
Case Study: Campos do Jordão, BR
+ Market grew 9x
+ Over 90% of the new listings are from new users
+ Low CPA
+ Primary approach is phone sales
+ Other approaches were less successful
Use qualitative research to understand what happened
Active Listing Growth
Improving listings through outreach
+ Initially not launched as an experiment and found positive impact
+ Launched as an experiment and found neutral impact
+ Don’t need market level approach here!
Use context to improve operations
+ Can investigate heterogeneity in treatment effects with higher N
+ Word of caution: can’t just compare those who were reached
by a call or email to the control (selection bias strikes again)
Additional Offline vs. Online Considerations
+ Opt-in biases
+ You know you are in an experiment (Hawthorne/John Henry effects)
+ Monetary incentives impact external validity, trade-off take-up rate
+ Takes time to adjust to a change
+ Sample size may be limited by ops capacity
+ Stakeholders may be less data-savvy
+ Real people delivering the experiment!
+ Ethical considerations
Always partner with customer support.
+ Controlled experiments are the way to go if you want to make causal inference
+ Use them to optimize operations!
+ Level of randomization - what impact do you want to measure?
+ Compare the right groups - no selection bias
+ Break down results to get the most from the analysis
+ Be practical/ethical - you are dealing with real people here