Wojciech Galuba, Decision Tools Lead, Facebook Experimentation is a valuable tool for supporting product decisions, iterating on features and gaining actionable insights into people's behavior. In this session Wojciech Galbua, Data Scientist at Facebook, presents an overview of Facebook's experimentation framework and how it is used to make day-to-day data-driven decisions at global scale. The talk focuses on the challenges of building and scaling the analytics infrastructure, designing the tools for ease-of-use and ensuring broad adoption of sound experimentation methodologies across all the teams.

1. N=109
Automated
Experimenta5on
at
Scale
Wojciech
Galuba
Decision
Tools
Lead,
Facebook
@wgaluba

3. N=109:
Automated Experimentation at Scale
Wojtek Galuba (wgaluba@fb)
Decision Tools Team Lead
Data Science Infrastructure
Facebook

4. History of Data Science Infra at FB
• Founded April 2012
• A group of data scientists and software engineers
• Experienced first hand the need for better infrastructure
• Need continues to grow
• Team doubled over the past year
• Expect continued rapid growth this year

5. Why do we experiment?

6. Experimentation
Product
changes
Experiment
to study this
Metrics

7. Experiment to:
Catch problems before they arise

8. Experiment to:
Choose between multiple options

9. Experiment to:
Challenge intuitions about product

10. Experiment to:
Not only evaluate ideas
but generate new ones

11. Challenges

12. Many experiments
• Experiments running in parallel
• Modifying many different aspects of the product
• Overlaps are possible and may conflict

13. Many metric dimensions
• Different contexts of user actions
• Thousands of device types
• Geography
• Demographics
• Time
• Enormous space of possible questions

14. Many teams
• Many ways to run an experiment
• Diverse audience for results
• Huge set of results from every experiment
• Many ways to interpret results

15. Experimentation at Facebook

16. An experiment

17. QuickExperiment
Dividepeoplerandomly
color: blue
size: medium"
color: blue"
size: big"
color: green"
size: medium"

18. QuickExperiment
• Centralized experiment management
• Purely config-level: no code pushes to iterate
• Automatic exposure logging

19. PlanOut

20. PlanOut
• Open sourced: http://facebook.github.io/planout/
• Flexible experimental design
• Full, programmatic control over param values

21. Experiment evaluation
Exposures
Metrics
% change from control to test
-1
0
1
2
-2
3
-3
posts
99.9 %99 %95 %Confidence:

22. Assess decision risk
99.9 %99 %95 %Confidence:

23. Lessons learned

24. Computing answers to exponential
number of possible questions
Pre-compute
• low specificity
• low dimensionality
• long-term
Compute on-the-
fly
• high specificity
• high dimensionality
• short-term
A balancing act

25. Tackling many dimensions
Two sets of tools
For exploration For extraction

26. Automated exploration

27. Enforce a lifecycle;
In particular:
clear experiment end dates

28. Why lifecycle policy?
• Unifies methodology across teams
• Prevents tech debt buildup
• Minimizes bad impact on product

29. Ease of rapid iteration;
Safe and scientifically valid iteration

30. Fast, but not too fast
• Novelty effect vs. top engaged users bump
• Understand if waiting helps

31. Ensure mutual exclusion;
Across platforms,
features and infra

32. Why mutual exclusion?
• Fewer experiment conflicts
• Lower metrics variance

33. Exposure log everything
• Measure effects on the exposed only
• Conditioning analyses on the time since last exposure

34. The culture
Experimentation gives focus;
But watch out for tunnel vision!

35. The culture
Cultivate sound practices;
Safe and low-impact experimentation

36. The culture
Educate on data interpretation;
Uniform decision-making
across teams

37. Understanding uncertainty
“Robust misinterpretation of confidence intervals”
Rink Hoekstra et al.
Psychonomic Bulletin & Review
• Only 3% of scientists got
all 6 answers right...
• How do we educate the
users of the tools?

38. The three stages of
experimentation
infrastructure

39. Stage 1: Artisanal
Photo credit: Abhisek

40. Stage 2: Power tools

41. Stage 2: Power tools

42. Stage 3: Industrialized
Photo credit: Steve Jurvetson

43. Conclusions
Empower, but don’t overwhelm

44. Conclusions
Filter and automate,
but maintain broad focus

45. Conclusions
Clean data and powerful tools are great, but
building the right experimentation
culture is equally important

46. N=109
Automated
Experimenta5on
at
Scale
Wojciech
Galuba
Decision
Tools
Lead,
Facebook
@wgaluba