In April 2020, GitHub announced a new Free for Teams plan. Behind the scenes, the engineering team was also setting up an experiment pipeline and an integration with Optimizely. In this session, we will take a peek at the process of setting up the integration, learning about the behavior of this new Free for Teams customer segment, and the next steps for this experiment pipeline.
3. ● Director of Software Engineering
UI Platform
● Korean adoptee immigrant
● Soccer, yoga, skiing
● Current book: The New Jim Crow
● Ambivert: 75% introvert, 25% extrovert
● Pronouns: he/him
About me
@philip_bremer
7. New orgs and teams
Global Nav Repo CTAsOrg Creation
8. Global nav experiment
Problem: Users can’t easily find their organization(s)
after creating or joining
Hypothesis: Adding a link to “Your organizations” in
user’s global dropdown menu will improve discovery
Results:
● Success!
● +134.46% improvement
● Oops! We spent a ton of impressions
● ~10% of total impression budget
9. ● Misunderstood the difference
between bucketing vs
activation vs tracking
● High traffic for global nav
● The experiment baked too long
How did we spend
so many
impressions?
10. Track
Log notable events
to calculate impact
of variation
Bucketing, activation, and tracking
Bucketing
Assign user
to variation
Activation
Log impression
when user sees
the experiment
11. ● Anonymous sessions are
read-heavy and repetitive
● Cache anonymous sessions to
reduce load on our application
● Goal: serve page for a subset of
anonymous sessions without
disrupting the benefits of our cache
● Goal: serve page with no negative
impact to our page load budget
Logged out experiments
12. Performance & DX
● Robust datafile management
● More internal API guardrails for
activate vs get_variation
● Improved logging and testing
across environments
Experimentation culture
● Empower feature teams to
experiment with evidence-
based delivery
Up Next?