SlideShare une entreprise Scribd logo
1  sur  13
Télécharger pour lire hors ligne
Building an Experiment
Pipeline for GitHub’s New
Free Team Offering
Philip Bremer
Director, Software Engineering
GITHUB
Building an experiment
pipeline for GitHub’s
Free for Teams plan
GitHub
Presented by Philip Bremer
● Director of Software Engineering

UI Platform 

● Korean adoptee immigrant 

● Soccer, yoga, skiing

● Current book: The New Jim Crow

● Ambivert: 75% introvert, 25% extrovert

● Pronouns: he/him
About me
@philip_bremer
GitHub’s growing product footprint
● Individuals and teams

● Price should not be a barrier

● Free private repos 

● Upgrade path for advanced
features
Free for Teams
Time to build an
experiment
pipeline!
New orgs and teams
Global Nav Repo CTAsOrg Creation
Global nav experiment
Problem: Users can’t easily find their organization(s)
after creating or joining

Hypothesis: Adding a link to “Your organizations” in
user’s global dropdown menu will improve discovery

Results:
● Success! 

● +134.46% improvement

● Oops! We spent a ton of impressions 

● ~10% of total impression budget
● Misunderstood the difference
between bucketing vs
activation vs tracking

● High traffic for global nav

● The experiment baked too long
How did we spend
so many
impressions?
Track
Log notable events

to calculate impact 

of variation
Bucketing, activation, and tracking
Bucketing
Assign user 

to variation
Activation
Log impression 

when user sees

the experiment
● Anonymous sessions are 

read-heavy and repetitive 

● Cache anonymous sessions to
reduce load on our application

● Goal: serve page for a subset of
anonymous sessions without
disrupting the benefits of our cache 

● Goal: serve page with no negative
impact to our page load budget
Logged out experiments
Performance & DX
● Robust datafile management

● More internal API guardrails for
activate vs get_variation

● Improved logging and testing
across environments

Experimentation culture
● Empower feature teams to
experiment with evidence-
based delivery
Up Next?
Thank you!

Contenu connexe

Tendances

Presentation - Test Automation in Digital Transformation - IITPSA SIGIST 2016042
Presentation - Test Automation in Digital Transformation - IITPSA SIGIST 2016042Presentation - Test Automation in Digital Transformation - IITPSA SIGIST 2016042
Presentation - Test Automation in Digital Transformation - IITPSA SIGIST 2016042
Wayne Sinclair
 

Tendances (20)

CCG Company Presentation
CCG Company PresentationCCG Company Presentation
CCG Company Presentation
 
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...
 
DevOPs Transformation Workshop
DevOPs Transformation WorkshopDevOPs Transformation Workshop
DevOPs Transformation Workshop
 
QA Automation - Agile Engineering
QA Automation - Agile Engineering QA Automation - Agile Engineering
QA Automation - Agile Engineering
 
Presentation - Test Automation in Digital Transformation - IITPSA SIGIST 2016042
Presentation - Test Automation in Digital Transformation - IITPSA SIGIST 2016042Presentation - Test Automation in Digital Transformation - IITPSA SIGIST 2016042
Presentation - Test Automation in Digital Transformation - IITPSA SIGIST 2016042
 
DevOps Continuous Integration & Delivery - A Whitepaper by RapidValue
DevOps Continuous Integration & Delivery - A Whitepaper by RapidValueDevOps Continuous Integration & Delivery - A Whitepaper by RapidValue
DevOps Continuous Integration & Delivery - A Whitepaper by RapidValue
 
Do you even need to automate the GUI?
Do you even need to automate the GUI? Do you even need to automate the GUI?
Do you even need to automate the GUI?
 
PWA with Kotlin
PWA with KotlinPWA with Kotlin
PWA with Kotlin
 
Comments provided
Comments providedComments provided
Comments provided
 
Cross Platform Devops with Visual Studio Team Services
Cross Platform Devops with Visual Studio Team ServicesCross Platform Devops with Visual Studio Team Services
Cross Platform Devops with Visual Studio Team Services
 
Story Based Burn Down
Story Based Burn DownStory Based Burn Down
Story Based Burn Down
 
Continuous Delivery Presentation
Continuous Delivery PresentationContinuous Delivery Presentation
Continuous Delivery Presentation
 
GitHub Integration for Orangescrum Cloud Released!
GitHub Integration for Orangescrum Cloud Released!GitHub Integration for Orangescrum Cloud Released!
GitHub Integration for Orangescrum Cloud Released!
 
Language Translations for Custom Scoped Applications in ServiceNow
Language Translations for Custom Scoped Applications in ServiceNowLanguage Translations for Custom Scoped Applications in ServiceNow
Language Translations for Custom Scoped Applications in ServiceNow
 
Web.dev extended : What's new in Web [GDG Taichung]
Web.dev extended : What's new in Web [GDG Taichung]Web.dev extended : What's new in Web [GDG Taichung]
Web.dev extended : What's new in Web [GDG Taichung]
 
Agile Data Warehousing at Telstra, TDWI Melbourne, October 2013
Agile Data Warehousing at Telstra, TDWI Melbourne, October 2013Agile Data Warehousing at Telstra, TDWI Melbourne, October 2013
Agile Data Warehousing at Telstra, TDWI Melbourne, October 2013
 
Continuous Performance - Load testing for developers with gatling @ iSense 2016
Continuous Performance - Load testing for developers with gatling @ iSense 2016Continuous Performance - Load testing for developers with gatling @ iSense 2016
Continuous Performance - Load testing for developers with gatling @ iSense 2016
 
Adopting the Scaled Agile Framework: The Theory and the Practice - Dallas ALN...
Adopting the Scaled Agile Framework: The Theory and the Practice - Dallas ALN...Adopting the Scaled Agile Framework: The Theory and the Practice - Dallas ALN...
Adopting the Scaled Agile Framework: The Theory and the Practice - Dallas ALN...
 
Definition of done training
Definition of done trainingDefinition of done training
Definition of done training
 
Firebase remote config tips & tricks
Firebase remote config tips & tricksFirebase remote config tips & tricks
Firebase remote config tips & tricks
 

Similaire à Building an Experiment Pipeline for GitHub’s New Free Team Offering

HOW TO OPTIMIZE NON-CODING TIME, ORI KEREN, LinearB
HOW TO OPTIMIZE NON-CODING TIME, ORI KEREN, LinearBHOW TO OPTIMIZE NON-CODING TIME, ORI KEREN, LinearB
HOW TO OPTIMIZE NON-CODING TIME, ORI KEREN, LinearB
DevOpsDays Tel Aviv
 
RailsGirls Rotterdam - Github (and Octocats!)
RailsGirls Rotterdam - Github (and Octocats!)RailsGirls Rotterdam - Github (and Octocats!)
RailsGirls Rotterdam - Github (and Octocats!)
Anton Bangratz
 
Introducing Github and Octocats by Tony Bangratz
Introducing Github and Octocats by Tony BangratzIntroducing Github and Octocats by Tony Bangratz
Introducing Github and Octocats by Tony Bangratz
RailsGirls_RTM
 

Similaire à Building an Experiment Pipeline for GitHub’s New Free Team Offering (20)

Github Developer Pack
Github Developer PackGithub Developer Pack
Github Developer Pack
 
github student developer pack with student benifits
github student developer pack with student benifitsgithub student developer pack with student benifits
github student developer pack with student benifits
 
GDSC PCE Hacktoberfest 1.pptx
GDSC PCE Hacktoberfest 1.pptxGDSC PCE Hacktoberfest 1.pptx
GDSC PCE Hacktoberfest 1.pptx
 
Hacktoberfest GDSC Pillai College of Engineering
Hacktoberfest GDSC Pillai College of EngineeringHacktoberfest GDSC Pillai College of Engineering
Hacktoberfest GDSC Pillai College of Engineering
 
Github Actions for CI/CD Setup | MuleSoft Mysore Meetup #5
Github Actions for CI/CD Setup | MuleSoft Mysore Meetup #5Github Actions for CI/CD Setup | MuleSoft Mysore Meetup #5
Github Actions for CI/CD Setup | MuleSoft Mysore Meetup #5
 
Github 101 An Adventurer's Guide To Open Source
Github 101   An Adventurer's Guide To Open SourceGithub 101   An Adventurer's Guide To Open Source
Github 101 An Adventurer's Guide To Open Source
 
HOW TO OPTIMIZE NON-CODING TIME, ORI KEREN, LinearB
HOW TO OPTIMIZE NON-CODING TIME, ORI KEREN, LinearBHOW TO OPTIMIZE NON-CODING TIME, ORI KEREN, LinearB
HOW TO OPTIMIZE NON-CODING TIME, ORI KEREN, LinearB
 
Tool Development Process and Software Engineering
Tool Development Process and Software EngineeringTool Development Process and Software Engineering
Tool Development Process and Software Engineering
 
When JHipster meets Microsoft-JHipster and Microsoft products
When JHipster meets Microsoft-JHipster and Microsoft productsWhen JHipster meets Microsoft-JHipster and Microsoft products
When JHipster meets Microsoft-JHipster and Microsoft products
 
How to use github to evaluate developers
How to use github to evaluate developersHow to use github to evaluate developers
How to use github to evaluate developers
 
Using Github Insight as metric for the Developer collaboration and work metri...
Using Github Insight as metric for the Developer collaboration and work metri...Using Github Insight as metric for the Developer collaboration and work metri...
Using Github Insight as metric for the Developer collaboration and work metri...
 
Overcoming the Fear of Contributing to Open Source
Overcoming the Fear of Contributing to Open SourceOvercoming the Fear of Contributing to Open Source
Overcoming the Fear of Contributing to Open Source
 
Let's talk FOSS!
Let's talk FOSS!Let's talk FOSS!
Let's talk FOSS!
 
Increase the Velocity of Your Software Releases Using GitHub and DeployHub
Increase the Velocity of Your Software Releases Using GitHub and DeployHubIncrease the Velocity of Your Software Releases Using GitHub and DeployHub
Increase the Velocity of Your Software Releases Using GitHub and DeployHub
 
RailsGirls Rotterdam - Github (and Octocats!)
RailsGirls Rotterdam - Github (and Octocats!)RailsGirls Rotterdam - Github (and Octocats!)
RailsGirls Rotterdam - Github (and Octocats!)
 
Introducing Github and Octocats by Tony Bangratz
Introducing Github and Octocats by Tony BangratzIntroducing Github and Octocats by Tony Bangratz
Introducing Github and Octocats by Tony Bangratz
 
Intro to Kubernetes SIG Contributor Experience
Intro to Kubernetes SIG Contributor ExperienceIntro to Kubernetes SIG Contributor Experience
Intro to Kubernetes SIG Contributor Experience
 
Open source
Open sourceOpen source
Open source
 
GITHUB
GITHUBGITHUB
GITHUB
 
How kubernetes works community, velocity, and contribution - osls 2017 (1)
How kubernetes works  community, velocity, and contribution - osls 2017 (1)How kubernetes works  community, velocity, and contribution - osls 2017 (1)
How kubernetes works community, velocity, and contribution - osls 2017 (1)
 

Plus de Optimizely

Plus de Optimizely (20)

Clover Rings Up Digital Growth to Drive Experimentation
Clover Rings Up Digital Growth to Drive ExperimentationClover Rings Up Digital Growth to Drive Experimentation
Clover Rings Up Digital Growth to Drive Experimentation
 
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
 
The Science of Getting Testing Right
The Science of Getting Testing RightThe Science of Getting Testing Right
The Science of Getting Testing Right
 
Atlassian's Mystique CLI, Minimizing the Experiment Development Cycle
Atlassian's Mystique CLI, Minimizing the Experiment Development CycleAtlassian's Mystique CLI, Minimizing the Experiment Development Cycle
Atlassian's Mystique CLI, Minimizing the Experiment Development Cycle
 
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
 
Zillow + Optimizely: Building the Bridge to $20 Billion Revenue
Zillow + Optimizely: Building the Bridge to $20 Billion RevenueZillow + Optimizely: Building the Bridge to $20 Billion Revenue
Zillow + Optimizely: Building the Bridge to $20 Billion Revenue
 
The Future of Optimizely for Technical Teams
The Future of Optimizely for Technical TeamsThe Future of Optimizely for Technical Teams
The Future of Optimizely for Technical Teams
 
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
 
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
 
Evolving Experimentation from CRO to Product Development
Evolving Experimentation from CRO to Product DevelopmentEvolving Experimentation from CRO to Product Development
Evolving Experimentation from CRO to Product Development
 
Overcoming the Challenges of Experimentation on a Service Oriented Architecture
Overcoming the Challenges of Experimentation on a Service Oriented ArchitectureOvercoming the Challenges of Experimentation on a Service Oriented Architecture
Overcoming the Challenges of Experimentation on a Service Oriented Architecture
 
Making Your Hypothesis Work Harder to Inform Future Product Strategy
Making Your Hypothesis Work Harder to Inform Future Product StrategyMaking Your Hypothesis Work Harder to Inform Future Product Strategy
Making Your Hypothesis Work Harder to Inform Future Product Strategy
 
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives RevenueKick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
 
Experimentation through Clients' Eyes
Experimentation through Clients' EyesExperimentation through Clients' Eyes
Experimentation through Clients' Eyes
 
Test Everything: TrustRadius Delivers Customer Value with Experimentation
Test Everything: TrustRadius Delivers Customer Value with ExperimentationTest Everything: TrustRadius Delivers Customer Value with Experimentation
Test Everything: TrustRadius Delivers Customer Value with Experimentation
 
Optimizely Agent: Scaling Resilient Feature Delivery
Optimizely Agent: Scaling Resilient Feature DeliveryOptimizely Agent: Scaling Resilient Feature Delivery
Optimizely Agent: Scaling Resilient Feature Delivery
 
The Future of Software Development
The Future of Software DevelopmentThe Future of Software Development
The Future of Software Development
 
Practical Use Case: How Dosh Uses Feature Experiments To Accelerate Mobile De...
Practical Use Case: How Dosh Uses Feature Experiments To Accelerate Mobile De...Practical Use Case: How Dosh Uses Feature Experiments To Accelerate Mobile De...
Practical Use Case: How Dosh Uses Feature Experiments To Accelerate Mobile De...
 
Run High Impact Experimentation with High-quality Customer Discovery
Run High Impact Experimentation with High-quality Customer DiscoveryRun High Impact Experimentation with High-quality Customer Discovery
Run High Impact Experimentation with High-quality Customer Discovery
 
Using Empathy to Build Custom Solutions at Scale
Using Empathy to Build Custom Solutions at ScaleUsing Empathy to Build Custom Solutions at Scale
Using Empathy to Build Custom Solutions at Scale
 

Dernier

CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICECHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
9953056974 Low Rate Call Girls In Saket, Delhi NCR
 
CALL ON ➥8923113531 🔝Call Girls Badshah Nagar Lucknow best Female service
CALL ON ➥8923113531 🔝Call Girls Badshah Nagar Lucknow best Female serviceCALL ON ➥8923113531 🔝Call Girls Badshah Nagar Lucknow best Female service
CALL ON ➥8923113531 🔝Call Girls Badshah Nagar Lucknow best Female service
anilsa9823
 
CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online ☂️
CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online  ☂️CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online  ☂️
CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online ☂️
anilsa9823
 
TECUNIQUE: Success Stories: IT Service provider
TECUNIQUE: Success Stories: IT Service providerTECUNIQUE: Success Stories: IT Service provider
TECUNIQUE: Success Stories: IT Service provider
mohitmore19
 

Dernier (20)

Tech Tuesday-Harness the Power of Effective Resource Planning with OnePlan’s ...
Tech Tuesday-Harness the Power of Effective Resource Planning with OnePlan’s ...Tech Tuesday-Harness the Power of Effective Resource Planning with OnePlan’s ...
Tech Tuesday-Harness the Power of Effective Resource Planning with OnePlan’s ...
 
Right Money Management App For Your Financial Goals
Right Money Management App For Your Financial GoalsRight Money Management App For Your Financial Goals
Right Money Management App For Your Financial Goals
 
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICECHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
 
Diamond Application Development Crafting Solutions with Precision
Diamond Application Development Crafting Solutions with PrecisionDiamond Application Development Crafting Solutions with Precision
Diamond Application Development Crafting Solutions with Precision
 
CALL ON ➥8923113531 🔝Call Girls Badshah Nagar Lucknow best Female service
CALL ON ➥8923113531 🔝Call Girls Badshah Nagar Lucknow best Female serviceCALL ON ➥8923113531 🔝Call Girls Badshah Nagar Lucknow best Female service
CALL ON ➥8923113531 🔝Call Girls Badshah Nagar Lucknow best Female service
 
Optimizing AI for immediate response in Smart CCTV
Optimizing AI for immediate response in Smart CCTVOptimizing AI for immediate response in Smart CCTV
Optimizing AI for immediate response in Smart CCTV
 
Vip Call Girls Noida ➡️ Delhi ➡️ 9999965857 No Advance 24HRS Live
Vip Call Girls Noida ➡️ Delhi ➡️ 9999965857 No Advance 24HRS LiveVip Call Girls Noida ➡️ Delhi ➡️ 9999965857 No Advance 24HRS Live
Vip Call Girls Noida ➡️ Delhi ➡️ 9999965857 No Advance 24HRS Live
 
How To Use Server-Side Rendering with Nuxt.js
How To Use Server-Side Rendering with Nuxt.jsHow To Use Server-Side Rendering with Nuxt.js
How To Use Server-Side Rendering with Nuxt.js
 
Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...
Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...
Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...
 
Steps To Getting Up And Running Quickly With MyTimeClock Employee Scheduling ...
Steps To Getting Up And Running Quickly With MyTimeClock Employee Scheduling ...Steps To Getting Up And Running Quickly With MyTimeClock Employee Scheduling ...
Steps To Getting Up And Running Quickly With MyTimeClock Employee Scheduling ...
 
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...
 
CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online ☂️
CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online  ☂️CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online  ☂️
CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online ☂️
 
Software Quality Assurance Interview Questions
Software Quality Assurance Interview QuestionsSoftware Quality Assurance Interview Questions
Software Quality Assurance Interview Questions
 
Hand gesture recognition PROJECT PPT.pptx
Hand gesture recognition PROJECT PPT.pptxHand gesture recognition PROJECT PPT.pptx
Hand gesture recognition PROJECT PPT.pptx
 
TECUNIQUE: Success Stories: IT Service provider
TECUNIQUE: Success Stories: IT Service providerTECUNIQUE: Success Stories: IT Service provider
TECUNIQUE: Success Stories: IT Service provider
 
Unlocking the Future of AI Agents with Large Language Models
Unlocking the Future of AI Agents with Large Language ModelsUnlocking the Future of AI Agents with Large Language Models
Unlocking the Future of AI Agents with Large Language Models
 
Try MyIntelliAccount Cloud Accounting Software As A Service Solution Risk Fre...
Try MyIntelliAccount Cloud Accounting Software As A Service Solution Risk Fre...Try MyIntelliAccount Cloud Accounting Software As A Service Solution Risk Fre...
Try MyIntelliAccount Cloud Accounting Software As A Service Solution Risk Fre...
 
Microsoft AI Transformation Partner Playbook.pdf
Microsoft AI Transformation Partner Playbook.pdfMicrosoft AI Transformation Partner Playbook.pdf
Microsoft AI Transformation Partner Playbook.pdf
 
Unveiling the Tech Salsa of LAMs with Janus in Real-Time Applications
Unveiling the Tech Salsa of LAMs with Janus in Real-Time ApplicationsUnveiling the Tech Salsa of LAMs with Janus in Real-Time Applications
Unveiling the Tech Salsa of LAMs with Janus in Real-Time Applications
 
A Secure and Reliable Document Management System is Essential.docx
A Secure and Reliable Document Management System is Essential.docxA Secure and Reliable Document Management System is Essential.docx
A Secure and Reliable Document Management System is Essential.docx
 

Building an Experiment Pipeline for GitHub’s New Free Team Offering

  • 1. Building an Experiment Pipeline for GitHub’s New Free Team Offering Philip Bremer Director, Software Engineering GITHUB
  • 2. Building an experiment pipeline for GitHub’s Free for Teams plan GitHub Presented by Philip Bremer
  • 3. ● Director of Software Engineering
 UI Platform 
 ● Korean adoptee immigrant ● Soccer, yoga, skiing ● Current book: The New Jim Crow ● Ambivert: 75% introvert, 25% extrovert ● Pronouns: he/him About me @philip_bremer
  • 5. ● Individuals and teams ● Price should not be a barrier ● Free private repos ● Upgrade path for advanced features Free for Teams
  • 6. Time to build an experiment pipeline!
  • 7. New orgs and teams Global Nav Repo CTAsOrg Creation
  • 8. Global nav experiment Problem: Users can’t easily find their organization(s) after creating or joining
 Hypothesis: Adding a link to “Your organizations” in user’s global dropdown menu will improve discovery Results: ● Success! ● +134.46% improvement ● Oops! We spent a ton of impressions ● ~10% of total impression budget
  • 9. ● Misunderstood the difference between bucketing vs activation vs tracking
 ● High traffic for global nav
 ● The experiment baked too long How did we spend so many impressions?
  • 10. Track Log notable events
 to calculate impact 
 of variation Bucketing, activation, and tracking Bucketing Assign user 
 to variation Activation Log impression 
 when user sees
 the experiment
  • 11. ● Anonymous sessions are 
 read-heavy and repetitive 
 ● Cache anonymous sessions to reduce load on our application
 ● Goal: serve page for a subset of anonymous sessions without disrupting the benefits of our cache 
 ● Goal: serve page with no negative impact to our page load budget Logged out experiments
  • 12. Performance & DX ● Robust datafile management ● More internal API guardrails for activate vs get_variation ● Improved logging and testing across environments Experimentation culture ● Empower feature teams to experiment with evidence- based delivery Up Next?