Will 2020 mark the shift to a remote-first world in the long run? For GitHub, a distributed workforce is nothing new. Join Sha Ma, VP of Engineering, and Gregory Ceccarelli, Director of Data Science, to learn how they built and scaled a successful experimentation program. They'll share their experience implementing Optimizely across timezones, a remote workforce, and a new business model.
In this session, you'll learn how to:
Optimize UX for a freemium business model
Use data to deliver customer-centered products
Scale experimentation and accelerate growth
3. Our Story
GitHub started in 2008 as a way
to make it easier for developers
to host and share their code.
4. 50M+
Our global community today
87M+
Developers globally building on GitHub in
2020; we want to hit 100M developers by
2025.
Of users are contributing from outside of
the US.
Pull requests merged in 2019 - and 28%
more developers opened their first pull
request in 2019 than in 2018.
80%
100M+Repositories across every continent
on earth.
5. 2K
Rapid internal growth
15+
Employees at GitHub, significant growth
in the last year.
Employees work outside of our San
Francisco HQ, distributed across the globe.
Countries and regions, where full time
employees live and work. We hire in all 50
States of the US.
70%
7. 1. User Experience Research
2. Focus on Minimum Viable Product
3. Data Driven Experimentation
Ship to learn in practice
8. UX Research:
The story
behind GitHub
Actions
All customers found the
Actions sidebar
valuable
Customers thought the
suggestions were very
helpful for figuring out
how to edit the config
file
Customers wanted to
see what ‘Variables’
included - secrets?
“Want to see some mechanism
(global variables or something
else) that would allow me to
retrieve that value from within
the build. And some way to say
only I can see this....” -
Customer X
10. Vision:
Home for All
Developers
“We want every developer and team
on earth to be able to use GitHub for
their development, whether it’s
private or public development.”
11. We now have to learn what to ship
● How to understand new behavior as developers start to
adopt a fundamentally different product
● How to reorient our Revenue function’s strategy to grow
our new freemium self serve business
● How to apply developed product pricing principles to
package our SKUs
13. Developer
Signups
GitHub Free’s launch premise
Free Orgs Creating
Private Repos
Increase the YoY growth rate in the volume
of new signups
Increase the YoY growth rate in the count of
developers that consume content on GitHub
Monthly
Contributors
Remove barriers to entry for small teams to
use the core GitHub workflow and
accelerate private repo adoption
Monthly
Engaged Users
Increase the YoY growth rate in the count of
developers that create content on GitHub
14. And in early 2020...
Implemented Optimizely to
replace our homegrown platform
17. Causation
allows us to
isolate the
impact on Y
because of X It is often easier to generate a plan,
execute against it, and declare success,
with the key metric being: "percent of plan
delivered," ignoring whether the feature
has any positive impact to key metrics
(Kohvai et al. 2013)
22. 1. A Hubber logs an experiment
proposal issue in the Experiment
Council repo
2. The team meets weekly on Wednesdays
to review and approve proposals
3. Next a Data Scientist is assigned
to issues determine metric
baselines and power (if required)
4. The experiment is then instrumented
in Optimizely by Engineering
5. Results are measured and
conclusions about the hypothesis
are drawn
1. Proposer (e.g. Product) makes a
decision about whether to ship the
feature or go back to the drawing
board
Focusing in on how
we actually
experiment...
23. Process out of the way...
Now let’s chat about some
experiments!
24. 🧪 Can we encourage repo growth within Orgs?
Hypothesis: “If we make our Org UI more informative, then more users will
create repos in their Org.”
Control Variant
25. 🧪 Can we encourage more Org page activity 🎓
Hypothesis: “If we make org pages easier to find, then viewership and org
activity will commensurately increase.”
Control Variant
27. 1. Pipeline Inspiration: # of proposals in
pipeline
1. Process Velocity: % of proposals run and
the average latency to go from proposal to
approved experiment to instrumented and
shipped experiment
1. Quality: # of shipped experiments without
issue (customer facing or related to
measurement) and # of experiments shut
down or shipped as features
Current
Program
Metrics 📈
28. Experimentation in their words
Carmel Schetrit (@Carmel-S)
Demand Generation Manager
Growth
“The experimentation council
helped the Growth team in 2020
to evaluate the complexities of
designing, instrumenting, and
measuring experiments on
github.com with Optimizely.
The council often shed light on
scenarios requiring more careful
consideration, including those
that conflict with concurrent
experiments or cannot reach
statistical significance.”
29. Experimentation in their words
Katie Sipos (@ohitsmekatie)
Senior Product Manager
Education
“Our team is rolling out changes to
our GitHub Classroom onboarding
flow through Optimizely.
Being able to definitively prove
which experience is best for our
users and then ship that with
confidence is great!
Without it, we would be flying blind
and shipping features with our gut
feelings and not data.”
31. Accelerating our process
1. Top of Funnel: Have a healthy
proposal pipeline and backlog
1. Middle: Diagnose and triage process
bottlenecks (e.g. do you have
enough designers?)
1. Bottom: Invest in integrating
downstream product metric
enrollment to take full advantage of
Optimizely’s Stats Engine
32. Accelerating our program maturity
1. Strategic Prioritization: # of
experiments selected based on priority
score (versus last in, first out) and
alignment with OKRs
1. Culture: # of times we broadly
syndicate learnings and % business
functions integrated into the process @
GitHub
MVP A: It may need no development work, It’s generally _not_ a basic future release with a release backlog
MVP B: Generally done when assumptions are less risky or customer research has already validated them
Power analysis is used to determine the necessary number of subjects needed to detect an effect of a given size
Primary Hypothesis: If we provide easier accessibility to the org page then access and viewership to that page will increase
Impact: We saw a 131% increase in users visiting this page in the alternative arm compared to the control!
Result: Growth Lifecycle shipped this a feature for all users and it supports a goal of increasing Monthly Active orgs.