Here’s the playbook Intuit uses to increase its experimentation velocity — even when they face traffic limitations.
Mike Loveridge is not new to running experimentation teams. Before Intuit, he built out programs at Ancestry.com, GE, Humana, and CheapOair. He's an expert at making experimentation work at high velocity, even in traffic-challenged situations.
In this webinar, Mike Loveridge shared his best practices for making CRO work at high velocity, key lessons from scaling multiple teams, and why he's bullish on the future of "test and learn".
3. How to Scale Your
Experimentation Program
WEBINAR
Mike Loveridge
Intuit
The playbook Intuit uses to increase its experimentation
velocity — even when they face traffic limitations.
Wes McCabe
Optimizely
5. Digital Experience Optimization:
Digital Products, Commerce & Campaigns
Up to 5X Increase in Yield:
Revenue, Share of wallet, Funnel conversion,
Risk mitigation, Ops efficiency
Partner of Choice:
Work with leading global enterprises & “digital
disruptors” including 26 of F100
OUR COMPANY
Digital Experimentation Platform:
Next gen “Test and Learn” System
Replaces Digital Guesswork:
with Evidence-based optimization
Speeds Innovation & Optimization:
Single platform for marketing & product teams
Best in class stats & machine learning
Consumer grade usability
Enterprise program management & prof. services
OUR SOLUTION
6. “Our success is a function
of how many experiments
we do per year, per month,
per week, per day.”
“Instead of saying ‘I have an
idea,’ what if you said
‘I have a new hypothesis,
let’s go test it.’
“Our company culture
encourages
experimentation and
free flow of ideas.”
“One of the things I’m
most proud of, and I think
what is the key to our
success, is this testing
framework we’ve built.”
Experimentation is the
Next Great Business
Transformation
Jeff Bezos Larry Page
Mark Zuckerberg Satya Nadella
The Surprising
Power of Online
Experiments
7. 10x more experiments
Consumer-grade usability
Open data integration
Maximum yield of business value
UX and feature-level experiments
and personalization at every digital
touch point
Enterprise-wide
management & governance
Captures, governs and shares
ideation, analyses & results
Optimizely X Unlocks the Experimentation Best Practices
of the World’s Greatest Digital Companies
World’s most trusted outcomes
BestinclassStatsEngine
FasttimetoresultsviaML
Accelerates digital innovation
Speeds devops&deployment
De-riskscontinuous feature delivery
Ensuressuccessofnewfeatures
Unifying flagging & experiments
enablescontrolledtestingof new
features while maintaining high
performance
Ideate
Manage
StoreGovern
Analyze
Share
Open Data
Integration
Security &
compliance
Stats
Engine
Stats
Accelerator
Consumer-
grade
usability
APIs &
Developer
Tools
Feature
Flags
Open source
SDKs
X-Channel
Full Stack
Experimentation
Personali-
zation
Recommen-
dations
Web
Experimentation
PRODUCTS
COMMERCE
CAMPAIGNS
8. We’re Proud to Work With
Great Global Enterprises
21 of the Fortune 100 have Chosen Optimizely to Drive Their Digital Experience
9. B U S I N E S S V A L U E
VELOCITY/VOLUME
LEVEL 1
Executional
Start
LEVEL 2
Foundational
Growth
LEVEL 3
Cross-functional
Advancement
LEVEL 4
Operational
Excellence
LEVEL 5
Culture of
Experimentation
Our Products and Services Take
You on Your Experimentation Journey
16. 1. More learnings
• More tests*
• Testing in more places
• Testing through additional mediums (usability, surveys, etc.)
• Efficient designs (MVT)
2. Reduced time to launch = cost savings
3. Timely analysis & recommendations = quicker time to market
4. Quicker implementation of winners = cost savings
*Without sacrificing quality
How can velocity be defined?
17. 1. Bottom line impact
2. Number of tests
3. Development time by t-shirt size (S, M, L, XL)
4. Analysis & recommendation time
5. Implementation time
6. Error rate
7. Win rate
8. Number of learnings
9. Test quality score
Supporting KPIs
18. “If one day the speed kills me, do not cry
because I was smiling.”
Paul Walker (1973 – 2013)
27. Everything you need to track & schedule employee time
Time tracking and scheduling that’s accurate to the second
Time tracking and scheduling that’s full-featured & easy to use
Time tracking that’s full-featured & easy to use
Baseline
+3.7%
+26.9%
+9.2%
Homepage Test Results
28. More Homepage Testing Results - AWARDS
● Control: Time tracking with over 8,000 five-star reviews
● T1: 250 million timesheets processed and counting
● T2: 246,897,755 timesheets processed and counting
● T3: Time tracking with 500,000 active users and counting
● T4: PC Magazine Editors’ Choice “Best Time Tracking Software”
● T5: G2 Crowd Winner for “Best Time Tracking Software”
● T6: TSheets, the highest-rated time tracking app on top review hubs
29. ● Control: Time tracking with over 8,000 five-star reviews
● T1: 250 million timesheets processed and counting (+5.5%)
● T2: 246,897,755 timesheets processed and counting
● T3: Time tracking with 500,000 active users and counting (+8%)
● T4: PC Magazine Editors’ Choice “Best Time Tracking Software”
● T5: G2 Crowd Winner for “Best Time Tracking Software”
● T6: TSheets, the highest-rated time tracking app on top review hubs (+6.4%)
More Homepage Testing Results - AWARDS
33. Don’t be afraid to run tests when a lower sample size makes it difficult to
imagine getting statistical significance. You will still learn things!
We test to generate learnings.
We also test to obtain statistical significance.
34.
35. • Usability
• Online surveys
• A/B tests with rich data sets
• Heatmaps, clickmaps, scrollmaps, confetti reports
• Video recordings
• Track all clickable items the page being tested
• Multiple KPIs… do they all align?
• Have a strong hypothesis
• Does the data you get back make sense in explaining the hypothesis?
• Why or why not?
• Alternative statistical models
• Patience. Just let it run… for a LONG time.
• Be conservative with mission-critical pages
• Test higher impact changes
Ways to experiment in low traffic situations
36. Homepage Pricing
Signup
Buy Now
Signup
Complete
Signup
Free Trial
Control
Sign up for a 2-week free trial. Then, buy it!
KPI 1: Free trial conversion rate (How many start a trial)
KPI 2: Activation rate (How many buy after the trial)
Treatment
Sign up for a 2-week free trial. Then, buy it! OR… Just buy it right now and get 50% off for the first 3 months.
KPI 1: Free trial conversion rate (How many start a trial)
KPI 2: Activation rate (How many buy right now or buy after the trial)
KPI 3: Revenue per user
KPI 4: Cancellation rate
37. Test Results (WEB.039.18 - Homepage Buy Now) TEST DATES: JULY 17 - AUG 1, 2018
LEARNINGS
● FREE TRIAL SIGNUPS DECREASED BY 30%
● WHEN MOBILE WAS REMOVED, THE DECREASE WAS 10%
● USERS ARE CHOOSING BUY NOW WITH 50/50 INTENT. SIGNIFICANT FALLOUT IN THE BILLING FUNNEL.
● LARGE FALLOUT ON THE PRICING PAGE, DESPITE HUGE GAINS IN ENGAGEMENT
● ACTIVATIONS WERE FLAT
Control Treatment
RESULTS
ACTIVATIONS WERE FLAT
44. • Dedicated (“Center of Excellence” executes all tests)
• Specialists are efficient
• Head of Experimentation has more control
• Dispersed (Each product team does their own testing)
• Product owners OWN performance
• Hybrid (“Center of Excellence” teaches a man to fish)
Dedicated vs Dispersed Teams
45. • Repeatable, written process
• Executive sponsorship
• Having a PM is nice… especially for complex, cross functional tests
Key Elements for ANY Model