Do the Thing That Makes the Money

1. opticon2017
Chris Neumann
CEO, CROmetrics
Do the Thing
That Makes the
Money

2. opticon2017
What is the Thing
That Makes the
Money?
• Let data tell you what customers want
• Make testing and experimentation a strategy
• Identify & focus on one, most important, metric
• Accelerate throughout to increase ROI

3. opticon2017
Successful Companies are Using Data
Strategically to Give Their Customers What
They Want
Media RetailB2B
• Amazon
• YouTube
• Netflix
• Buzzfeed
• Facebook
• Amazon
• Chewy
• eBay
• Jet.com
• Amazon (AWS)
• Google (Cloud, apps)
• Shopify

4. Amazon Views
Experimentation as
Critical to Growth
Our success at Amazon is
a function of how many
experiments we do per year,
per month, per week, per day.
- Jeff Bezos

5. opticon2017
The strategy of Facebook is to learn as quickly as possible
what our community wants us to do – and that requires a culture
that encourages people to try things and test things and fail,
and it also requires building infrastructure to enable
people to do that.
Testing and Learning is the Core Strategy for
Serving Customer Needs and Operating a Growth
Company
““
Mark Zuckerberg
Masters of Scale Podcast w Reid Hoffman

6. opticon2017
Facebook Today Represents the Future
of Operating a Tech Company
• Any idea proven better in testing can ship immediately
• No hierarchical approvals needed to run a test
• 10,000 tests running concurrently at any given
moment

7. opticon2017
Redesign Showdown:
Amazon vs Facebook

8. But Marketers Have
Long Sought to Use
Data to Drive Decisions
Advertising people who ignore
research are as dangerous as
generals who ignore decodes
of enemy signals.
- David Ogilvy

9. Why Don’t Companies Do the Thing That
Makes the Money?

10. opticon2017
Many Companies aren’t Embracing
Experimentation as Quickly as
Competitors
• Testing and research historically difficult
o Expensive
o Time consuming
o Unreliable
o Competitively complicated
• Experimentation program success requires
speed and infrastructure
• Cultural blockers can encumber testing
strategy

11. “Old” Approach to
Satisfying Customers:
Not Data Driven
• Prior experience and
company “Folklore”
• “This is how it’s always been
done”
• “Mailing 20% off coupons is
what works”

12. opticon2017
“Hunches” & HiPPOs / opinions

13. opticon2017
Creative function
should be data driven
• “Brand Guidelines”
• “Creative know-how”
• Opportunity to leverage
creative talent exists

14. opticon2017opticon2017
Testing as a Strategy:
Real Estate Pop Quiz
Can Intuition and
Experience Beat Data-
Driven Experimentation?

15. opticon2017
Variation A - Icons
Action bar located directly
below the main property details
section.

16. opticon2017
Variation B - Icons
with Labels
Expand buttons and add labels
to indicate purpose.

17. opticon2017
Variation Improvement Conversion Rate
A: Icons --- 0.20%
B: Labeled Icons +333% 0.86%
Labeling Icons Leads to 333% Lift in
Agent Contacts

18. opticon2017opticon2017
Testing as a Strategy:
Media Pop Quiz
Can Intuition and
Experience Beat Data-
Driven Experimentation?

19. opticon2017
Tested Multiple Ad-Block Messaging Variations
Please disable your ad blocker for the optimal browsing experience. Whitelist [site] >
We noticed you have an ad blocker enabled. Please whitelist us so we can
continue to provide free content.
Disable Now >
Your ad blocker may cause this site to load incorrectly. Disable Now >
A
B
C

20. opticon2017
More Variations Provide More Opportunities to Win
Variation Improvement Conversion Rate
A: Optimal Experience +41% 6.11%
B: Free Content +109% 9.03%
C: Load Incorrectly +38% 5.99%

21. opticon2017opticon2017
How to Develop
Strategic Customer
Data that Drives Growth

22. opticon2017
Heatmaps Show Us Where People Actually Click

23. opticon2017
Analytics Offer Insights on User Behavior

24. opticon2017
Customer Feedback Yields Winning Copy Ideas

25. opticon2017
Tailored Experiences Based on Inbound
Traffic Source Drive Increased Efficiency
Normal Site PPC Landing Page

26. opticon2017
Experiment Results Yield New Ideas to Test
+ 20%
conversion

27. opticon2017opticon2017
Testing as a Strategy:
Financial Lead Gen
Can Intuition and
Experience Beat Data-
Driven
Experimentation?

28. opticon2017
Variation A –
Solution Icons
Display the four solution
icons to the left of the hero
and the four solutions further
down the page.

29. opticon2017
Variation B - Icons with Solution Type in
Header Subnav
On internal pages & the
homepage, expose sub-navigation
with images of each solution
below the main navigation.

30. opticon2017
The Improved Header Subnav Saw over
20% Higher Conversion
Goal: Submitted leads for Desktop only
Variation Improvement Conversion Rate
A: Solutions Icons --- 10.58%
B: Header Subnav +21.2% 12.82%

31. opticon2017opticon2017
Kickstart a Testing
Strategy and Drive Toward
a Culture of
Experimentation

32. opticon2017
Identify the One Thing
That Makes the Money
• Facebook: Daily Active Users
• Uber/Lyft: Weekly rides
• Optimizely: Monthly experiment starts
• Airbnb: Nights booked
• CROmetrics: Weekly experiments run

33. opticon2017
Prioritize Tests to Maximize ROI
each
mpact
onfidence
ase
R
I
C
E
70%
Desktop
30%
mobile

34. opticon2017
Impact
each
mpact
onfidence
ase
R
I
C
E
+/- 5%
vs
+/- 25%

35. opticon2017
Prioritize Tests to Maximize ROI
each
mpact
onfidence
ase
R
I
C
E

36. opticon2017
Prioritize Tests to Maximize ROI
each
mpact
onfidence
ase
R
I
C
E

37. opticon2017
Prioritize Tests to Maximize ROI
each
mpact
onfidence
ase
R
I
C
E
+♥

38. opticon2017
Increase Experiment Throughput to
Maximize ROI and Customer Learning
Experiments
Time

39. opticon2017
Develop Key
Competencies for
Program Success
• Drive culture of experimentation
• Track strategic insights and develop playbook
• Augment internal staff
• Increase experimentation velocity
• Unify cross functional teams
• Dedicated engineering team
• Experimentation cycle faster than internal development cycle
• Experiment code doesn’t need to be checked into source
• Core developers not stressed or distracted

40. opticon2017opticon2017
Testing as a Strategy:
Final Exam
Can Intuition and
Experience Beat Data-
Driven
Experimentation?

41. opticon2017
Variation A – Pay Rate
Pay rate added above the
fold of a multi-step mobile
conversion form.

42. opticon2017
Variation B – Note
from Manager
Pay rate added
along with a
“personal” note
from the manager.

43. opticon2017
Variation Improvement Conversion Rate
A: Pay Rate +14.4% 12.66%
B: Note from Manager +13.6% 12.57%
Goal: Completed Application
Both Variations Yielded the Same Results
for Completed Applications

44. opticon2017
Variation Improvement Conversion Rate
A: Pay Rate +16.8% 6.24%
B: Note from Manager +54.2% 8.24%
Goal: Scheduled Interview
Note from Manager Improved Down-funnel
Interviews Metric by 3X vs Pay Rate

45. opticon2017
Do the Thing That
Makes the Money
• Make testing and experimentation a strategy
• Develop strategic customer data
• Kickstart your program
• Identify one, most important, metric
• Prioritize experiments
• Accelerate throughput to increase ROI

46. opticon2017
Q&
A

47. opticon2017
THANK YOU