Dan McGaw - KISSmetrics - @danielmcgaw ! Dan is the Director of Marketing at KISSmetrics, where he get’s the opportunity to help the best marketers in the world measures their marketing and business efforts. Prior to KISSmetrics, Dan was VP of growth at Code School, where he helped the company grow to over 500k users and $400k+ in monthly revenue. Brendan O’Rourke - Optimizely - @optimizely ! Brendan is an Enterprise Solutions Engineer at Optimizely. He joined Optimizely two years ago, after realizing his dream of becoming an actuary was very... dry. He now spends his time helping enterprises like Microsoft, NBC Universal, and Yelp turn data into action through testing and conversion rate optimization. Why A/B Tests Matter for Growth … it’s all about the customers “Just run an experiment.” Stage 1: Testing Through Product Early Customers 2 An Optimizely experiment: Pricing page circa 2012 Variation An Optimizely experiment: Stage 2: Optimizing for Early Customers Online Expansion 3 Where are my customers? Analytics can help to determine where visitors are clicking and when they are dropping off What is my bounce rate? Why do visitors come to my site? • What content is being added to my site? At what rate? • How do you expect visitors will navigate to the final conversion? Website Navigation: Stage 3: Click Paths and Online Expansion Original An Optimizely Experiment: An Optimizely Experiment: Variation Variation = +20% traffic to blog +25% traffic to help center Optimizely website circa 2012 An Optimizely Experiment: Original Redesign An Optimizely Experiment: Stage 4: A Pivotal Redesign Redesign = +28.3% Segmenting Audiences 5 • Do you have a marketing program with many channels? Example: Littopia Original SEM Landing Page Breadcrumb navigation Search fields Suggestions for similar resorts Example: Littopia Variation SEM Landing Page No breadcrumb navigation Removed search fields No recommended resorts Stage 5: Segmenting Audiences Example: Littopia Variation = +23.7% New Markets 5 Stage 5: New Markets • What big initiatives are you tackling this year? • New marketing campaigns • Product launch • Mobile experience • Personalization • Internationalization • What role does your online marketing play? Optimization and Expansion Product page Cart page Example: Veggie Tales Product page Cart page Example: Veggie Tales Step 6: New Markets Variation = +28.1% 1 Product Development 2 Early Revenue 3 Online Expansion Today, we’ve covered: 4 First Overhaul 5 Defining Audiences 6 Optimizing for Market Changes But Wait, There Is More! Measure All The Results Of Your Test KISSmetrics A/B Test Report Measuring the first conversion is just part of the equation KISSmetrics Funnel Report Make sure to evaluate the entire funnel when running and A/B Test KISSmetrics Cohort Report All A/B tests affect your retention rates Optimize for your customers … and always be testing

1. Brendan O’Rourke, Optimizely
Dan McGaw, KISSmetrics
A/B Tests for 6 Stages of Your
Company’s Growth
May 1, 2014

2. Dan McGaw - KISSmetrics - @danielmcgaw
!
Dan is the Director of Marketing at KISSmetrics, where he
get’s the opportunity to help the best marketers in the world
measure their marketing and business efforts. Prior to
KISSmetrics, Dan was VP of growth at Code School, where he
helped the company grow to over 500k users and $400k+ in
monthly revenue.
Brendan O’Rourke - Optimizely - @optimizely
!
Brendan is an Enterprise Solutions Engineer at Optimizely.
He joined Optimizely two years ago, a er realizing his dream
of becoming an actuary was very... dry. He now spends his
time helping enterprises like Microso , NBC Universal, and
Yelp turn data into action through testing and conversion
rate optimization.

3. @optimizely #KISSwebinar
Join us on Twi er

4. @danielmcgaw #KISSwebinar
Follow-me and join the conversation

5. Why A/B Tests
Ma er for Growth

6. … it’s all about
the customers

7. 1 Product Development
2 Early Revenue
3 Online Expansion
Six Stages
4 First Overhaul
5 Defining Audiences
6 Optimizing for Market Changes

8. WATCH WEBINAR RECORDING NOW

9. Product Development
1

10. “Just run an experiment.”

11. Stage 1: Testing Through Product Development
• Experimental mindset
• Quantitative > Qualitative
• Start with what won’t scale
• User testing
• Customer interviews
• Always create variations
• Practice documenting
hypotheses
Wireframing Optimizely.com
Early Stages:

12. Early Customers
2

13. Stage 2: Optimizing for Early Customers
• Where is your most
important conversion?
• Pricing page
• Checkout experience
• Product pages
• What keeps your soon-to-be
customer from converting?
• Prime area for impactful A/B
tests
What does your pricing page look like?
Where do you acquire users/customers?

14. Stage 2: Optimizing for Early Customers
An Optimizely experiment:
Pricing page circa 2012

15. Stage 2: Optimizing for Early Customers
Variation
An Optimizely experiment:

16. Stage 2: Optimizing for Early Customers
• High-impact test
• Understand audience
triggers
• Social proof
• Recommendations
• Promotional offers
• Increase visibility of the
page
Variation = 40% increase
in Gold subscriptions
An Optimizely experiment:

17. Online Expansion
3

18. Where are my customers?

19. Stage 3: Click Paths and Online Expansion
Analytics can help to determine where visitors are
clicking and when they are dropping off
What is my bounce rate?

20. Stage 3: Click Paths and Online Expansion
• Why do visitors come to
my site?
• What content is being
added to my site? At what
rate?
• How do you expect visitors
will navigate to the final
conversion?
Website Navigation:

21. Stage 3: Click Paths and Online Expansion
Original
An Optimizely Experiment:

22. Stage 3: Click Paths and Online Expansion
An Optimizely Experiment:
Variation

23. Stage 3: Click Paths and Online Expansion
Variation =
+20% traffic to blog
+25% traffic to help center
3X increase in demo requests
An Optimizely Experiment:
• Maximize inbound traffic and
interest on your site
• Choose your conversion goals
based on business objectives:
• Educate and inform
• Solicit signups or orders
• Engage and socialize
• Keep funnels and click paths
short

24. Pivotal Redesign
4

25. Stage 4: A Pivotal Redesign

26. Stage 4: A Pivotal Redesign
Optimizely website circa 2012
An Optimizely Experiment:

27. Stage 4: A Pivotal Redesign
Original Redesign
An Optimizely Experiment:

28. Stage 4: A Pivotal Redesign
Redesign =
+28.3% engagement with editor
+4.6% accounts created
• Choose key metrics before
redesign process starts
• Measure new design against
the current one
• Validates a major investment
• Even a draw can be a win
An Optimizely Experiment:

29. Segmenting Audiences
5

30. Stage 5: Segmenting Audiences
• Do you have a marketing
program with many channels?
• What types of traffic does your
site receive?
• New vs. Returning
• Customers vs. Non-Customers
• Paid vs. Organic
• How do you communicate with
your audiences at scale?
Engaging Audiences

31. Stage 5: Segmenting Audiences
Example: Li opia
Original SEM Landing Page
Breadcrumb
navigation
Search fields
Suggestions for
similar resorts

32. Stage 5: Segmenting Audiences
Example: Li opia
Variation SEM Landing Page
No breadcrumb
navigation
Removed
search fields
No recommended
resorts

33. Stage 5: Segmenting Audiences
Example: Li opia
Variation =
+23.7% Li ticket purchases
• Match your visitor’s
expectation and intent
• Use analytics tools to follow a
click path and identify
meaningful audiences
• Use A/B testing to determine
your strategy for engaging
different audiences
• Messaging and imagery
• Custom landing page experience

34. New Markets
5

35. Stage 5: New Markets
• What big initiatives are you
tackling this year?
• New marketing campaigns
• Product launch
• Mobile experience
• Personalization
• Internationalization
• What role does your online
marketing play?
Optimization and Expansion

36. Step 6: New Markets

37. Step 6: New Markets
Product page Cart page
Example: Veggie Tales

38. Step 6: New Markets
Product page Cart page
Example: Veggie Tales

39. Step 6: New Markets
Variation =
+28.1% Revenue per visitor
• Use information from previous
tests to inform your next
initiative
• Create strong hypotheses
• Use qualitative information to
determine what to test and
optimize
• User testing
• Survey feedback
• Reapply Explore-Refine strategy
Optimization and Expansion

40. 1 Product Development
2 Early Revenue
3 Online Expansion
Today, we’ve covered:
4 First Overhaul
5 Defining Audiences
6 Optimizing for Market Changes

41. But Wait, There Is More!

42. Measure All The Results
Of Your Test

43. KISSmetrics A/B Test Report
Measuring the first conversion is just part of the equation

44. KISSmetrics Funnel Report
Make sure to evaluate the entire funnel
when running and A/B Test

45. KISSmetrics Cohort Report
All A/B tests affect your retention rates

46. LOG IN WITH GOOGLE
Start Your Free KISSmetrics Trial

47. Optimize for your
customers
… and always be testing

48. Questions?
Brendan O’Rourke
Enterprise Solutions Engineer
Optimizely
@optimizely
Dan McGaw
Director of Marketing
KISSmetrics
@danielmcgaw

49. THANK YOU