This document discusses experimentation and A/B testing. It provides an overview of what A/B testing is and how to effectively conduct experiments. Key points include: - A/B testing involves comparing two variations of a webpage to see which performs better. - To set up an experiment, you need analytics tools, a hypothesis to test, ways to create test variations, and metrics to analyze results. - Case studies show how experiments can significantly impact goals like reducing returns or increasing signups. - Platforms like Optimizely make experimentation easy for both technical and non-technical users.

2. Matt Stodolnic
VP Marketing & Alliances
Pantheon
Brad Taylor
Senior Frontend Engineer
Optimizely
Matthew Auerbach
Developer Evangelist
Optimizely

3. ➔
➔
➔
➔
➔

5. Importance of Experimentation
●
●
○
●

7. What is A/B Testing?
Comparing two variations of a webpage to see which performs better

8. Hypothesis: Attracting more attention to the SHOP NOW button will increase
impressions to the item catalog
A B

9. Business Objective: Reduce cost of free
shipping program and deliver better
online shopping experience
Goal: Encourage visitors to purchase
only one instead of multiple sizes
Experiment: Target a helpful modal to
‘size-conscious’ shoppers to assist them
with finding the correct fit
Impact: 80% decrease in return rate for
‘size-conscious’ shoppers
Case Study Brooks

10. VARIATION
80%
Fewer
returns

11. Visual Editor

12. The Optimizely Snippet
https://help.optimizely.com/Set_Up_Optimizely/Implement_the_Optimizely_snippet
Each project has its own project code,
called the snippet.
Optimizely snippet contents:
• Active Experiments / Campaigns*
• Draft Experiments / Campaigns*
• Project Metrics
• Project Audiences
• Extensions*
• Optimizely execution and tracking logic
* Optimizely can be configured to either remove these from the snippet
or load them asynchronously.

13. Drupal Module
Makes it easy to define which
pages to load the Optimizely JS, no
native editor functionality.
WordPress
Install JS snippet directly in the head
module
Easy Access to A/B Testing

15. ● Secure the necessary tools
● Observe user behavior
● Develop a hypothesis
● Create tests for your hypothesis
● Analyze results

16. ● Google Analytics or other analytics
tool (make sure tools are working).
● Optimizely or other easy to use, easy
to setup (SaaS/JS include) testing tool.
● Crazy Egg or other user behavior /
click tracking tool

17. Google Behavior Flow.
Where are people going on our website?
Click-path analysis (by URL)
Crazy Egg
What are people clicking on our website?
More visual click-path analysis (by clicks)

18. More clicks, please.

19. More clicks, please.

22. ● New Messaging will connect with our
visitors and more our our visitors will
navigate to the tools pages of our
website.
● Sign-Ups will increase because we are
our value propositions are more aligned
to our audience.

24. ● A| B Testing Almost anything on your
website that affects visitor behavior can
be A/B tested.
● Testing Complete Pages like
Pantheon did for this test requires more
planning for redirection of traffic and
measuring multiple changes at once.

28. 50% - CONTROL 50% - VARIATION

29. ● Engagement - % of visitors who clicked
on any part of the experiment page
● Traffic to new user-specific landing
pages - % of visitors who visited any of
the three new landing pages
● Developer Registration - % of visitors
who registered for a free developer
account

30. A B C

33. ● New Homepage - A new homepage is
both an A/B test and a multivariate test.
● Just Start Testing - Agree with testing
purists and move on. Improve the site,
create new tests. Improve the site,
create new tests.

36. • Improve the user experience
• Verify new product features
• Drive specific KPIs
• Test messaging
• Measure algorithm performance
• Build a culture of experimentation
Why is A/B testing important?

37. Working with your organization

38. Enable Rapid Experimentation by Everyone
Built for business users
See results in real-time
Act with confidence
and developers
Always-on
experimentation

39. Experiment across your organization
and business objectives
Marketing
Data Science
Engineering
Operations
Product
Grow revenue
Reduce costs
Build brand
Maximize
Profits
Experiment to roll out new features
Experiment to optimize your site’s functionality
Experiment with user experience to drive engagement
Experiment to better understand recommendation algorithms
Experiment for lead generation and conversion rate

40. Why stats engine

41. 1920s
Data is expensive
Data is slow
Practitioners are trained
experts
2017
Data is cheap
Data is real-time
Everyone is a practitioner
Back then, stats was hard!

42. Before Stats Engine: Lots of work to get valid results
To get valid results for a test, you had
to follow some rules:
● Set a sample size in advance
● No peeking
● Can’t test too many goals and
variations

43. After Stats Engine: Always valid results to make
decisions in real-time
Sequential testing
Look at results in real time
False discovery rate control
Test as many goals and
variations as needed
= Always valid results with
Stats Engine

44. Optimizely platform from a dev perspective
Experiment SDKs
Developer Tools
REST API
Behavior API
Event API
Data Export
Extensions
Web Tools
Customer
Profiles

45. Optimizely X: Operationalize Experimentation Across
the Organization
Enable Rapid Experimentation
by Everyone
Experiment Everywhere on
Any Channel and Device
Deliver High-Performing
Personalized Experiences
Optimize Anything that
Matters to your Business

47. Extensions
• Reusable templates for making changes in the visual editor.
• A developer can build a extension with HTML, CSS, and JavaScript
• Customizable fields allow any user to alert the Extension
• Pop-up message, banner, countdown timer, etc…
• Sample Library

48. Extensions

49. Get started with experimentation today!
Documentation and developer information: developers.optimizely.com
Free trial: optimizely.com/free-trial