A/B testing is an essential element in any product managers playbook. However having the freedom and flexibility to customize testing based on what the data is saying often requires a lot of time and effort, particularly when it comes to engineering resources. Optimizely offers a flexible approach to experimentation through the use of feature testing, which provides more customization options without the additional development effort typically required to implement these feature optimizations. Megan Bubley, a Senior Product Manager at The Zebra, will share her experience working with Optimizely’s feature tests to create a results page where users can compare multiple auto insurance options driven by actual user needs, as well as her experience customizing the experience based on device platform.

1. How The Zebra Utilized Feature
Experiments To Increase Carrier Card
Engagement by 10%
Megan Bubley
Sr. Product Manager
THE ZEBRA

2. 2
The Zebra
We make insurance black and white.
The Zebra was founded with a single goal: to
simplify insurance, and in doing so we’ve
become the nation’s leading insurance
comparison site.
Our vision is to be a trusted partner for
consumers, helping them to better manage and
understand all their insurance needs through
every stage of life, even in the face of the
unexpected.

3. Carrier Card Redesign
Auto Results Page - Multivariate Feature Test
3
Our goal with this redesign was to rapidly test and learn. We
validated our design choices with usability tests but still had
questions around which combination of features would create
the best user experience and how performance would differ
based on device.
We wanted to test a combination of carrier card features but
were concerned about the time it would take to analyze test
performance and the amount of engineering bandwidth
required to introduce a new round of A/B tests.
In the past running a multivariate experiment this robust would
require weeks of engineering time and would drastically
increase our delivery timelines, however through the use of
feature testing we were able to test multiple variants with
limited engineering effort in a short amount of time!

4. Differentiation by Platform
Desktop vs. Mobile
4
For our first round of tests we wanted to focus on
information layout and creating consistency across
platforms so we could better assess how design
impacted user engagement.
We knew that historically desktop and mobile have
performed very differently in terms of engagement.
● Mobile typically performed worse when more
elements were added to carrier cards vs.
desktop.
● Since desktop carrier cards have more real
estate to utilize, more content was typically
added to these cards creating a very different
user experience across platforms.

5. 5
Step One: Work with design and engineering to identify the carrier
card elements we want to test along with the number of variants
we wanted to start with.
Step Two: Create feature variables within Optimizely that allow us
to turn on or off elements on the carrier card so we can customize
the design.
Step Three: Once the test goes live, use real time data to monitor
test performance and come up with future variants we want to test
based on carrier card conversion rates.
Development Strategy

6. Feature Variables
6
Square Logo
Rectangle Logo
Star Ratings
Coverage Value Prop
First Month Pricing
Price in CTA
Carrier Taglines
List View
Card View

7. 7
Running The Test
The most challenging part of A/B testing has typically
been saving space for optimizations, particularly when a
feature is testing and developers are no longer actively
working it.
In the past we have had issues with prioritizing future
optimizations based on how we think the test will
perform in order to keep the momentum going with
engineers.
With feature based testing we were able to make changes
to our test based on the data without any additional
engineering effort—essentially we had the ability to create
a custom carrier card without any engineering resources.
Example of the feature test setup used for the carrier card
redesign.
7

8. 8
Variant Designs
Mobile

9. Monitoring Performance & Use of Feature Flags
Achieving Our Long Term Vision
9
While we had hoped to implement a similar experience across
desktop and mobile we noticed that different variants performed
better on different platforms—for example rectangular logos
performed better on mobile and square logos did better on desktop.
Since we wanted to come back to some of these findings and retest
at a later date we utilized the feature flag functionality once we had
a winning variant to promote that feature to 100% of traffic without
eating into our traffic allotment in Optimizely.

10. 10
10%
Increase in carrier card engagement across platforms
60%
Reduction in engineering effort and development time
Final Outcomes
We ran the carrier card feature test for over 6 months and
worked closely with stakeholders to make changes to our
variants as our product changed.
By running this test as a feature experiment we were able
to implement changes to our variants without any
additional development effort—something that would have
taken us weeks to implement in the past.
We were also able to test different variants on mobile and
desktop which allowed us to differentiate our experience
by platform based on user behavior.
As a result of these tests we saw significant improvement
to carrier card engagement across platforms.

11. 11
Benefits
1. Lots of flexibility when it comes to
testing multiple designs.
2. Ability to turn on and off feature
variables with the flip of a switch—no
engineering effort required.
3. Ability to move the experiment behind a
feature flag to reduce the strain on
traffic allotments
4. Ability to retest features at a later date —
once new features have been
launched—to understand full impact of
the final product on the user experience.
Challenges
1. Lots of frontloading required to identify
the different feature variables and
testing strategy.
2. Additional engineering effort required up
front in order to set up the feature test—
was more challenging to implement
than a straight A/B test.
3. Additional testing required to ensure all
feature variables display correctly in
production.
4. Multiple dashboards needed to record
performance and there was not a super
easy way to track performance in
Optimizely.

12. 12
PRESENTATIO
N TITLE
Questions?
Contact Information:
Email: mbubley@thezebra.com
Linkedin: www.linkedin.com/in/megan-bubley