[CXL Live 16] When, Why and How to Do Innovative Testing by Marie Polli

•

1 like•6,175 views

The document discusses different strategies for innovative testing on websites, including iterative and innovative testing. Iterative testing involves changing small elements like button colors or copy, while innovative testing involves changing multiple elements or entire designs. It recommends using innovative testing when an iteration won't work, testing potential is small, or traffic is low. This allows uncovering bigger gains. Innovative testing carries more risk but also higher potential benefit than iterative testing. The document advises making sure the testing process is solid by asking questions like whether the test matches the hypothesis and goals are set up correctly. It also recommends continuing qualitative research for feedback.

Marketing

WHEN, WHY & HOW TO DO
INNOVATIVE TESTING

3 STRATEGIES
FOR BETTER DESIGN
BETTER CR, MORE SALES

ITERATIVE TESTING
‣ BUTTON COLOR CHANGE
‣ COPY CHANGE
‣ DIFFERENT HERO IMAGE
‣ REMOVING/ADDING A PIECE OF CONTENT

ITERATIVE TESTING
‣ BUTTON COLOR CHANGE
‣ COPY CHANGE
‣ DIFFERENT HERO IMAGE
‣ REMOVING/ADDING A PIECE OF CONTENT
ADD, CHANGE OR REMOVE ONE ELEMENT

“THERE'S NO REASON TO LIMIT
YOURSELF TO A SINGLE STRATEGY,
BECAUSE THE 3 APPROACHES
COMPLEMENT EACH OTHER WELL.”

INNOVATIVE TESTING
‣ NAVIGATION CHANGES
‣ RADICAL REDESIGN
‣ NEW FUNCTIONALITIES, FEATURES
‣ COMBINATION OF MULTIPLE CHANGES

INNOVATIVE TESTING
‣ NAVIGATION CHANGES
‣ RADICAL REDESIGN
‣ NEW FUNCTIONALITIES, FEATURES
‣ COMBINATION OF MULTIPLE CHANGES
ADD, CHANGE OR REMOVE MULTIPLE ELEMENTS

“INNOVATION IS CHANGE THAT UNLOCKS
NEW VALUE”
JAMIE NOTTER

WHEN, WHY & HOW IS
INNOVATIVE TESTING A
GOOD IDEA?

#1 WHENAN ITERATION WON’T SUFFICE
#2 WHEN YOU’RE TESTING POTENTIAL IS SMALL
#3 WHEN YOU DON’T HAVE MUCH TRAFFIC

‣ BASICS ARE OFF: New
customers don’t
understand what the site
is about, who is it for and
what’s the ﬁrst thing they
should do there.
‣ MESSY NAVIGATION:
Visitors are unable to ﬁnd
the product/service they
need
‣ etc

CR UPLIFT OF 22.33%
Original CXL Variation

YOU’VE TESTED EVERY ELEMENT ON THE SITE ALREADY
Time spent testing
CR increase
detected
CR DIFFERENCE <5%

STATISTICAL SIGNIFICANCE <1%
~100 000 VISITORS REMAINING
TEST DURATION: 6 WEEKS
+1.2%

VERY SMALL UPLIFTS (1-5%)
ARE ALMOST IMPOSSIBLE TO
MEASURE !

+14.1%
STATISTICAL SIGNIFICANCE 36%
~81 481 VISITORS REMAINING
TEST DURATION: 6 WEEKS

WITH LOW TRAFFIC EVEN 10-30%
UPLIFTS ARE IMPOSSIBLE TO
MEASURE!

WHY
#2 IT’S STILL A SAFER CHOICE OVER A RADICAL REDESIGN
#1TO UNCOVER GROWTH POTENTIALS / BIGGER GAINS
#3 IT’S MORE FUN

#1 TO UNCOVER GROWTH POTENTIALS / BIGGER GAINS

TRANSFORM AND GROW YOUR
BUSINESS BY FIRST TESTING OUT
NEW IDEAS ONLINE

#2 IT’S STILL A SAFER CHOICE OVER A RADICAL REDESIGN
Original CXL Variation

#2 IT’S STILL A SAFER CHOICE OVER A RADICAL REDESIGN
Original CXL Variation
CR UPLIFT = 61%

CXL Variation CXL Variation 2
CR DECREASE = -7%

#3 IT’S MORE FUN
HYPOTHESIS LIVE TEST RESULT

YOU’LL GET THE WHOLE TEAM
EXCITED AND ON BOARD

HOW
#1MAKE SURE YOUR TESTING PROCESS IS SOLID
#2 KEEP DOING QUALITATIVE RESEARCH

1. Is the test based on a data driven hypothesis?

1. Is the test based on a data driven hypothesis?
2. Does the test design exactly match the
hypothesis?

1. Is the test based on a data driven hypothesis?
2. Does the test design exactly match the
hypothesis?
3. Has the test been setup correctly with all
goals attached?
HINT: use event tracking and universal integration both

#2 KEEP DOING QUALITATIVE RESEARCH
What’s holding you back from completing
your purchase today?
YOU’LL HAVE REAL TIME FEEDBACK
IF SOMETHING IS WRONG!

TESTING STRATEGIES
ITERATIVE INNOVATIVE
COST
RISK
POTENTIAL BENEFIT
SAMPLE SIZE
LOW
LOW
LESS < ~10%
BIGGER
HIGH (UNLESS LUCKY)
HIGH
~10% < MORE
SMALLER

People who don’t take risks
generally make about two big mistakes a year.
People who do take risks
generally make about two big mistakes a year.
Peter F. Drucker

Super profitable conversion advice FREE
every week!
Marie Polli
Twitter: @MariePolli
http://conversionxl.com/
Questions?

What's hot

[CXL Live 16] Opening Keynote by Peep Laja

CXL

Test for business growth - analyzing A/B-test with a Bayesian approach

Online Dialogue

Opticon 2017 Experimenting with Stats Engine

Optimizely

Where do You Start? Where to begin? How do I track the data? How do I setup an A/B test? When do I know if the test is conclusive? 1.The CRO Mindset 1 Not hitting goals? Do more with less. Failing is Part of the Game Follow the Data 2.Start with a Plan 2 Don’t just test to test Create a backlog 3.Collecting Data 3 Lots of Options The standard My preference They are directly integrated into most AB testing tools First we need to build our funnel KISSmetrics makes this dead simple Experiments show in funnel reports automagically LOG IN WITH GOOGLE Start Your Free KISSmetrics Trial 4.Collecting Insights 4 Qualitative is important On-page survey tools Ask questions at funnel drop oﬀ points How can we help? Are you looking for something we don't have? Do you need assistance? How can we improve? What is preventing you from purchasing? UX and Usability testing Heatmapping Look at your analytics :) Use these insights to improve backlog 5.Start Testing 5 What is an A/B Test? A/B Testing Tools Add Javascript to site

Conversion Rate Optimization 101 - Kick-Start Your Growth Engine

Kissmetrics on SlideShare

When to run how many A/B-tests (based on data) - MeasureCamp Amsterdam - Ton ...

Online Dialogue

Most optimizers aren’t big fans of the traditional redesign cycle. They promote a continuous optimizing process. And that’s fine. But sometimes websites just suck donkey balls. And a new website is a must. Also: new companies need new websites too. So how can we stop those clients from making shitty websites? Which research methods and tools can you use in which stage of the development cycle? And how do you this?

[Elite Camp 2016] Karl Gilis - How to Make Sure Your New Website Won’t Be a F...

CXL

Ton was asked to talk about things that get him excited as a web analyst looking at conversion rate optimization. He picked 5 things: - The real fun part of web analytics is analyzing how user behavior is changing (analyzing experiments), not creating campaign reports... - Win: inject your website feedback form responses into your analytics and be able to segment behavior based on goals. - Run your experiments with an automated and free GTM / GA / EXCEL results set-up! - To make sure business gets it: apply Bayesian statistics on experiment results, don't report on P values, confidence levels etc. - Bandit algoritms: use www.smartnotifications.com for automated persuasive messaging on your website.

Keynote Ton Wesseling at the Web Analytics Wednesday Copenhagen #wawcph at Se...

Online Dialogue

Optimizely Workshop: Take Action on Results with Statistics

Optimizely

If you’re a marketer it’s very likely that you’ve run an A/B test. It’s also likely that you’ve never calculated the sample size for your tests, and instead, you run tests until they reach statistical significance. If this is the case, your strategy is statistically flawed. Conforming to sample size requires marketers to wait longer for test results, but choosing to ignore it will bear false positives and lead to bad decisions. This deck was created for an email audience for there are valuable lessons for anyone who runs A/B tests.

SAMPLE SIZE – The indispensable A/B test calculation that you’re not making

Zack Notes

4 Steps Toward Scientific A/B Testing

Janessa Lantz

Everything you've learned about A/B Testing is based on the fundamentally flawed belief that there's one right answer. But the era of mass-market, one-right-answers is over. A/B Testing is our most valuable tool in the battle to create a more engaging web. But our strategy is broken. Don't worry, we can gain a better understanding of our users with a little data science. And we can reinvent A/B Testing... I will show you how. At Civis Analytics, we specialize in Data Science. From here, we can clearly see that all people are not the same. So why are A/B Tests designed to search for a single solution? In this session I'll show you where A/B Testing is headed next. See you in Austin!

SXSW 2016 - Everything you think about A/B testing is wrong

Dan Chuparkoff

Optimizely Experience Customer Story - Atlassian

Optimizely

Viacheslav Kravchuk: Conversion rate optimisation. What’s really proved to m...

Meet Magento Poland

A/B Mythbusters: Common Optimization Objections Debunked

Optimizely

Ecommerce Conversion World, London March 23 2017 - Ton Wesseling keynote

Online Dialogue

Digital Elite Camp 2016: Optimize for €€€ - Annemarie Klaassen & Tom van den ...

Online Dialogue

Opticon 2017 - Building a Culture of Experimentation at IBM

Optimizely

De-Risky Business: Techniques for Mitigating and Managing Risk

Matthew Philip

This presentation by Anna Marie Clifton, Product Manager at Yammer, covers the important topics of when to use A/B testing, how to implement it and most importantly, how to measure the results. The content is directed for software engineers who want to transition to product management, MBA's with finance/consulting background who wish to work high-tech companies as product managers and Project Managers, Marketers, and Designers who are seeking opportunities in product management.

A/B Testing with Yammer's Product Manager

Product School

Bad Experiments: The #18 Ways You’re A/B Tests are Going Wrong.

Martijn Scheijbeler

What's hot (20)

[CXL Live 16] Opening Keynote by Peep Laja

Test for business growth - analyzing A/B-test with a Bayesian approach

Opticon 2017 Experimenting with Stats Engine

Conversion Rate Optimization 101 - Kick-Start Your Growth Engine

When to run how many A/B-tests (based on data) - MeasureCamp Amsterdam - Ton ...

[Elite Camp 2016] Karl Gilis - How to Make Sure Your New Website Won’t Be a F...

Keynote Ton Wesseling at the Web Analytics Wednesday Copenhagen #wawcph at Se...

Optimizely Workshop: Take Action on Results with Statistics

SAMPLE SIZE – The indispensable A/B test calculation that you’re not making

4 Steps Toward Scientific A/B Testing

SXSW 2016 - Everything you think about A/B testing is wrong

Optimizely Experience Customer Story - Atlassian

Viacheslav Kravchuk: Conversion rate optimisation. What’s really proved to m...

A/B Mythbusters: Common Optimization Objections Debunked

Ecommerce Conversion World, London March 23 2017 - Ton Wesseling keynote

Digital Elite Camp 2016: Optimize for €€€ - Annemarie Klaassen & Tom van den ...

Opticon 2017 - Building a Culture of Experimentation at IBM

De-Risky Business: Techniques for Mitigating and Managing Risk

A/B Testing with Yammer's Product Manager

Bad Experiments: The #18 Ways You’re A/B Tests are Going Wrong.

Viewers also liked

[CXL Live 16] Optimizing Retention - The Silent Killer and King of Growth by ...

CXL

Web marketers have never faced a more dangerous time to have their visitors hit that "back" button. Facebook, Google, and even Twitter are measuring engagement and punishing the sites and brands that lose too much of their audience to an "I'm outta here" click. We've moved from a world where conversion rate optimization happens only in the purchase funnel to one where converting from a 5-second visitor to a 45-second, more engaged visitor who leaves with an answer is hugely important. In this presentation, Rand will show how Google's Rankbrain, Facebook's engagement algorithms, and other platforms' metrics are transforming this landscape and what web marketers can do to fight back.

[CXL Live 16] Fight Back Against Back by Rand Fishkin

CXL

[CXL Live 16] "Best Practices" or "Common Practices" - Which Is It? by Justin...

CXL

[CXL Live 16] How to Give Your Data an Annual Checkup by Annie Cushing

CXL

Are your executives and clients falling asleep during your testing presentations? Chances are your slide design and data visualizations are obscuring your valuable insights. With her special blend of neuroscience-based visualization principles, practical hands-on design techniques, and entertaining “tough love”, Lea will equip you with a fresh new toolbox that will get you and your data presentations remembered and acted upon.

[CXL Live 16] How To Present Your Testing Results to Get Results by Lea Pica

CXL

[CXL Live 16] How to Boost Conversions with 1:1 Personalization by Karl Wirth

CXL

[CXL Live 16] A/B Testing Pitfalls: Getting Numbers is Easy; Getting Numbers ...

CXL

[CXL Live 16] The Grand Unified Theory of Conversion Optimization by John Ekman

CXL

Ideas are great - no doubt - but what do you do once you have an entire backlog of ideas? Prioritization is a critical part when building an efficient, impactful testing program. Creating a framework to make smarter choices and thinking deeply about the key factors will make your program successful in the long term. In this session we will talk about how to create a strong prioritization process, how to keep it running, and how to constantly benchmark and optimize your process.

[CXL Live 16] What to Test Next - Prioritizing Your Tests by Pauline Marol

CXL

In this session Dr. Cugelman will focus on the core elements of motivation, to help you build interactive technologies that are engineered to push your users' emotional hot-buttons. You will enjoy a presentation on motivation and influence, followed by an interactive session on the ways that ancient emotions, play out in landing page design elements. We'll invite you to complete a quick-and-dirty psychometric test, and then discus different landing page elements that have been designed to resonate with different decision making styles, personality dispositions, and more. This session will give you actionable tips on how to design page elements that boost your users good feelings, and avoids accidentally triggering negative emotions.

[CXL Live 16] Motivational Chemistry and Susceptibility to Digital Persuasion...

CXL

[CXL Live 16] How to Create Landing Pages That Address the Emotional Needs of...

CXL

Ok, you're largely exploiting your testing bandwidth? Cool! You probably run more experiments than an average professor at university! But what do your results mean? What are you truly learning? And how do you keep track of and combine all those granular insights? In his mind-blowing talk Bart will take you on a ride along valuable psychological insights. He will show why these are your solution for accelerating your knowledge growth and even provide you with a psychology based optimization framework to acquire a truly deep understanding of what drives your customers buying journeys.

[CXL Live 16] Persuasive Journey Mapping by Bart Schutz

CXL

[CXL Live 16] From WTF to Hell Yes - How to Come Up With Copy That Persuades ...

CXL

[CXL Live 16] Hooked: How to Build Habit-Forming Products by Nir Eyal

CXL

[CXL Live 16] Neuro-Persuasion - Brain-Based Strategies for Online Marketers ...

CXL

Social Proof Persuasive Content Framework

Angie Schottmuller

Viewers also liked (16)