Ecommerce Conversion World, London March 23 2017 - Ton Wesseling keynote

SUBTITLE BELOW
ONLINE EXPERIMENTS
About how amazing this is,
but also a lesson
when to use it in your growth strategy

Ton.Wesseling@onlinedialogue.com
Since 1999
Ton Wesseling
@TonW

Challenge

Optimisation

InnovationOptimisation

We believe

We believe in

First Party Data

Analysing
First Party Data

Running
Online Experiments
on First Party Data

Online Experiments

Digital growth optimisation
Analytics Psychology

Business cases of our clients
Investment Revenue after implementation in 6 months
Company X €82.000 €1.200.000
Company X €89.000 €1.800.000
Company X €150.000 €5.500.000
Company X €166.000 €5.800.000
Company X €230.000 €16.500.000

InnovationOptimisation
Online Experiments

Prioritize, manage and learn

“If you double the number of
experiments you do per year
you’re going to double your
inventiveness”
Jeff Bezos, CEO Amazon

“Our success at Amazon
is a function of
how many experiments
we do per year, per month, per
week, per day…”
Jeff Bezos, CEO Amazon

How many
online experiments
are you running
per year?

When should you run how many
online experiments
and what kind of experiments
do you need to run?

the
ROAR
model

Optimisation phases - ROAR
Time span
Conversionspermonth

Time span
Conversionspermonth
Risk

Time span
Conversionspermonth
Risk + Optimisation

Time span
Conversionspermonth
Risk + Optimisation + Automation

Time span
Conversionspermonth
Risk + Optimisation + Automation Re-think

Time span
Conversionspermonth
1.000 conversions
per month

Time span
Conversionspermonth
10.000 conversions
per month
1.000 conversions
per month

When could we experiment?
Map out your different audiences and
determine:
Can I run a test on this page with a
Power of >=80%?

Power
“Statistical power is the likelihood that an
experiment will detect an effect when there
is an effect there to be detected”.
Depends on:
• Sample size
• Effect size
• Significance level

Start calculating abtestguide.com/calc

Power: 64,7%
Start calculating abtestguide.com/calc

Power & Significance
Power
Only test on pages with a high Power (>80%) 
otherwise you don’t detect effects when there is an
effect to be detected (False negatives).
Significance
Test against a high enough significance level (90% or
95%)  otherwise you’ll declare a winner, when in
reality there isn’t an effect (False positives).

Time span
Conversionspermonth
10.000 conversions
per month
1.000 conversions
per month
15% impact
needed
5% impact
needed
Impact needed: all conversions with an average conversion rate of 2%, a test length of 3 weeks and a power of 80%

Phase 1: bridge the gap – Risk

Get out of the office

Risk

Yes, I’ve got those 1000+ conversions
Time span
Conversionspermonth
1.000 conversions
per month
10.000 conversions
per month

With 1000+ conversions a month
Your test capacity is max.
20+ tests a year
with a statistical power of 80%
on predicted uplifts of 15%

Experiments with measureable impact
On average
1 out of 3
Simply truth: you’re just not always able to create a winner with enough impact

So in this case, you’ve got impact
Every 7 or 8 weeks

Slower? Energy will run out!
And your continuous optimisation program will die

What to test? Business cases!
Delivery –test with
in stock. 4 weeks
or 2 days?

Service charge, do
it or not? Include in
price or not?

ROAR – Optimisation – moving up
Time span
Conversionspermonth
1.000 conversions
per month
10.000 conversions
per month

Tests that can be done

ROAR – Optimization – moving up
Time span
Conversionspermonth
1.000 conversions
per month
10.000 conversions
per month

Now you’re testing
On your way to
4 tests a week
Max capacity with 1 teammember on each discipline: analyst, ux, developer, psychologyst

Test capacity
maximum number of
experiments per year
that your site or app
can handle

Regret
amount of money
you are losing per year by
not using your full test capacity

A/B-test development resources

ROAR – Automation
Time span
Conversionspermonth
1.000 conversions
per month
10.000 conversions
per month

Automation

DEEP
LEARNING
BAYESIANEVOLUTIONARY
ALGORITHMS
NEURO
EVOLUTION
Sentient
Ascend™

+ Optimisation + Automation
ROAR – Automation
Time span
Conversionspermonth
Risk Re-think
1.000 conversions
per month
10.000 conversions
per month

ROAR – You don’t stop optimising
Time span
Conversionspermonth
1.000 conversions
per month
10.000 conversions
per month

ROAR – Re-think
Time span
Conversionspermonth
1.000 conversions
per month
10.000 conversions
per month

Experiments with significant insights
months

CRO brings user knowledge

Customer knowledge center
Making money from conversion improvements
Creating more and more customer insights
Which can be monetized
on even a larger scale

Bridging the gap

SUBTITLE BELOW
Slides PDF & links to mentioned tools:
http://ondi.me/ECW2017
Twitter: @tonw
Linkedin: tonweseling
Email: ton@onlinedialogue.com

Ecommerce Conversion World, London March 23 2017 - Ton Wesseling keynote

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (10)

Similaire à Ecommerce Conversion World, London March 23 2017 - Ton Wesseling keynote

Similaire à Ecommerce Conversion World, London March 23 2017 - Ton Wesseling keynote (20)

Plus de Online Dialogue

Plus de Online Dialogue (20)

Dernier

Dernier (20)

Ecommerce Conversion World, London March 23 2017 - Ton Wesseling keynote

Notes de l'éditeur