Conducting the Experimentation Orchestra

1
Conducting the
Experimentation Orchestra
Jill Brown Thomas
Wolverine Worldwide
David J. Neff
Clearhead, part of
Accenture Interactive
How a traditional wholesale company is building a
testing program across 12 brands.
Wes McCabe
Optimizely

2
OPTIMIZELY.COM/DIGITAL-CHANGE-AGENTS

3
Housekeeping
! We are recording this webinar
! You’ll receive the recording and slides
! We’ll answer all questions at the end

4
Agenda
! Introduction
! What is Optimizely?
! Introducing Clearhead
! Experimentation at Wolverine
! Q&A

5
Digital Experience Optimization:
Digital Products, commerce & campaigns
Up to 5X Increase in Yield:
Revenue, share of wallet, funnel conversion,
risk mitigation, ops efficiency
Partner of Choice:
Work with leading global enterprises & “digital
disruptors” including 26 of F100
OUR COMPANY
Digital Experimentation Platform:
Next gen “Test and Learn” system
Replaces Digital Guesswork:
with evidence-based optimization
Speeds Innovation & Optimization:
Single platform for marketing & product teams
Best-in-class stats & machine learning
Consumer-grade usability
Enterprise program management & prof. services
OUR SOLUTION

6
“Our success is a function of
how many experiments
we do per year, per month,
per week, per day.”
“Instead of saying ‘I have an
idea,’ what if you said
‘I have a new hypothesis,
let’s go test it.’”
“Our company culture
encourages experimentation
and
free flow of ideas.”
“One of the things I’m
most proud of, and I think
what is the key to our
success, is this testing
framework we’ve built.”
Experimentation is the
Next Great Business
Transformation
Jeff Bezos Larry Page
Mark Zuckerberg Satya Nadella
The Surprising
Power of Online
Experiments

7
10x more experiments
Consumer-grade usability
Open data integration
Maximum yield of business value
UX and feature-level experiments
and personalization at every digital
touch point
Enterprise-wide
management & governance
Captures, governs and shares
ideation, analyses & results
World’s most trusted outcomes
BestinclassStatsEngine
FasttimetoresultsviaML
Accelerates digital innovation
Speeds dev ops & deployment
De-risks continuous feature delivery
Ensures success of new features
Unifying flagging & experiments
enables controlled testing of new
features while maintaining high
performance
Ideate
Manage
StoreGovern
Analyze
Share
Open Data
Integration
Security &
compliance
Stats
Engine
Stats
Accelerator
Consumer-
grade
usability
APIs &
Developer
Tools
Feature
Flags
Open source
SDKs
X-Channel
Full Stack
Experimentation
Personali-
zation
Recommen-
dations
Web
Experimentation
PRODUCTS
COMMERCE
CAMPAIGNS
Optimizely X Unlocks the Experimentation Best Practices
of the World’s Greatest Digital Companies

8
26 of the fortune 100 have chosen Optimizely to drive their digital experience
We’re Proud to Work With Great Global Enterprises

9
B U S I N E S S V A L U E
VELOCITY/VOLUME
LEVEL 1
Executional
Start
LEVEL 2
Foundational
Growth
LEVEL 3
Cross-Functional
Advancement
LEVEL 4
Operational
Excellence
LEVEL 5
Culture of
Experimentation
Our Products and Services Take
You on Your Experimentation Journey

12
Background
From brand marketing manager to content practitioner, then
user experience lead to optimization director – each step has
been driven by an interest in advancing the consumer
experience in a measurable and impactful way.
Looking forward
Leading the organizational transition from fractured,
subjective experience strategy to an experience optimization
program built on data and consumer insight.
DIGITAL
EXPERIENCE
OPTIMIZATION
CONDUCTING THE
EXPERIMENTATION ORCHESTRA
Jill Brown Thomas
Director, Digital Experience Optimization
Wolverine Worldwide

13
01 Wolverine World Wide
02 Test Prioritization
03 Case Studies
04 A Cultural Shift
DIGITAL
EXPERIENCE
OPTIMIZATION
CONDUCTING THE
EXPERIMENTATION ORCHESTRA
This session will outline my experience building an optimization center of
excellence to support a portfolio of diverse footwear brands.
Our optimization journey has covered quite a bit of ground so far – from
alignment on people to buying technology and building process, we’ve
learned a lot about what works (and what doesn’t). And we continue to
advance our agenda to create a culture always testing and driven by
lessons learned.

14
01 WHO IS WOLVERINE WORLD WIDE?
CONDUCTING THE EXPERIMENTATION ORCHESTRA
+ A company with a rich history (130+ years) of
manufacturing and selling footwear to retailers and
distributors
+ Offices in Rockford, MI | Waltham, MA | London
+ Revenue of $2.35B in 2017; sold in 200 countries
and territories globally
+ Family of 12 owned and licensed brands, all on one
core commerce platform (Salesforce Commerce
Cloud) but managed by commercial owners

15
Experimentation has emerged as a priority for two reasons:
1. Data-driven support for core UX updates; prevent experiences that aren’t proven to add value.
+ The Optimization team is a central team reporting into the Technology + Experience group. We inform and drive
improvements to the core platform.
2. Create best practice “standard operating procedures” for the commercial functions
+ eCommerce at WWW is comprised of 8 brand-focused commercial teams (focused on content, merchandising)
01 WHO IS WOLVERINE WORLD WIDE?
+ 2017 was a year of transformation; focused on innovation, efficiency and driving a deeper connection with our
consumer.
+ A key element of our 2018 Global Growth Agenda is a digital-direct offense seeking to drive our owned eCommerce
growth beyond 20% through more effective digital engagement.
+ Experience Optimization is a key strategy in the plan to achieve this goal.

16
WHERE DO IDEAS COME FROM?
02 TEST PRIORITIZATION
+ Optimization team research
+ Clearhead analysis
+ Commercial/business questions
+ Organizational priorities

17
• Builds skill in creating more actionable, data-driven hypotheses by providing a
common format and baseline score to aid in prioritization discussion between teams.
• Helps testing scale by prioritizing time to focus on the most impactful hypotheses.
HOW ARE IDEAS SCORED?
Now part of Accenture Interactive
• As you scale up the number of tests you consider you will need a solid prioritization
scoring matrix to help your optimization team.
• Our matrix has simple, customizable inputs: what’s the hypothesis, what problem
does it map to, how long will it run, and what pages and device types will be
impacted.
• Our matrix then asks your testing team to add in: what’s the estimated level of effort
to run this idea, what data supports it, and is this idea applicable to other sites.
• The matrix then uses estimated revenue per page per brand along with average
conversion rates and level of effort to determine an average score for these ideas
from 1-100.
• This final score (which can be weighted with other inputs) helps determine what
hypotheses should be prioritized for which tests to run.

18
WHAT DO WE MOVE ONTO THE ROADMAP?
+ Formula is a mix of science and art…
+ High opportunity scores
+ Current resource level
+ Open/available space to test
+ Leadership or organizational priorities
+ Currently seeking to understand the mix of available
traffic across pages and sites + resources to keep test
pipeline at capacity
+ Next Steps: Set appropriate expectations of team for
response and reactiveness to ideas; review
opportunities for testing v. user insight

19
Case study #1: Mobile Homepage
03 KEY LEARNINGS
+ Problem: Homepage strategy is a key question for our
teams as they balance brand and commerce interests
+ Signal: In Sperry heatmaps, we noticed a very high level of
engagement on secondary category links.
+ Hypo: I believe that presenting the user with large
category buttons on the mobile homepage, above the
hero, will allow users to more easily navigate to product. If
I’m right, we’ll see an increase in PDP views and
downstream metrics including order conversion.
v1
v0
v2

20
Case study #1 : Mobile Homepage
03 KEY LEARNINGS
+ Result: Flat result between control and v1,
although v1 saw much higher engagement. V2
experienced decline in PDP views and CVR.
+ Learning: Small changes can have a big impact
– not only to site performance, but to test
performance
+ Next steps: Currently retesting on Merrell with a
new link set. Testing on other sites to gather
insight on impact to lifestyle brands
v1
v0
v2

21
Case study #2 : Promotion slider
03 KEY LEARNINGS
+ Problem: How do we prioritize promotion messaging
while maintaining user experience?
+ Signal: We receive a steady stream of OpinionLab
comments on the difficulty of interacting with the site
with an open promo area.
+ Hypothesis: I believe that adjusting the rules so that
the promotion area is noticeable, but less aggressive,
we’ll see decrease in bounce as well as an increase in
order conversion rate and engagement with
promotions.
v1
v0

22
Case study #2 : Promotion slider
03 KEY LEARNINGS
+ Results: Control won by convincing margin, showing
that sometimes the conversion driver isn’t the
“better” experience.
+ Learning: We recommended the rest of the portfolio
open their promotion area for the highly promotional
holiday season. Losing tests can have winning
impact, especially in a multi-site business.
+ Next steps: Test different methods of creating a
better user experience while maintaining productivity
of this space.
v1
v0

23
Case study #3 : Cart
03 KEY LEARNINGS
+ Problem: Our checkout flow is cluttered, offers too
many CTAs and messages that distract and
overwhelm the user.
+ Signal: Benchmark data from Salesforce shows we
aren’t keeping pace with our peer set in this area.
+ Hypotheses: I believe that reducing the size and
number of CTAs on the cart page, simplifying the
experience, will reduce user distraction, leading to a
reduction in cart abandonment and ultimately an
increase in Order Conversion Rate.
v0
v1

24
Case study #3 : Cart
03 KEY LEARNINGS
+ Result: Mobile is a clear win! Desktop has been more
complicated and is still testing.
+ Learning: Promotion area placement can have a
significant impact on overall test results.
+ Next steps: With mobile and desktop tests wrapped,
we’ll create a promotion field recommendation that
takes device and seasonality into account.
v0
v1

25
04 A CULTURAL SHIFT
In 12 months, we’ve…
+ On-boarded both new agency and testing tool; ran ~40 tests
+ Installed Optimizely on 6 sites; tests actively running on 4
+ Created Q2 test roadmap for internal team to takeover from Clearhead
team
+ 80% complete with Clearhead consulting engagement; data science and
technology recommendations complete. Executing education and cultural
elements in coming weeks.
+ Created a new business function and secured funding for a team of 4
2018

26
In the next 12 months, we’ll…
04 A CULTURAL SHIFT
+ Complete hiring and rollout new
processes with a full team:
+ Optimization Manager (in-process)
+ Designer
+ Front-end developer
+ Analyst (in-process)
+ Educate eCommerce team, across all
campuses, via optimization training
modules and field guide
+ Create data-driven Optimization
best practice reports for use as
commercial experience
guidelines.
+ Create accountability and
reporting methods; regularly and
transparently track progress to
Optimization ROI goals
+ Execute against robust testing
roadmap
+ Checkout with CH on high traffic
sites
+ Navigation, homepage, PLP, PDP
with internal team across 6 sites
+ Clearhead relationship evolves to
address strategic opportunities –
data-driven design and
personalization

Conducting the Experimentation Orchestra

Conducting the Experimentation Orchestra

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Conducting the Experimentation Orchestra

Similaire à Conducting the Experimentation Orchestra (20)

Plus de Optimizely

Plus de Optimizely (20)

Dernier

Dernier (20)

Conducting the Experimentation Orchestra