Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
The Past, Present and 
Future of Website 
Optimisation 
In association with Qubit 
Linus Gregoriadis 
Research Director 
@...
Introduction and Methodology 
The Business Case 
Foundations and Resourcing 
Moving Towards Personalisation 
| 6 | The Pas...
Introduction and Methodology 
The Business Case 
Foundations and Resourcing 
Moving Towards Personalisation 
| 7 | The Pas...
Aims of the report 
● To find out how companies are approaching the issue of 
testing and optimisation. 
● To identify the...
| 9 | The Past, Present and Future of Website Optimisation
Introduction and Methodology 
The Business Case 
Foundations and Resourcing 
Moving Towards Personalisation 
| 10 | The Pa...
| 11 | The Past, Present and Future of Website Optimisation 
Photo credit: 123dan321 on stock.xchng 
Why test anyway?
Testing is key to improving website 
performance 
| 12 | The Past, Present and Future of Website Optimisation 
Source: Con...
Testing is key to improving website 
performance 
● Companies seeing significant increases in sales 
typically run more A/...
“If it is going to fail, fail fast… 
“We see a positive benefit from around a 
third of tests.” 
Chris Howard, Head of Dig...
Case Study: Topshop 
| 15 | The Past, Present and Future of Website Optimisation 
● Wanted to improve US 
experience 
● Te...
| 16 | The Past, Present and Future of Website Optimisation 
Photo credit: 123dan321 on stock.xchng 
What to test? 
What t...
“The onus is on optimising the funnel and we 
can’t do it all at once. We’ve started having 
grown-up conversations about ...
Advanced Website Optimisation is yet to 
get over the adoption curve… 
| 18 | The Past, Present and Future of Website Opti...
Introduction and Methodology 
The Business Case 
Foundations and Resourcing 
Moving Towards Personalisation 
| 19 | The Pa...
How much to test? 
| 20 | The Past, Present and Future of Website Optimisation
“We’ve learned what degrees of success you 
can have. You can chip away and only move 
the dial so far unless you are doin...
Size matters 
| 22 | The Past, Present and Future of Website Optimisation
“Realistically we can run three to four tests a 
month simultaneously… we never get the 
volume to the extent of retail ba...
“Apart from choosing the right tools, most of 
the obstacles were related to internal culture 
and getting buy-in from int...
“Testing has been an internal 
communications piece and especially 
because A/B testing involves so many 
different teams....
Decisions on 
technology 
| 26 | The Past, Present and Future of Website Optimisation
• Bought-in software, 
either off the shelf or 
customised? 
• Built in-house? 
• Managed services 
(when needs are 
const...
Can your vendor 
grow with you? 
| 28 | The Past, Present and Future of Website Optimisation
Introduction and Methodology 
The Business Case 
Foundations and Resourcing 
Moving Towards Personalisation 
| 29 | The Pa...
“Personalisation is the most exciting 
challenge and the hardest. There’s so much 
we can do but it’s finding out what you...
Case Study: Farfetch 
| 31 | The Past, Present and Future of Website Optimisation 
● Farfetch hypothesised that 
people co...
93% of companies saw 
an uplift in conversion rates 
since implementing 
personalisation. 
| 32 | The Past, Present and Fu...
Ensure incremental 
| 33 | The Past, Present and Future of Website Optimisation 
gain
“Testing by revenue is critical to our 
business. Conversion tells only part of the 
story.” 
Simon Pritchard, Group Digit...
One to One? 
| 35 | The Past, Present and Future of Website Optimisation
“It’s the nature of fashion for consumers 
largely to behave like the herd in many ways. 
Perhaps one-to-one conversion is...
Case Study: DFS 
| 37 | The Past, Present and Future of Website Optimisation 
● Hypothesised: European 
customers may need...
Over time the majority will adopt best 
practice 
| 38 | The Past, Present and Future of Website Optimisation
Final Takeaways 
● Testing has seen a surge of 
growth but barriers still 
remain. 
● People elements are often as 
signif...
PEOPLE, PROCESS, TECHNOLOGY 
NAOSHI YAMAUCHI, CPO 
@Yamalytics
Brooks Bell 
The premier firm focused exclusively on 
enterprise-level A/B testing, optimization and 
personalization serv...
TALENT, CULTURE, PROCESS ISSUES CAUSING 
‘UTILIZATION GAP’ 
Optimization tools increasing in sophistication 
• Tracking ac...
TALENT, CULTURE, PROCESS ISSUES CAUSING 
‘UTILIZATION GAP’ 
Optimization tools increasing in sophistication 
• Tracking ac...
44 
Utilization ranks 20th out 
of 24 total categories in 
Top 
Scoring 
Assessment Brooks Bell Strategic Assessment 
Cate...
So, What can you do? 
4 TIPS ON HOW TO CLOSE THE UTLILZATION 
GAP
46 
ASK YOURSELF: 
“Do we really have 
the winning team?” 
If the answer is no, 
you’re not alone ☺. 
tip1
OPTIMIZATION TEAM 
Strategy 
Design 
Engineering 
Project 
Management 
Analytics 
Creative
EVANGELIZE & 
BUILD CULTURE 
GET 
INFLUENCERS 
RECOGNIZED IN 
TEST WINS 
tip2
WHO IS BROOKS BELL? 
WE RING A GONG EVERY TIME WE LAUNCH A TEST
CREATE A REPEATABLE PROCESS 
Standardize and then document 
campaign, dev, QA, test archives 
YOU CANNOT SCALE AND 
MAINTA...
51 
THE 
STORY 
OF TESTING 
AT BRAND X 
2011 2012 2013 2014 
COMPLEXITY 
DEMAND 
TEAM SIZE 
# OF TESTS 
PROCESS 
CULTURE 
...
tip4 
START TARGETING SEGMENTS FOR TESTING. 
IT WILL MAKE IDEATION MORE EFFECTIVE. 
DOWNLOAD! Campaign QA Checklist – bbi....
Case Study: New v. Returning 
Segment: New customers 
Goal: Increase orders 
Issue: Confused as to what 
the different fit...
54 
TIP 1: Ask yourself “Do we really have a winning team?” 
TIP 2: Get influencers recognized in test wins 
TIP 3: Create...
Who, how much, and how often? The difficult questions answered around your A/B testing strategy
Who, how much, and how often? The difficult questions answered around your A/B testing strategy
Who, how much, and how often? The difficult questions answered around your A/B testing strategy
Who, how much, and how often? The difficult questions answered around your A/B testing strategy
Who, how much, and how often? The difficult questions answered around your A/B testing strategy
Prochain SlideShare
Chargement dans…5
×

Who, how much, and how often? The difficult questions answered around your A/B testing strategy

1 013 vues

Publié le

Qubit's webinar with Econsultancy and Brooks Bell gave an honest insight into A/B testing challenges faced by Topshop, Shop Direct , Gatwick Airport and more!

  • Soyez le premier à commenter

Who, how much, and how often? The difficult questions answered around your A/B testing strategy

  1. 1. The Past, Present and Future of Website Optimisation In association with Qubit Linus Gregoriadis Research Director @LinusGreg
  2. 2. Introduction and Methodology The Business Case Foundations and Resourcing Moving Towards Personalisation | 6 | The Past, Present and Future of Website Optimisation
  3. 3. Introduction and Methodology The Business Case Foundations and Resourcing Moving Towards Personalisation | 7 | The Past, Present and Future of Website Optimisation
  4. 4. Aims of the report ● To find out how companies are approaching the issue of testing and optimisation. ● To identify the barriers to optimisation. − Technological − Cultural ● To understand the processes and approaches used. ● To highlight any potential future challenges. | 8 | The Past, Present and Future of Website Optimisation
  5. 5. | 9 | The Past, Present and Future of Website Optimisation
  6. 6. Introduction and Methodology The Business Case Foundations and Resourcing Moving Towards Personalisation | 10 | The Past, Present and Future of Website Optimisation
  7. 7. | 11 | The Past, Present and Future of Website Optimisation Photo credit: 123dan321 on stock.xchng Why test anyway?
  8. 8. Testing is key to improving website performance | 12 | The Past, Present and Future of Website Optimisation Source: Conversion Rate Optimisation Report 2014
  9. 9. Testing is key to improving website performance ● Companies seeing significant increases in sales typically run more A/B and multivariate tests Source: Conversion Rate Optimisation Report 2014 | 13 | The Past, Present and Future of Website Optimisation
  10. 10. “If it is going to fail, fail fast… “We see a positive benefit from around a third of tests.” Chris Howard, Head of Digital, Shop Direct | 14 | The Past, Present and Future of Website Optimisation
  11. 11. Case Study: Topshop | 15 | The Past, Present and Future of Website Optimisation ● Wanted to improve US experience ● Tested using size formats more familiar to North American shoppers ● Measured not on conversion but revenue optimisation ● Increased revenue by 4.5% with 99% statistical significance
  12. 12. | 16 | The Past, Present and Future of Website Optimisation Photo credit: 123dan321 on stock.xchng What to test? What to aim for?
  13. 13. “The onus is on optimising the funnel and we can’t do it all at once. We’ve started having grown-up conversations about what is the best point to focus optimisation on.” Daniel Sale, Head of Digital Marketing, Investec Private Bank | 17 | The Past, Present and Future of Website Optimisation
  14. 14. Advanced Website Optimisation is yet to get over the adoption curve… | 18 | The Past, Present and Future of Website Optimisation
  15. 15. Introduction and Methodology The Business Case Foundations and Resourcing Moving Towards Personalisation | 19 | The Past, Present and Future of Website Optimisation
  16. 16. How much to test? | 20 | The Past, Present and Future of Website Optimisation
  17. 17. “We’ve learned what degrees of success you can have. You can chip away and only move the dial so far unless you are doing something significantly wrong.” Paul Wishman, Group Ecommerce Director, LV | 21 | The Past, Present and Future of Website Optimisation
  18. 18. Size matters | 22 | The Past, Present and Future of Website Optimisation
  19. 19. “Realistically we can run three to four tests a month simultaneously… we never get the volume to the extent of retail banks for example. We need statistically significant samples and that takes time.” Daniel Sale, Head of Digital Marketing, Investec Private Bank | 23 | The Past, Present and Future of Website Optimisation
  20. 20. “Apart from choosing the right tools, most of the obstacles were related to internal culture and getting buy-in from internal stakeholders.” David Paice, Group Ecommerce and Admissions Director, Merlin Entertainments | 24 | The Past, Present and Future of Website Optimisation
  21. 21. “Testing has been an internal communications piece and especially because A/B testing involves so many different teams.” Rachel Waller, Senior Digital Marketing and Social Manager, Farfetch | 25 | The Past, Present and Future of Website Optimisation
  22. 22. Decisions on technology | 26 | The Past, Present and Future of Website Optimisation
  23. 23. • Bought-in software, either off the shelf or customised? • Built in-house? • Managed services (when needs are constantly changing)? | 27 | The Past, Present and Future of Website Optimisation
  24. 24. Can your vendor grow with you? | 28 | The Past, Present and Future of Website Optimisation
  25. 25. Introduction and Methodology The Business Case Foundations and Resourcing Moving Towards Personalisation | 29 | The Past, Present and Future of Website Optimisation
  26. 26. “Personalisation is the most exciting challenge and the hardest. There’s so much we can do but it’s finding out what you should and what will add value.” Rachel Waller, Senior Digital Marketing and Social Manager, Farfetch | 30 | The Past, Present and Future of Website Optimisation
  27. 27. Case Study: Farfetch | 31 | The Past, Present and Future of Website Optimisation ● Farfetch hypothesised that people convert more when they understand the business model ● Segment of users: new visitors with over 10 page views in their session ● A message layer slides in, prompting users to read FAQs ● Result: 17% increase in conversion
  28. 28. 93% of companies saw an uplift in conversion rates since implementing personalisation. | 32 | The Past, Present and Future of Website Optimisation Source: Conversion Rate Optimisation Report 2014
  29. 29. Ensure incremental | 33 | The Past, Present and Future of Website Optimisation gain
  30. 30. “Testing by revenue is critical to our business. Conversion tells only part of the story.” Simon Pritchard, Group Digital Director, Arcadia | 34 | The Past, Present and Future of Website Optimisation
  31. 31. One to One? | 35 | The Past, Present and Future of Website Optimisation
  32. 32. “It’s the nature of fashion for consumers largely to behave like the herd in many ways. Perhaps one-to-one conversion isn’t necessary. If you can identify groups of people, that’s good enough.” Stuart McMillan, Deputy Head of Ecommerce, Schuh | 36 | The Past, Present and Future of Website Optimisation
  33. 33. Case Study: DFS | 37 | The Past, Present and Future of Website Optimisation ● Hypothesised: European customers may need more information when shopping on UK websites ● Targeted users from outside the UK with a layer informing them that DFS shipped to their country ● Result: 23% uplift in conversions for segment
  34. 34. Over time the majority will adopt best practice | 38 | The Past, Present and Future of Website Optimisation
  35. 35. Final Takeaways ● Testing has seen a surge of growth but barriers still remain. ● People elements are often as significant as technological ones. ● A pragmatic yet continual approach is most likely to pay future dividends. | 39 | The Past, Present and Future of Website Optimisation
  36. 36. PEOPLE, PROCESS, TECHNOLOGY NAOSHI YAMAUCHI, CPO @Yamalytics
  37. 37. Brooks Bell The premier firm focused exclusively on enterprise-level A/B testing, optimization and personalization services. SOME OF OUR CLIENTS
  38. 38. TALENT, CULTURE, PROCESS ISSUES CAUSING ‘UTILIZATION GAP’ Optimization tools increasing in sophistication • Tracking across platforms • Personalization • Mobile • More data! 42 Capabilities
  39. 39. TALENT, CULTURE, PROCESS ISSUES CAUSING ‘UTILIZATION GAP’ Optimization tools increasing in sophistication • Tracking across platforms • Personalization • Mobile • More data! 43 Insufficient Team & Talent • Business savvy analysts • Technical talent • Data savvy executive influencers Capabilities Broken Process • Lack of standards • Minimal documentation Resistant Culture • Gut-driven decision making • Minimal training & sharing UTILIZATION GAP Use DRIVING ISSUES
  40. 40. 44 Utilization ranks 20th out of 24 total categories in Top Scoring Assessment Brooks Bell Strategic Assessment Categories Worst Scoring 24
  41. 41. So, What can you do? 4 TIPS ON HOW TO CLOSE THE UTLILZATION GAP
  42. 42. 46 ASK YOURSELF: “Do we really have the winning team?” If the answer is no, you’re not alone ☺. tip1
  43. 43. OPTIMIZATION TEAM Strategy Design Engineering Project Management Analytics Creative
  44. 44. EVANGELIZE & BUILD CULTURE GET INFLUENCERS RECOGNIZED IN TEST WINS tip2
  45. 45. WHO IS BROOKS BELL? WE RING A GONG EVERY TIME WE LAUNCH A TEST
  46. 46. CREATE A REPEATABLE PROCESS Standardize and then document campaign, dev, QA, test archives YOU CANNOT SCALE AND MAINTAIN MOMENTUM WITHOUT PROCESS & STRUCTURE tip3
  47. 47. 51 THE STORY OF TESTING AT BRAND X 2011 2012 2013 2014 COMPLEXITY DEMAND TEAM SIZE # OF TESTS PROCESS CULTURE DOWNLOAD! Campaign QA Checklist – bbi.bz/qa-2014
  48. 48. tip4 START TARGETING SEGMENTS FOR TESTING. IT WILL MAKE IDEATION MORE EFFECTIVE. DOWNLOAD! Campaign QA Checklist – bbi.bz/qa-2014
  49. 49. Case Study: New v. Returning Segment: New customers Goal: Increase orders Issue: Confused as to what the different fits mean CONTROL CHALLENGER
  50. 50. 54 TIP 1: Ask yourself “Do we really have a winning team?” TIP 2: Get influencers recognized in test wins TIP 3: Create a repeatable process TIP 4: Start targeting segments for testing

×