1. UX & Split Testing UX, MVT, A | B testing, UCD, persuasion architecture, web analytics and any other buzz words you can think of... Death by PowerPoint courtesy of Lucy Spence

3. What is it? The different types of split tests (terminology) A | B Two versions of the same thing – traffic split 50 | 50 Useful for deciding between radically different solutions A | B | n Same as A | B test but with more than 2 MVT (multivariate tests) Many different variations of many different section of a page Useful for optimising established funnels

4. What is it?What sort of things can you test? A | B | n Radically different designs Designs don’t need common elements Templates can be structured in completely distinct ways Content doesn’t need to be shared Focus on: Key messages Content structure MVT Primarily used to test different treatments of elements: Layout tweaks & positioning of elements Text amends Buttons - size, colour, style, labels & icons Removal of particular elements Focus on: Optimizing what is already working for you

5. A diversion...My first thoughts when I heard about MVT Wow. Cool. OMG! I’m going to be out of a job Hang on – no - wait (thinking, thinking, thinking...) ...maybe there’s a way I can be part of it... Maybe designers no longer design solutions but design experiments to test possible solutions Yay – we can do whatever we want! It’s play time! It doesn’t really matter anyway it’s all justrandom anyway

6. A diversion... What I’ve discovered (mostly through mistakes) MVT can be fun, informative and deliver strong commercial results MVT can also be painful, confusing, time consuming, divisive and deliver no commercial benefits Its not quite as simple as everyone makes it out to be You need to be familiar with the tools that you’re using and what is / isn’t possible Most only work on a page at a time – yet mostly we’re testing multi step funnels You need to understand statistical significance in order to get results you trust You need to be crystal clear about your goals and what you’re trying to achieve

7. Why should you do it?What are the benefits? Because on the whole we’re a strange and peculiar species We’re unpredictable - people are strongly influenced by subconscious thoughts and subtle cues in most environments We can’t simulate someone’s actual purchasing decisions We don’t have a better way of measuring the persuasiveness of different solutions Businesses can make quite a bit of money out of it without doing very much

8. Who should do it?Is it suitable for your organisation? Organisations with lots of traffic - you’ve got to get a reasonable amount of visitors through a test to get a statistically significant result Testing 2 versions of the same page with a 30% conversion rate and 1000 visitors a day will take 77 days to statistically prove a 2% improvement with 95% confidence. If you need to work it out, use a calculator: https://www.google.com/analytics/siteopt/siteopt/help/calculator.html Organisation that are making money through their site It helps clarify your goals Do you have targeted landing pages for PPC activity? If not, it’s probably not for you. Bloggers (this is an exception) - it’s a easy / cheap way to do engagement experiments such as bounce rate or commenting

9. When to use it?Can I use it for everything I do? It’s not a magic wand - it’s just a tool – use it appropriately Use A | B | n testing early on to help determine direction Use static HTML for tests, don’t bother developing complex solutions Don’t test complex solutions that you’ve already built Use MVT on your final funnel solutions after all other user testing is done. Repeat MVT until you’re struggling to come up with any other options, ie. never stop

10. How to use it?Where do you start and how do you progress Start off simply – Google’s Website Optimizer is free and easy Build up your level of complexity as you get more experience Different vendors provide different levels of sophistication Factors you might want to consider: Level of support / analysis Level of self service Segmentation Integration with other data centres & web analytics Email, PPC campaigns, mobile, other platforms Move toward the ultimate – full functional split testing / experience testing (but only a very small number will ever make it that far)

11. What to avoid?What are the pain points? Don’t test something if you’re just going to put it live anyway Multiple sources of lies can build distrust if not reconciled / understood Lift isn’t guaranteed even when testing positively Nothing remains static Seeing trends against natural variation is almost impossible Control limit analysis often doesn’t work – timescales are too long i18n / localization may produce different results

12. UCD / UX / UED How does it fit with UX? It’s not a magic wand - it’s just a tool – use it appropriately It’s about the only real way to measure persuasiveness accurately Use A | B | n testing early on to help determine direction Use static HTML for tests, don’t bother developing dynamic solutions Use MVT on your final funnel solutions after all other user testing is done. Make what works, work harder The next step – design with MVT in mind Know what elements you want to move around in the funnel

13. Contact details If you found this interesting and want to stay in touch, or have some questions: Linked in: http://uk.linkedin.com/in/lucyspence Twitter: http://twitter.com/lucyjspence Email: lucyjspence@googlemail.com And if you want a 2 month free trial for LOVEFiLM use the promo code: HBAQRRMDD