This document discusses user experience (UX) testing, specifically A/B testing. It provides an overview of A/B testing, including that it involves creating two versions of a page and splitting traffic between them to see how users interact. The document outlines why testing is important, different types of testing tools, considerations for running a successful A/B test such as defining goals and metrics, and examples of famous A/B tests. It stresses that testing takes time and one should not expect overnight results.

1. YOU SHOULD TEST THAT!
Kelley Howell, UX Product Manager

2. Why should we test?
 How we think users will act is often
different than how they really act
 It is the best way to settle disputes
about designs

3. Why should we test?
Testing provides a direct way for us to
identify users and their interactions with
the website

4. Other tools to find out how we’re
doing
 Usability Testing
 Field Studies
 Eye Tracking
 Click Mapping
 Multivariate Testing
 A/B Testing

5. What is A/B Testing?

6. A/B Testing is…
• Two versions of a page are created
• Traffic is split between versions
• Users have a 50% chance of seeing
one of the versions

7. It is important that only one element on
a page is changed. Any more than
that creates a multivariate test which
has a different set of testing
parameters.

8. Advantages of A/B Testing

9. Easy to create and implement tests

10. Flexibility in choosing testing objectives

11. Can be adapted for any traffic numbers

12. Measures actual behavior of users

13. Popular A/B Testing tools

15. How to A/B Test

16. Before you start testing
• Prioritize pages
• Define website goals
• Create an hypothesis of what you expect
to occur
• Define which metrics you are going to
measure and what those metrics are
going to measured against

17. Decide which one thing you want to test

18.  Know what variable you are testing
for
Determine which variable to test

19. Redacted
 Create alternatives
Create Alternatives

20. Redacted
Devote time to let results mature

21. Redacted
Test, test, and keep testing!

22. What Can you A/B Test?

23. Call to Actions

24. Microcopy

25. Page Layout

26. Form Length

27. A/B testing can’t answer everything

28. Famous A/B Tests

29. Fact: A simple change was worth 300
Million Dollars

30. Fact: Google infamously tested 41

31. Fact: This A/B test made A/B testing cool

32. Things that can ruin your A/B test

33. Selecting Certain Population Sources

34. Ignoring Temporal Effects

35. Stopping Your Test Early

36. Disadvantages in A/B Testing

37. 1) Results change as time marches
forward

38. A/B testing: one at a time

39. At a certain level of maturity, dramatic
results usually require dramatic design
tests

40. 4) Not all A/B Testing Goals are testable

41. A/B Testing in a nutshell

42.  Define your goals
 Make up a hypothesis to test
 Create a variation of your original
 Set goals on what to fine
 Run you test for long period
 Test, and keep testing to
see if result holds true

43. That’s a Wrap!
Remember: Testing takes time, don’t expect
overnight miracles– and if that happens

44. For more research-based insights, check out the UX insights
portal: http://redacted.com
THOUGHTS?
QUESTIONS?