This document summarizes a presentation on data-driven UI/UX design and A/B testing. It discusses how Obama raised $60 million through simple experiments on his website, testing different button text and media campaigns. The presentation outlines a 4-step process for A/B testing: 1) using data to identify problems, 2) forming hypotheses, 3) running experiments, and 4) evaluating results and deciding next steps. It emphasizes establishing key metrics and sample sizes to determine statistically significant results. Iteration is important - new hypotheses should be based on learnings. Regular testing allows continuous improvement even with small changes.

1. Data-Driven UI/UX Design
TopDev Event, Aug 2017
Jack Nguyen, Country Manager VN & TH

2. Rules for today?
No rules
Engaged & Ask questions
Quiz & Discussion

3. Truly understand the value
What I hope
Clear approach to get started
Practical do & don’t

4. Obama raised $60 Million by
Running a Simple Experiment.

5. Obama raised $60 Million by Running a Simple Experiment.
Challenge: Raise money if no one has heard of you and you only have a
website?
Idea: Every visitor to the site is an opportunity to generate lead, convert
them into volunteer for donation

6. Optimize the sign up pages to get more leads

7. Experiment different variations of CTA buttons

8. Experience different variations of Media
Campaign staff’s favorite

9. … And the winner is?
E-mail Sign up Rate
11.6%
+40.6%
Vs. Original 8.26%
10% of email subscribers convert into volunteers
21$ donation per volunteers on average
+2,880,000 e-mails
+60 millions in donation

11. A scientific method to measure how a new idea
(version B/variation) performs against an
existing implementation (version A/control)

12. Where does this belong to?
● Conversion Rate Optimization (CRO)
● A/B Test: Best 1 for All
● Segmentation: Best 1 for Some
● Personalization: Best 1 for 1
Our focus for today!

13. Why are we doing this?

14. HiPPO
Highest Paid Person’s Opinion
Highest Paid Person’s in the Office
Higher-paid or higher-rank employees have the biggest say when a decision has to be made.
Describe an organization's reliance on human instinct rather than data in the decision-making process.

15. “My design is much more beautiful and intuitive compared to what we
have now! It’s so obvious, don’t think we need to run an A/B test”
“My wife (of the boss) told me that this banner is confusing. Let’s just
remove it!”
“Let’s put our team focus on launching this feature. This is gonna be
huge!”

16. QUIZ TIME!

26. Why is it important for YOU?

27. As a Product Manager ...

28. “I just need to redesign my website”

29. Why is it important for your business?

30. Increasing Marketing Cost
How much longer can you survive this battle?

31. 3x Increase in CPC in 2016 vs 2008

32. All in the past 4 months

33. Low Conversion Rate
How much money are you leaving on the table?

36. Opportunity for you to gain competitive
advantage!

37. How does it work?

38. Users are randomly splitted

39. Data is generated when users browse through
pages and do their tasks

40. When there is enough data to make a decision,
evaluate the results and make a decision
(A or B)

41. The moment of truth ...

42. Data-driven action means ...

43. How do I start?

44. 4 Steps Framework of A/B Testing
Step 1
Data
Figure out what
to improve
Step 2
Hypothesis
Make an
educated guess
Step 3
Experiment
Test
your guess
Step 4
Act
Decide what’s
next

45. Step 1: Data
Figure out what to improve

48. Data helps you identify the most painful
problems people have (NOT the one bother
you the most)
Find the problems worth solving!

49. Quantitative Data
Tells you where to test

50. What kind of quantitative data you need?
● User’s page browsing behaviour
● User’s feature interaction behaviour
● User’s marketing source information
● User’s historical purchasing behaviour
● ...

51. Analytics Platform

52. Find pages with HIGH traffic and HIGH bounce rate
● Landing pages in top 10-20
with bounce rate much higher
than site average
● This is where you are burning
your marketing dollars without
caring about the full user’
journey

53. Find pages with HIGH traffic but LOW page value
● These are the pages with highest traffic
going through
● But does not contribute much to
conversion action
● How to make it work for you?

54. Find the MOST painful areas on the buying journey
● Identify the steps with highest
drop-off rate to optimize
● Understand where users go to
next

55. Find the MOST interacted features with LOW Conversion Rate

56. Qualitative Data
Give you an idea of what should be tested

57. What kind of qualitative data you need?
● User testing (video of how users browse your site)
● Survey data (for detailed feedbacks)
● Heat mapping (quick glance at elements interaction)
● Customer service
● ...

58. A few qualitative tools

59. Use exit surveys to learn why people leave
Apply this survey on the funnel step with the highest drop-off

60. Use post-purchase survey to understand why people buy
Know what drive conversions to know your selling points, customer’s perception to drive
conversions for others

61. Get feedback on the goal of discovery pages

62. Type of A/B Test

63. Different A/B Test type
● Element testing
● Multivariate testing (MVT)
● Page testing
● Flow testing

64. Local Maxima vs Global Maxima

65. This is what
you should
have by
now!
… Now
What?

66. Prioritize your testing roadmap for maximum results

67. 1. Population Size
3 Key Criterias
2. Potential Performance Improvement
3. Estimated Technical Difficulty to Set up
You can assign a score for each of this and relevant weightages

68. Estimate potential value of a Test

69. Step 2: Hypothesis
Make an educated guess

70. What is a hypothesis?

71. What’s the aim of having a hypothesis?
The hypothesis should:
● Capture the essence of the change you propose to make and what you
think the effect will be
● A clear understanding and a plan that addresses what you will learn by
testing the hypothesis

72. Ask yourself these questions when crafting hypothesis
● If you fail, what did you learn that you will apply to future designs?
● If you succeed, what did you learn that you will apply to future
designs?
● How much work are you willing to put into your testing in order to
get this learning?

73. Structure of a good hypothesis

74. Structure of a good hypothesis

75. Structure of a good hypothesis

76. Is this a good hypothesis?

77. Is this a good hypothesis?

78. Step 3: Experiment
Test your educated guess

79. LIFT Model
Look at marketing touch points through the user perspective
to design test variations

80. Landing page Influence Function for Tests
Widerfunnel concept

81. Widerfunnel example

82. Widerfunnel example

83. Widerfunnel example

84. Widerfunnel example

85. Widerfunnel example

86. Widerfunnel example

87. Widerfunnel example

88. Key KPIs
Without this, NEVER start your experiments!

89. 2 Types of KPIs
Primary metrics: The key metrics that will be directly impacted if the
hypothesis hold true.
Example: If you A/B test a clearer CTA message, then: KPI = CTR on this
CTA
Secondary metrics: The indirect but important metrics that would be resulted if
the hypothesis hold true.
Example: If you A/B test a clearer CTA message, then: KPI = CR (as an users
would be more likely to checkout due to clearer messaging)

90. Good vs Bad KPIs
Bad KPIs:
● No. of pageviews
● Time spend on pages
● No. of sign-ups
● No. of transactions
● ...
Good KPIs:
● Share of page views with
at least 75% scroll
● Share of page views with
at least 10 mins. on page
● Sign-up Rate
● Conversion Rate
● ...
2 common problems:
● Vanity metrics
● Not an effectiveness metrics

91. Step 4: Act
Results evaluation & Decide what’s next

92. How to know if the
result is real?

94. What is statistically significance?
To be safer!
Less likely to
be wrong!

95. Confidence Interval (CI)
P = Conversion Rate
Z will be calculated based on significance level
The more confidence you want to be that the
difference is real → The bigger statistical
significance → The bigger z will be
Hence, if you want to have a more confident
range of the actual CR, the bigger sample size
will be required

96. Z - Score Table

97. Launch or keep testing?
1.3%
1.6%
1.7%
2.5%
Control
Var A
Var B
Var C
Conversion Rate
Confidence Interval

98. If you not only want to be certain of the change but also of the magnitude of the change
Minimum Detectable Effect (MDE)
The minimum relative change in conversion rate you would like to be able to detect - MDE
You’ll need less traffic to detect big changes and more traffic to detect small changes
To demonstrate, let’s use an example with a 20% baseline conversion rate and a 5% MDE.
Based on these values, your experiment will be able to detect 80% of the time when a variation
underlying conversion rate is actually 19% or 21% (20%, +/- 5% × 20%)

99. The smaller the improvement
impacts that you want to
detect (MDE), the bigger the
required sample size to be
really sure that you can detect
that small change when it
happens - MORE
OPPORTUNITY
When to stop the test to evaluate?
The more significant you want
to be (as less chance of being
wrong as possible), the
bigger the required sample
size - SAFER

100. Always define the sample size before running the test
● Should not peek the result before you hit the deadline
● Should not end the experiment early before the deadline even when you hit stat. significance
● Should not extend the test past the deadline hoping to reach significance

101. How to act based on result?

102. Iteration is crucial

103. Deeper analysis on experience results

104. To take the next step actions

105. What happen if you don’t analyze carefully

106. Iteration based on learnings is key to success

107. 4 Steps Framework of A/B Testing
Step 1
Data
Figure out what
to improve
Step 2
Hypothesis
Make an
educated guess
Step 3
Experiment
Test
your guess
Step 4
Act
Decide what’s
next

108. Before & After

109. Homepage

110. Catalog Page

111. Product Page

113. Bonus Tips

114. Don’t give up too soon!

120. If you are not testing, you are not learning
Always have test running, no matter how small
If you don’t have a process, you won’t
continuously improve

121. Thank you!
Jack Nguyen
jack@useinsider.com