1. Metrics, Analytics &
Experiments
Saturday, 16 February 13

2. Outcomes
Saturday, 16 February 13

3. Outcomes
• Theory
Saturday, 16 February 13

4. Outcomes
• Theory
• What are analytics & experimentation?
Saturday, 16 February 13

5. Outcomes
• Theory
• What are analytics & experimentation?
• Why are they important
Saturday, 16 February 13

6. Outcomes
• Theory
• What are analytics & experimentation?
• Why are they important
• How do you do them?
Saturday, 16 February 13

7. Outcomes
• Theory
• What are analytics & experimentation?
• Why are they important
• How do you do them?
• Practical
Saturday, 16 February 13

8. Outcomes
• Theory
• What are analytics & experimentation?
• Why are they important
• How do you do them?
• Practical
• Analytics plan
Saturday, 16 February 13

9. Outcomes
• Theory
• What are analytics & experimentation?
• Why are they important
• How do you do them?
• Practical
• Analytics plan
• Experiment plan
Saturday, 16 February 13

10. Sim on

11. Cast
@simo ncast
Saturday, 16 February 13

12. Saturday, 16 February 13

13. Saturday, 16 February 13

14. Saturday, 16 February 13

15. Saturday, 16 February 13

16. Saturday, 16 February 13

17. Saturday, 16 February 13

18. Analytics Metrics
Saturday, 16 February 13

19. Saturday, 16 February 13

20. “
Saturday, 16 February 13
”

21. “ Measures the state of
the product
Saturday, 16 February 13
”

22. “What you don’t
measure you can’t
improve!
Saturday, 16 February 13
”

23. Concepts
Saturday, 16 February 13

24. Concepts
• Data points
Saturday, 16 February 13

25. Concepts
• Data points
• Segmentation
Saturday, 16 February 13

26. Concepts
• Data points
• Segmentation
• Funnels
Saturday, 16 February 13

27. Concepts
• Data points
• Segmentation
• Funnels
• Cohorts
Saturday, 16 February 13

28. Data Points
Saturday, 16 February 13

29. Data Points
• These are the user actions that are
measured
Saturday, 16 February 13

30. Data Points
• These are the user actions that are
measured
• Includes a datetime stamp
Saturday, 16 February 13

31. Data Points: Example
Saturday, 16 February 13

32. Segmentation
Saturday, 16 February 13

33. Segmentation
• Grouping users together
based on some characteristic
of the users
Saturday, 16 February 13

34. Segmentation
• Grouping users together
based on some characteristic
of the users
• Shows patterns otherwise
hidden by noise averages
Saturday, 16 February 13

35. Segmentation
• Grouping users together
based on some characteristic
of the users
• Shows patterns otherwise
hidden by noise averages
• Focus on those who are
most important to you
Saturday, 16 February 13

36. Segmentation: Example
Saturday, 16 February 13

37. Funnels
Saturday, 16 February 13

38. Funnels
• Flow: series of steps users go
through
Saturday, 16 February 13

39. Funnels
• Flow: series of steps users go
through
• Funnels measure flows
Saturday, 16 February 13

40. Funnels
• Flow: series of steps users go
through
• Funnels measure flows
• Shows where the leakage is
in a flow
Saturday, 16 February 13

41. Funnel: Example
Saturday, 16 February 13

42. Cohorts
Saturday, 16 February 13

43. Cohorts
• Grouping by point in time
Saturday, 16 February 13

44. Cohorts
• Grouping by point in time
• How does behaviour change over
time?
Saturday, 16 February 13

45. Cohorts
• Grouping by point in time
• How does behaviour change over
time?
• Relative or absolute
Saturday, 16 February 13

46. Cohorts
• Grouping by point in time
• How does behaviour change over
time?
• Relative or absolute
• Comparison between cohorts
Saturday, 16 February 13

47. Cohorts: Example
Saturday, 16 February 13

48. Saturday, 16 February 13

49. Start with a plan
Saturday, 16 February 13

50. Start with a plan
Saturday, 16 February 13

51. Saturday, 16 February 13

52. Product
Vision
Saturday, 16 February 13

53. Product
Vision
Saturday, 16 February 13

54. Product
KPIs
Vision
Saturday, 16 February 13

55. Product
KPIs
Vision
Saturday, 16 February 13

56. Product
KPIs Metrics
Vision
Saturday, 16 February 13

57. Product
KPIs Metrics
Vision
Saturday, 16 February 13

58. Product
KPIs Metrics Funnel
Vision
Saturday, 16 February 13

59. Product
KPIs Metrics Funnel
Vision
Saturday, 16 February 13

60. Product
KPIs Metrics Funnel
Vision
Data
points
Saturday, 16 February 13

61. Vision
“ What is the product
suppose to do?
Saturday, 16 February 13
”

62. KPIs
Saturday, 16 February 13

63. KPIs
• The KPIs (key performance indicators) are
derived from the product vision
Saturday, 16 February 13

64. KPIs
• The KPIs (key performance indicators) are
derived from the product vision
• Need to take into account at what stage
your product is
Saturday, 16 February 13

65. KPI: Example
Saturday, 16 February 13

66. KPI: Example
• Dollar value of checkouts
Saturday, 16 February 13

67. KPI: Example
• Dollar value of checkouts
• Number of documents stored
Saturday, 16 February 13

68. Metrics
• Represents the KPIs
• What can be manipulated to influence the
KPIs
• Calculated from multiple data points
Saturday, 16 February 13

69. Metrics
Saturday, 16 February 13

70. Metrics
• Quantitative
Saturday, 16 February 13

71. Metrics
• Quantitative
• Comparative
Saturday, 16 February 13

72. Metrics
• Quantitative
• Comparative
• Actionable
Saturday, 16 February 13

73. Metrics: Example
• Average checkout value
• New documents per day
Saturday, 16 February 13

74. Flows Funnels
• Focus on the flows that directly effect the
metrics
• What data represents each step in the
flow?
Saturday, 16 February 13

75. Funnel: Example
Saturday, 16 February 13

76. Data Points
• What makes up metrics
• What makes up funnels
• Achievable
• Segments
Saturday, 16 February 13

77. Data Points: Example
• Number of new visitors
• Number who register
• Number who reach each step of the
• Value of each checkout chart
• Number of checkouts
• Number of documents uploaded
Saturday, 16 February 13

78. Implementation
Saturday, 16 February 13

79. Implementation
• Record the data points being collected
Saturday, 16 February 13

80. Implementation
• Record the data points being collected
• In-house versus 3rd party
Saturday, 16 February 13

81. Implementation: Tools
Saturday, 16 February 13

82. Implementation: Tools
Saturday, 16 February 13

83. Implementation: Tools
Saturday, 16 February 13

84. Implementation: Tools
Saturday, 16 February 13

85. Implementation: Tools
Saturday, 16 February 13

86. Experimentation
Saturday, 16 February 13

87. Why What?
Saturday, 16 February 13

88. Why What?
• Apply scientific method to get a better
product
Saturday, 16 February 13

89. Why What?
• Apply scientific method to get a better
product
• Test different assumptions made about the
product
Saturday, 16 February 13

90. Why What?
• Apply scientific method to get a better
product
• Test different assumptions made about the
product
• Test different variations to see what effects
the KPIs
Saturday, 16 February 13

91. How?
Saturday, 16 February 13

92. How?
Saturday, 16 February 13

93. Saturday, 16 February 13

94. Saturday, 16 February 13

95. The planning process
Saturday, 16 February 13

96. The planning process
• Question
Saturday, 16 February 13

97. The planning process
• Question
• Hypothesis
Saturday, 16 February 13

98. The planning process
• Question
• Hypothesis
• Experiment to test hypothesis
Saturday, 16 February 13

99. Saturday, 16 February 13

100. Question
• Related to KPIs and assumptions about the
product
• How to increase y?
• Why is x so?
• Is z assumption true?
Saturday, 16 February 13

101. Question: Example
Why is the conversion rate on the landing
page 10% and not 30%?
Saturday, 16 February 13

102. Hypothesis
Saturday, 16 February 13

103. Hypothesis
• Proposed reason or explanation for a
phenomena
Saturday, 16 February 13

104. Hypothesis
• Proposed reason or explanation for a
phenomena
• An answer to the question or explanation
of the question
Saturday, 16 February 13

105. Hypothesis
• Proposed reason or explanation for a
phenomena
• An answer to the question or explanation
of the question
• Testable with independent variable that can
be controlled and a dependent variable that
can be measured
Saturday, 16 February 13

106. Hypothesis: Example
• The call to action button should be red
• The message is not clear about the value of
registering
• There are too many different call-to-actions
on the page
Saturday, 16 February 13

107. Hypothesis: Structure
Saturday, 16 February 13

108. Hypothesis: Structure
• IF I water the plants THEN the plants will
grow
Saturday, 16 February 13

109. Hypothesis: Structure
• IF I water the plants THEN the plants will
grow
• IF I don’t water the plants THEN the plants
will not grow
Saturday, 16 February 13

110. Hypothesis: Example
Saturday, 16 February 13

111. Hypothesis: Example
• IF the call to action button is red THEN
the number of people registering will
increase
Saturday, 16 February 13

112. Hypothesis: Example
• IF the call to action button is red THEN
the number of people registering will
increase
• IF we change the copy explaining the value
of registering THEN the number of
people registering will go up
Saturday, 16 February 13

113. Hypothesis: Example
• IF the call to action button is red THEN
the number of people registering will
increase
• IF we change the copy explaining the value
of registering THEN the number of
people registering will go up
• IF we remove all but one call-to-action on
the page THEN the number of people
registering will increase
Saturday, 16 February 13

114. Experiment
Saturday, 16 February 13

115. Experiment
• Type
Saturday, 16 February 13

116. Experiment
• Type
• Control
Saturday, 16 February 13

117. Experiment
• Type
• Control
• Variant(s)
Saturday, 16 February 13

118. Experiment
• Type
• Control
• Variant(s)
• Goal
Saturday, 16 February 13

119. Variants
• Flow from the hypothesis
• They are the independent variable in the
hypothesis
Saturday, 16 February 13

120. Experiment: Example
IF the call to action button is red THEN the
number of people registering will increase
Saturday, 16 February 13

121. Goals
• Flow from hypothesis
• The data point that is expected to change
Saturday, 16 February 13

122. Experiment: Example
IF the call to action button is red THEN
the number of people registering will increase
Saturday, 16 February 13

123. Types
Saturday, 16 February 13

124. Types
• A/B Test
Saturday, 16 February 13

125. Types
• A/B Test
• Split URL
Saturday, 16 February 13

126. Types
• A/B Test
• Split URL
• Multivariate
Saturday, 16 February 13

127. A/B
A (control) B (variant)
vs
Saturday, 16 February 13

128. Split URL
Saturday, 16 February 13

129. Mulitvariate
Saturday, 16 February 13

130. Saturday, 16 February 13

131. Running Experiment
Saturday, 16 February 13

132. Running Experiment
• Record the experiments results
Saturday, 16 February 13

133. Running Experiment
• Record the experiments results
• Segment the traffic
Saturday, 16 February 13

134. Running Experiment
• Record the experiments results
• Segment the traffic
• Traffic (random split to remove bias)
Saturday, 16 February 13

135. Running Experiment
• Record the experiments results
• Segment the traffic
• Traffic (random split to remove bias)
• How long to run?
Saturday, 16 February 13

136. Running Experiment
• Record the experiments results
• Segment the traffic
• Traffic (random split to remove bias)
• How long to run?
• Who to test?
Saturday, 16 February 13

137. Running Experiment
• Record the experiments results
• Segment the traffic
• Traffic (random split to remove bias)
• How long to run?
• Who to test?
• Test well - don’t take short cuts
Saturday, 16 February 13

138. Running Experiment
• Record the experiments results
• Segment the traffic
• Traffic (random split to remove bias)
• How long to run?
• Who to test?
• Test well - don’t take short cuts
• Negative result
Saturday, 16 February 13

139. Closing the Loop
Saturday, 16 February 13

140. Closing the Loop
• What do these results mean for the
product development prioritisation?
Saturday, 16 February 13

141. Closing the Loop
• What do these results mean for the
product development prioritisation?
• Why did I get these results? or,
Saturday, 16 February 13

142. Closing the Loop
• What do these results mean for the
product development prioritisation?
• Why did I get these results? or,
• Why didn’t I get the results expected?
Saturday, 16 February 13

143. Closing the Loop
• What do these results mean for the
product development prioritisation?
• Why did I get these results? or,
• Why didn’t I get the results expected?
• Revamp priorisation produce the next
set of experiments
Saturday, 16 February 13

144. Tools
Saturday, 16 February 13

145. Tools
Saturday, 16 February 13

146. Tools
Saturday, 16 February 13

147. Tools
Saturday, 16 February 13

148. Caveat
Saturday, 16 February 13

149. Caveat
Saturday, 16 February 13

150. Practice
Saturday, 16 February 13