Test & Learn: The Alchemy & Science of Product Metrics - Choosing Metrics That Work
•
0 likes•4,629 views
Quantitative measurement is the key to scaling businesses, processes, and products and making them better. It sounds easy: just pick a number, and improve it. It's not. Choosing a metric is an exploration of a many-dimensional space with no map and no guide. Until now. This talk will teach you the science of choosing metrics that will guide you to building a better product. When you attend this webinar, you will: Learn how applying data is as much about questions as it is about answers Discover the trade-offs between different metrics Understand systematic ways to measure your product funnel Learn how to isolate the most important metrics to maximize your product impact
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Test & Learn: The Alchemy & Science of Product Metrics - Choosing Metrics That Work
Similar to Test & Learn: The Alchemy & Science of Product Metrics - Choosing Metrics That Work (20)
More from Optimizely
More from Optimizely (20)
Recently uploaded
Recently uploaded (20)
Test & Learn: The Alchemy & Science of Product Metrics - Choosing Metrics That Work
- 1. Ketan Gangatirkar Vice President of Engineering – Job Seeker Indeed The Alchemy & Science of Product Metrics: Choosing Metrics That Work
- 3. Search
- 5. Apply
- 6. Interview
- 9. In theory…
- 10. How do we know if it worked?
- 11. Use Big Data!
- 13. “Computers are useless. They can only give you answers.” — Pablo Picasso
- 14. “Data is useless. It can only give you answers.” — This Guy
- 15. What questions are you asking?
- 16. (no matter how much good data you have) Bad Questions Bad Science
- 17. Chemistry
- 18. Alchemy Rules of thumb, recipes, and fantasy
- 19. Useful
- 20. ? ¯_(ツ)_/¯
- 21. People died all the time
- 22. This is not science
- 23. This is not science ● Incomplete ● Inflexible ● Inexplicable
- 24. The alchemy of metrics
- 25. Lots of recipes frameworksLots of recipes frameworks
- 27. Creating a science of metrics
- 28. Quality attribute, property, or characteristic of a phenomenon
- 29. Unit a count or measure of this quality
- 30. Metric = number * unit
- 31. Mass = 2 Kilograms
- 32. Interaction = 6 clicks
- 33. Searches Job alert subscriptions Job applications Mobile app installs Resume uploads Clicks Hires Saving a job Page count Net promoter score Unsubscriptions Bounce rate Click-through rate Adoption Repeat visits Unique visitors Number of accounts Happiness Conversions Positive responses Blocked jobs Shares Dwell time Follows Open rate Uninstalls Ratings Actions Engagement Interviews Page views Sessions Daily active users What do we measure?!?!?!
- 34. Before what comes why
- 35. Everyone has different goals
- 36. There is no one right metric for everyone
- 37. There is no one right metric for everyone
- 38. Lifetime value
- 39. Total product benefit generated over all time
- 40. OR More Jobs Data Scientist Microsoft 4,230 reviews - London The role We are looking to growing our data team and if you love understanding the customers through the data lens this position is for you. We are poised with finding new ways for uncovering data insights to increase our ability to build an More Resumes President president@indeed.com Work Experience President Indeed - Austin, TX Oct. 2015 -Present Indeed is the #1 job site worldwide with over 180 million unique visitors and 2 billion searches per month in more than 60 countries. SVP, Product & Engineering Indeed - Austin, TX October 2010 - October 2015 Indeed is the #1 job site worldwide with over 180 million unique visitors and 2 billion searches per month in more than 60 countries.
- 41. More Jobs More Resumes President president@indeed.com Work Experience President Indeed - Austin, TX Oct. 2015 -Present Indeed is the #1 job site worldwide with over 180 million unique visitors and 2 billion searches per month in more than 60 countries. SVP, Product & Engineering Indeed - Austin, TX October 2010 - October 2015 Indeed is the #1 job site worldwide with over 180 million unique visitors and 2 billion searches per month in more than 60 countries. Lifetime Value Data Scientist Microsoft 4,230 reviews - London The role We are looking to growing our data team and if you love understanding the customers through the data lens this position is for you. We are poised with finding new ways for uncovering data insights to increase our ability to build an
- 42. OR
- 43. Lifetime value!
- 44. Lifetime value works for everything
- 45. How long does it take to measure lifetime value?
- 46. 2018-2094 Manuela Jobseeker "Now we can finally finish that A/B test."
- 47. We need to ship next week
- 49. Sensitivity how quickly and easily you can influence a metric
- 50. S E N S I T I V I T Y Slow and difficult to change Fast and easy to change
- 51. Correlation frequency of agreement with lifetime value
- 52. High correlation Usually leads toward lifetime value Rarely leads elsewhere Lifetime value
- 53. Medium correlation Generally leads toward lifetime value Occasionally leads elsewhere Lifetime value
- 54. Low correlation Sometimes leads toward lifetime value Often leads elsewhere Lifetime value
- 55. CORRELATION Almost always leads toward lifetime value Unpredictably leads toward lifetime value
- 59. Mobile app install Saving a job DAU Phase of the moon Mouse pointer movement Lifetime value S E N S I T I V I T Y CORRELATION
- 62. Vilfredo Pareto
- 63. Pareto Efficiency When you can't improve one thing without making something else worse
- 66. Pareto meets sensitivity and correlation
- 70. Time for some funnel
- 71. Search Tap Apply Interview Offer Hire
- 72. Search Tap Apply Interview Offer “I need a job” ”I got a job!”
- 74. Where to validate changes
- 75. Modify this Maybe this? Or this? What about this? Measure this?
- 77. When to favor outcomes
- 79. Favor outcomes + Late in the funnel + Smaller opportunities + Have time for sophisticated analysis + Owner has more influence over outcome
- 80. Favor outcomes if you can
- 81. When to favor outputs
- 82. + Exploratory product discovering possible outcomes + Opportunities for big improvements + Only simple analysis + Less influence over outcome Favor outputs
- 83. Favor outputs if you must
- 84. I help people look at job descriptions.
- 85. What to measure in your funnel
- 86. Timeless needs
- 87. Your solutions are always evolving
- 88. Your users' needs and preferences are not
- 89. I very frequently get the question: "What’s going to change in the next 10 years?" I almost never get the question: "What’s not going to change in the next 10 years?" That second question is actually the more important of the two — because you can build a business strategy around the things that are stable in time… We know that customers want low prices, and I know that’s going to be true 10 years from now. They want fast delivery; they want vast selection. It’s impossible to imagine a future 10 years from now where a customer comes up and says, "Jeff I love Amazon; I just wish the prices were a little higher," [or] "I love Amazon; I just wish you’d deliver a little more slowly." Jeff Bezos Chairman, CEO, President, and Ruthless Cyborg Killer Amazon, Inc.
- 90. Many degrees of freedom
- 91. Choose metrics that can be improved multiple ways
- 92. Your first effort will fail
- 93. You need backup plans
- 94. Metric B' Metric C' Invalid hypotheses Metric A'Plan A Plan B Plan C
- 95. You need backup plans with the same success metrics
- 96. A better hypothesis The same metrics Plan A Plan B Plan C
- 97. Every interesting phenomenon is multi-dimensional
- 99. Apply
- 106. Apply Time
- 107. Apply Effort
- 108. Apply Cost
- 109. Apply Happiness
- 110. Funnel stage dimensions Apply 1 Quality of input 2 Quantity of input 3 Quality of output 4 Quantity of output 5 Time 6 Effort 7 Cost 8 Happiness
- 111. 1 Quality of input 2 Quantity of input 3 Time to complete 4 Effort to complete 5 Cost 6 Happiness Similar measures for your whole funnel
- 113. Quantity of outcome
- 114. Quality of outcome
- 115. Measuring a funnel 1 Quality of input 2 Quantity of input 3 Time to complete 4 Effort to complete 5 Cost 6 Happiness 7 Number of stages 8 Quality of outcome 9 Quantity of outcome
- 116. So many choices… 1 Quality of input 2 Quantity of input 3 Quality of output 4 Quantity of output 5 Time 6 Effort 7 Cost 8 Happiness 1 Quality of input 2 Quantity of input 3 Time to complete 4 Effort to complete 5 Cost 6 Happiness 7 Number of stages 8 Quality of outcome 9 Quantity of outcome
- 117. Finding your North Star
- 118. Outcome Maximize Outcome Quality Maximize Outcome Quantity
- 120. Why's the bottleneck? Low quality? Too slow? Too expensive? …
- 121. Your bottleneck is your North Star
- 122. NORTH ^
- 124. Bottlenecks move
- 126. Polaris Gamma Cephei Alderamin Deneb Your North Star will move
- 127. Finding your new North Star Need Outcome Bottleneck Diagnosis ActionHypothesis Your North Star (for now)
- 128. How many metrics?
- 129. Different metrics deviate in different ways
- 130. + + Combine multiple metrics to reduce deviation =
- 131. Too many metrics restrict movement
- 132. 50% chance of one metric improving
- 133. 25% chance of two metrics improving
- 134. 12.5% chance of three metrics improving
- 135. 6.25% chance of four metrics improving
- 136. F R E E D O M Degrees of freedom NUMBER OF METRICS
- 137. Marginal information gain Every metric you add should significantly improve the quality of your decisions
- 139. NUMBER OF METRICS Trading off Freedom Information I N F O R M A T I O N
- 140. Target 3-4 independent metrics
- 141. Better prediction using mental models
- 142. The measure of a metric is its utility (not truth or accuracy or meaning)
- 143. Focus on what optimizing the metric does
- 144. Thought experiments
- 146. Which actions improve my metric?
- 147. Which actions degrade my metric?
- 149. + Insincere activity + Indiscriminate targeting of jobs Optimizing for phone screens Encourages + Deeper engagement Discourages + Broad attention across many jobs + Selective targeting of jobs + Many applications to fewer jobs Is this what we want?
- 150. Choosing between similar options
- 151. Volume = total phone screens
- 152. Coverage = number of job seekers with phone screen
- 153. Volume vs. Coverage 3 4 5 45Volume 1Coverage
- 154. Where do they diverge?
- 155. Good for more job seekers Better for fewer job seekers versus
- 156. Metrics = strategy
- 157. Your choice of metrics defines your problem
- 158. You will never find strategy in the data
- 159. You have to bring your own strategy (and then measure it)
- 160. Solving problems with science
- 161. Increasing lifetime value Lifetime valuePersistent needOutcomeOutputBottleneckAction
- 162. 6 things to try in your next product meeting
- 163. 1) Estimate sensitivity and correlation 2) Find your Pareto frontier 3) Map your funnel and apply standard metrics 4) Trade off outputs and outcomes 5) Target 3-4 independent metrics 6) Run thought experiments
- 164. The end (for realz this time)
- 165. More information Contact ketan@scienceofmetrics.com Video of v1 of this talk https://go.indeed.com/alchemy-metrics Key points summary https://go.indeed.com/MetricsThatWork Indeed Engineering Blog http://engineering.indeedblog.com/blog/
- 166. Next Session How to Find Your Product's North Star Metric Presented by Alicia Shiu, Dropbox