Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Donal McMahon
Weapons of Math Instruction:
Evolving from Data-Driven to Science-Driven
Director of Data Science, Indeed
Convince you to use the scientific method.
Then, I’ll teach you how.
using data.
Indeed is data-driven
We’ve hosted 6 eng tech talks on this topic!
Many other industries are also now becoming data-driven
1 Democratize decision-making
2 Better decisions
3 Increase decision velocity
4 Improve collaboration via ego removal
Why ...
Is data-driven an accurate descriptor?
No, we’re science-driven
Why do you need to be science-driven?
A cautionary tale
Dreaming big and about to change the world
Donal PM
disclaimer: sadly not real childhood photos
Idea
Modernize our mobile site to improve job seeker experience
Control Treatment
Change 1
Increased spacing
between jobs
Control Treatment
Change 2
Replaced orange text with
buttons for:
● New
● Apply with your Indeed
Resume
Control Treatment
Change 3
Removed sponsored jobs
Control Treatment
Change 4
Other minor UI tweaks
● Salary range
● Home button
● Fonts
Control Treatment
Completely aligned
Donal PM
Convinced software developers to implement
We ran an A/B test and generated lots of data
Analysed it separately
We drew contradictory conclusions
What is job seeker experience?
What’s a job?
One that’s anywhere on the page, or one that’s viewed?
What’s an acceptable metric trade off?
Resolution strategy: be more data-driven
So, we threw
more data at each other
different hypotheses + different data + different metrics
∴ different conclusions
We learn geology the morning
after the earthquake.
Ralph Waldo Emerson.
A better solution does exist
The Scientific Method
Observation Question Hypothesis Experiment Analysis Conclusion
Remainder of this talk
1 What did we do?
2 Why was it wrong?
3 How can you do it better?
Stage 1: Observation
What did we do?
Observation Question Hypothesis Experiment Analysis Conclusion
Nothing
Observation Question Hypothesis Experiment Analysis Conclusion
Why was it wrong?
1
Didn’t establish baseline for job seeke...
Observation Question Hypothesis Experiment Analysis Conclusion
How can you do it better?
Nano
Study real job seeker sessio...
How can you do it better?
Nano: study real job seeker sessions
Query 1
Click on
Job A
Click on
Job C
Query 2
Click on
Job ...
Not only is the universe stranger
than we imagine, it is stranger
than we can imagine.
Sir Arthur Eddington
How can you do it better? A shameless plug
Micro: partner with experts (UX) to gather qualitative data
Observation Questio...
How can you do it better?
Micro: partner with experts (UX) to gather qualitative data
Observation Question Hypothesis Expe...
How can you do it better?
Macro: large scale data analysis and observation via experimentation
Common Question
What’s a wo...
How can you do it better?
Reality
You’re making trade-offs implicitly already
Observation Question Hypothesis Experiment A...
How can you do it better?
Learn your implicit local trade-off function
Run multiple simple perturbation experiments, all t...
Observation via experimentation
Applies
JobAlert
Signups
Observation via experimentation
Applies
JobAlert
Signups
Current state
Observation via experimentation
Applies
JobAlert
Signups
Current state
Learn your current implicit trade-offs via experimentation
Applies
JobAlert
Signups
Expt 1: bold Apply with
your Indeed Re...
Learn your current implicit trade-offs via experimentation
Applies
JobAlert
Signups
Expt 2: add pixel
whitespace to
JobAle...
Learn your current implicit trade-offs via experimentation
Applies
JobAlert
Signups
Compare your current state to all pareto efficient alternatives
Applies
JobAlert
Signups
For each pareto efficient alternative you have a tradeoff
Applies
JobAlert
Sign-ups
ΔApplies
ΔJobAlerts
How can you do it better?
Implicit tradeoff
Each JobAlert sign-up is worth 1.7 Applies
Observation Question Hypothesis Exp...
Stage 2: Question
What did we do?
Observation Question Hypothesis Experiment Analysis Conclusion
Nothing
Why was it wrong?
1
We never prioritized the most important question(s)
2
By bundling questions, we couldn’t answer any, l...
Observation Question Hypothesis Experiment Analysis Conclusion
Research Question
Potential
Impact
Complexity
Time To
Learn
What are good measures for job seeker experience? ? ? ?
How ca...
Stage 3: Hypothesis
What did we do?
Modernize the mobile interface to improve job seeker
experience
Observation Question Hypothesis Experiment...
Observation Question Hypothesis Experiment Analysis Conclusion
Why was it wrong?
1
Hypothesis was ill-defined and vague
2
No established metrics
3
No clear success/failure criteria
Obse...
How can you do it better?
1
Determine one or more hypothesis
“Does extra whitespace between job cards help job seekers to ...
Important Question #1
How many metrics?
Observation Question Hypothesis Experiment Analysis Conclusion
Observation Question Hypothesis Experiment Analysis Conclusion
Spoiler
3
Your product is a high dimensional hypercube
Observation Question Hypothesis Experiment Analysis Conclusion
2D hypercube 3D hypercube 4D hypercube
5D hypercube 6D hypercube 7D hypercube
2D hypercube 3D hypercube 4D hypercube
5D hypercube 6D hypercube 7D hypercube
2D hypercube 3D hypercube 4D hypercube
5D hypercube 6D hypercube 7D hypercube
2D hypercube 3D hypercube 4D hypercube
5D hypercube 6D hypercube 7D hypercube
How many metrics?
We need a low-dimensional representation
that preserves almost all of the signal
Observation Question Hy...
How many metrics?
Singular value decomposition (SVD)
Observation Question Hypothesis Experiment Analysis Conclusion
Observation Question Hypothesis Experiment Analysis Conclusion
How many metrics using SVD
Observation Question Hypothesis Experiment Analysis Conclusion
How many metrics using SVD
Observation Question Hypothesis Experiment Analysis Conclusion
Observation Question Hypothesis Experiment Analysis Conclusion
Important Question #2
How do you choose great metrics?
This is a full academic
discipline
Observation Question Hypothesis Experiment Analysis Conclusion
Some dedicated their 20’...
You need to decide on a target (θ)
Observation Question Hypothesis Experiment Analysis Conclusion
Choosing metrics
Observation Question Hypothesis Experiment Analysis Conclusion
Termed the estimand in statistics (θ)
Choose how you’ll aim for the target
Observation Question Hypothesis Experiment Analysis Conclusion
Choosing metrics
Estimator and Estimate (θ)
Observation Question Hypothesis Experiment Analysis Conclusion
Observation Question Hypothesis Experiment Analysis Conclusion
Mathematical criteria for metric evaluation
1
Bias
2
Varian...
Observation Question Hypothesis Experiment Analysis Conclusion
Mathematical criteria
1
Bias
2
Variance
3
System complexity
Observation Question Hypothesis Experiment Analysis Conclusion
Bias
Observation Question Hypothesis Experiment Analysis Conclusion
It can be easy to miss bias
Observation Question Hypothesis Experiment Analysis Conclusion
Observation Question Hypothesis Experiment Analysis Conclusion
Hidden bias in our example
Estimate “time to hire” for job ...
Job seeker First action Still active Hired
1 01/01/2016 Yes No
2 01/22/2016 No 01/25/2016
3 02/04/2016 No 02/23/2016
4 02/...
Observation Question Hypothesis Experiment Analysis Conclusion
Initial Metric Proposal
Average time to hire for job seeker...
But there’s a flaw
Job seeker First action Still active Hired
1 01/01/2016 Yes No
2 01/22/2016 No 01/25/2016
3 02/04/2016 No 02/23/2016
4 02/...
Job seeker First action Still active Hired
1 01/01/2016 Yes No
2 01/22/2016 No 01/25/2016
3 02/04/2016 No 02/23/2016
4 02/...
Job seeker First action Still active Hired
1 01/01/2016 Yes No
2 01/22/2016 No 01/25/2016
3 02/04/2016 No 02/23/2016
4 02/...
Observation Question Hypothesis Experiment Analysis Conclusion
Solution
Estimate typical time to hire using Kaplan-Meier E...
Observation Question Hypothesis Experiment Analysis Conclusion
Time (t)
Observation Question Hypothesis Experiment Analysis Conclusion
Estimated time to hire
Observation Question Hypothesis Experiment Analysis Conclusion
Mathematical criteria for metric evaluation
1
Bias
2
Varian...
Observation Question Hypothesis Experiment Analysis Conclusion
Variance - a measure of data spread
Low variance High varia...
Observation Question Hypothesis Experiment Analysis Conclusion
Variance is fundamental
for valid statistical inference
Observation Question Hypothesis Experiment Analysis Conclusion
Science assumes “innocent until proven guilty”
We often term this our null hypothesis (H0)
Observation Question Hypothesis...
Proof required beyond reasonable doubt
In order to reject the null hypothesis
Observation Question Hypothesis Experiment A...
Variance is your estimate of uncertainty, i.e. doubt
Observation Question Hypothesis Experiment Analysis Conclusion
Observation Question Hypothesis Experiment Analysis Conclusion
Note
We often choose the
Minimum Variance Unbiased Estimato...
Not Always MVUE
Occasionally you
might trade bias for
variance
e.g. machine learning
Low variance High variance
HighbiasLo...
Observation Question Hypothesis Experiment Analysis Conclusion
Mathematical criteria
1
Bias
2
Variance
3
System complexity
System complexity
Product development isn’t linear
Observation Question Hypothesis Experiment Analysis Conclusion
Observation Question Hypothesis Experiment Analysis Conclusion
Sometimes there are multiple potential targets
Observation Question Hypothesis Experiment Analysis Conclusion
Or the target is partially blocked
Observation Question Hypothesis Experiment Analysis Conclusion
Or it keeps moving
It can become stressful
There is no catch-all mathematical formula
to measure and account for system complexity
Observation Question Hypothesis Ex...
But that doesn’t mean you shouldn’t try to estimate
it and factor it into decisions
Observation Question Hypothesis Experi...
Search
Tap
Apply
Interview
Offer
“I need a job”
Hire
Observation Question Hypothesis Experiment Analysis Conclusion
Covere...
Observation Question Hypothesis Experiment Analysis Conclusion
A (strange) American staple
Which also involves prediction brackets
You predict a winner for each game and awarded points if correct
16
9
5
4
✅
✅
✅
1
9
5
4
̶ my prediction ̶ actual result
If you predict an upset early, success/failure compounds
16
9
5
4
✅
✅
✅
1
9
5
4
̶ my prediction ̶ actual result
9 1
4 ✅ 4
...
● Downstream compounded loss
● Number of bracket participants
● Points awarded at each stage
Observation Question Hypothes...
How to win your NCAA pool
Simulate the downstream effect of all potential decisions
Check whether it increases/decreases y...
Reminder - How can you do it better?
1
Determine one or more hypothesis
“Does extra whitespace between job cards help job ...
Stage 4: Experiment
What did we do?
Ran a single treatment experiment where we
simultaneously changed four components
Observation Question Hyp...
Observation Question Hypothesis Experiment Analysis Conclusion
Why was it wrong?
Observation Question Hypothesis Experiment Analysis Conclusion
Couldn’t disentangle the effects of the 4...
How can you do it better?
Observation Question Hypothesis Experiment Analysis Conclusion
Run a full factorial experiment
Observation Question Hypothesis Experiment Analysis Conclusion
Full Factorial Experiment
Suggestion
A: Whitespace
B: Orang...
Full Factorial Experiment
Observation Question Hypothesis Experiment Analysis Conclusion
Increased statistical power, and ...
Full Factorial Experiment
Observation Question Hypothesis Experiment Analysis Conclusion
i.e. you’ll learn more and learn ...
Stage 5: Analysis
What did we do?
1
Cobbled data together from different sources
2
Defined different metrics
3
Invested a lot of time analys...
To consult the statistician after an
experiment is finished is often merely
to ask her to conduct a post mortem
examinatio...
Why was it wrong?
Observation Question Hypothesis Experiment Analysis Conclusion
Opinion-driven, time sink, unsatisfying f...
How can you do it better?
Observation Question Hypothesis Experiment Analysis Conclusion
With correct setup, this should b...
Existing metric New metric
Existing product
New product
Observation Question Hypothesis Experiment Analysis Conclusion
Existing metric New metric
Existing product Uninteresting
New product
Observation Question Hypothesis Experiment Analysis ...
Existing metric New metric
Existing product Uninteresting Metric Innovation
New product
Observation Question Hypothesis Ex...
Existing metric New metric
Existing product Uninteresting Metric Innovation
New product Product Innovation
Observation Que...
Existing metric New metric
Existing product Uninteresting Metric Innovation
New product Product Innovation Uninformative
O...
Never use new data or metrics
to validate new products!
Observation Question Hypothesis Experiment Analysis Conclusion
Stage 6: Conclusion
What did we do?
Observation Question Hypothesis Experiment Analysis Conclusion
Drew two different conclusions
Why was it wrong?
1
Didn’t learn anything
2
Lost team trust
Observation Question Hypothesis Experiment Analysis Conclusion
How can you do it better?
Observation Question Hypothesis Experiment Analysis Conclusion
Should follow directly from analy...
The Goldilocks syndrome
Observation Question Hypothesis Experiment Analysis Conclusion
A/B test
(-1%, 1%] (1%, 5%] (5%, ∞]...
Retain healthy skepticism
Always look for bugs
Check for repeatability via holdbacks
The Complete Scientific Method
Observation Question Hypothesis Experiment Analysis Conclusion
nano,
micro,
macro
prioritiz...
Observation Question Hypothesis Experiment Analysis Conclusion
nano,
micro,
macro
prioritize,
implicit
trade-offs
bias &
v...
Data-driven can be disorientating in a world of abundant data
Be science-driven, i.e. use the scientific method to add nec...
~ finn ~
Weapons of Math Instruction: Evolving from Data0-Driven to Science-Driven
Weapons of Math Instruction: Evolving from Data0-Driven to Science-Driven
Weapons of Math Instruction: Evolving from Data0-Driven to Science-Driven
Weapons of Math Instruction: Evolving from Data0-Driven to Science-Driven
Weapons of Math Instruction: Evolving from Data0-Driven to Science-Driven
Prochain SlideShare
Chargement dans…5
×

Weapons of Math Instruction: Evolving from Data0-Driven to Science-Driven

Donal McMahon, Director of Data Science at Indeed, presented how to transition from data-driven to science-driven product development. You’ll make better business decisions. It’s provable!

  • Soyez le premier à commenter

Weapons of Math Instruction: Evolving from Data0-Driven to Science-Driven

  1. 1. Donal McMahon Weapons of Math Instruction: Evolving from Data-Driven to Science-Driven Director of Data Science, Indeed
  2. 2. Convince you to use the scientific method. Then, I’ll teach you how.
  3. 3. using data.
  4. 4. Indeed is data-driven
  5. 5. We’ve hosted 6 eng tech talks on this topic!
  6. 6. Many other industries are also now becoming data-driven
  7. 7. 1 Democratize decision-making 2 Better decisions 3 Increase decision velocity 4 Improve collaboration via ego removal Why data-driven?
  8. 8. Is data-driven an accurate descriptor?
  9. 9. No, we’re science-driven
  10. 10. Why do you need to be science-driven? A cautionary tale
  11. 11. Dreaming big and about to change the world Donal PM disclaimer: sadly not real childhood photos
  12. 12. Idea Modernize our mobile site to improve job seeker experience
  13. 13. Control Treatment
  14. 14. Change 1 Increased spacing between jobs Control Treatment
  15. 15. Change 2 Replaced orange text with buttons for: ● New ● Apply with your Indeed Resume Control Treatment
  16. 16. Change 3 Removed sponsored jobs Control Treatment
  17. 17. Change 4 Other minor UI tweaks ● Salary range ● Home button ● Fonts Control Treatment
  18. 18. Completely aligned Donal PM
  19. 19. Convinced software developers to implement
  20. 20. We ran an A/B test and generated lots of data
  21. 21. Analysed it separately
  22. 22. We drew contradictory conclusions
  23. 23. What is job seeker experience?
  24. 24. What’s a job? One that’s anywhere on the page, or one that’s viewed?
  25. 25. What’s an acceptable metric trade off?
  26. 26. Resolution strategy: be more data-driven
  27. 27. So, we threw more data at each other
  28. 28. different hypotheses + different data + different metrics ∴ different conclusions
  29. 29. We learn geology the morning after the earthquake. Ralph Waldo Emerson.
  30. 30. A better solution does exist
  31. 31. The Scientific Method Observation Question Hypothesis Experiment Analysis Conclusion
  32. 32. Remainder of this talk 1 What did we do? 2 Why was it wrong? 3 How can you do it better?
  33. 33. Stage 1: Observation
  34. 34. What did we do? Observation Question Hypothesis Experiment Analysis Conclusion Nothing
  35. 35. Observation Question Hypothesis Experiment Analysis Conclusion Why was it wrong? 1 Didn’t establish baseline for job seeker experience, or measures 2 When we failed, we had no knowledge backlog for future work
  36. 36. Observation Question Hypothesis Experiment Analysis Conclusion How can you do it better? Nano Study real job seeker sessions Micro Partner with experts (UX) to gather qualitative data Macro Large scale data analysis and observation via experimentation
  37. 37. How can you do it better? Nano: study real job seeker sessions Query 1 Click on Job A Click on Job C Query 2 Click on Job D Apply Observation Question Hypothesis Experiment Analysis Conclusion
  38. 38. Not only is the universe stranger than we imagine, it is stranger than we can imagine. Sir Arthur Eddington
  39. 39. How can you do it better? A shameless plug Micro: partner with experts (UX) to gather qualitative data Observation Question Hypothesis Experiment Analysis Conclusion medium.com/indeed-data-science
  40. 40. How can you do it better? Micro: partner with experts (UX) to gather qualitative data Observation Question Hypothesis Experiment Analysis Conclusion 1 Real-life observation 2 Interviews 3 Content analysis (surveys)
  41. 41. How can you do it better? Macro: large scale data analysis and observation via experimentation Common Question What’s a worthwhile/launchable metric trade-off? Observation Question Hypothesis Experiment Analysis Conclusion
  42. 42. How can you do it better? Reality You’re making trade-offs implicitly already Observation Question Hypothesis Experiment Analysis Conclusion Macro: large scale data analysis and observation via experimentation
  43. 43. How can you do it better? Learn your implicit local trade-off function Run multiple simple perturbation experiments, all the time Observation Question Hypothesis Experiment Analysis Conclusion Macro: large scale data analysis and observation via experimentation
  44. 44. Observation via experimentation Applies JobAlert Signups
  45. 45. Observation via experimentation Applies JobAlert Signups Current state
  46. 46. Observation via experimentation Applies JobAlert Signups Current state
  47. 47. Learn your current implicit trade-offs via experimentation Applies JobAlert Signups Expt 1: bold Apply with your Indeed Resume
  48. 48. Learn your current implicit trade-offs via experimentation Applies JobAlert Signups Expt 2: add pixel whitespace to JobAlert UI Expt 1: bold Apply with your Indeed Resume
  49. 49. Learn your current implicit trade-offs via experimentation Applies JobAlert Signups
  50. 50. Compare your current state to all pareto efficient alternatives Applies JobAlert Signups
  51. 51. For each pareto efficient alternative you have a tradeoff Applies JobAlert Sign-ups ΔApplies ΔJobAlerts
  52. 52. How can you do it better? Implicit tradeoff Each JobAlert sign-up is worth 1.7 Applies Observation Question Hypothesis Experiment Analysis Conclusion Macro: large scale data analysis and observation via experimentation
  53. 53. Stage 2: Question
  54. 54. What did we do? Observation Question Hypothesis Experiment Analysis Conclusion Nothing
  55. 55. Why was it wrong? 1 We never prioritized the most important question(s) 2 By bundling questions, we couldn’t answer any, learn and improve Observation Question Hypothesis Experiment Analysis Conclusion
  56. 56. Observation Question Hypothesis Experiment Analysis Conclusion
  57. 57. Research Question Potential Impact Complexity Time To Learn What are good measures for job seeker experience? ? ? ? How can we help job seeker navigate to their desired job more quickly? ? ? ? How can we clearly denote sponsored content? ? ? ? … ... ... ... How can you do it better? Observation Question Hypothesis Experiment Analysis Conclusion
  58. 58. Stage 3: Hypothesis
  59. 59. What did we do? Modernize the mobile interface to improve job seeker experience Observation Question Hypothesis Experiment Analysis Conclusion
  60. 60. Observation Question Hypothesis Experiment Analysis Conclusion
  61. 61. Why was it wrong? 1 Hypothesis was ill-defined and vague 2 No established metrics 3 No clear success/failure criteria Observation Question Hypothesis Experiment Analysis Conclusion
  62. 62. How can you do it better? 1 Determine one or more hypothesis “Does extra whitespace between job cards help job seekers to navigate quicker.” 2 Agree on the data, metrics and acceptable trade-offs up front Suggested metrics: (i) time to click, (ii) click rate, (iii) time to hire Observation Question Hypothesis Experiment Analysis Conclusion
  63. 63. Important Question #1 How many metrics? Observation Question Hypothesis Experiment Analysis Conclusion
  64. 64. Observation Question Hypothesis Experiment Analysis Conclusion Spoiler 3
  65. 65. Your product is a high dimensional hypercube Observation Question Hypothesis Experiment Analysis Conclusion
  66. 66. 2D hypercube 3D hypercube 4D hypercube 5D hypercube 6D hypercube 7D hypercube
  67. 67. 2D hypercube 3D hypercube 4D hypercube 5D hypercube 6D hypercube 7D hypercube
  68. 68. 2D hypercube 3D hypercube 4D hypercube 5D hypercube 6D hypercube 7D hypercube
  69. 69. 2D hypercube 3D hypercube 4D hypercube 5D hypercube 6D hypercube 7D hypercube
  70. 70. How many metrics? We need a low-dimensional representation that preserves almost all of the signal Observation Question Hypothesis Experiment Analysis Conclusion
  71. 71. How many metrics? Singular value decomposition (SVD) Observation Question Hypothesis Experiment Analysis Conclusion
  72. 72. Observation Question Hypothesis Experiment Analysis Conclusion
  73. 73. How many metrics using SVD Observation Question Hypothesis Experiment Analysis Conclusion
  74. 74. How many metrics using SVD Observation Question Hypothesis Experiment Analysis Conclusion
  75. 75. Observation Question Hypothesis Experiment Analysis Conclusion Important Question #2 How do you choose great metrics?
  76. 76. This is a full academic discipline Observation Question Hypothesis Experiment Analysis Conclusion Some dedicated their 20’s to this!
  77. 77. You need to decide on a target (θ) Observation Question Hypothesis Experiment Analysis Conclusion Choosing metrics
  78. 78. Observation Question Hypothesis Experiment Analysis Conclusion Termed the estimand in statistics (θ)
  79. 79. Choose how you’ll aim for the target Observation Question Hypothesis Experiment Analysis Conclusion Choosing metrics
  80. 80. Estimator and Estimate (θ) Observation Question Hypothesis Experiment Analysis Conclusion
  81. 81. Observation Question Hypothesis Experiment Analysis Conclusion Mathematical criteria for metric evaluation 1 Bias 2 Variance 3 System complexity
  82. 82. Observation Question Hypothesis Experiment Analysis Conclusion Mathematical criteria 1 Bias 2 Variance 3 System complexity
  83. 83. Observation Question Hypothesis Experiment Analysis Conclusion Bias
  84. 84. Observation Question Hypothesis Experiment Analysis Conclusion
  85. 85. It can be easy to miss bias Observation Question Hypothesis Experiment Analysis Conclusion
  86. 86. Observation Question Hypothesis Experiment Analysis Conclusion Hidden bias in our example Estimate “time to hire” for job seekers
  87. 87. Job seeker First action Still active Hired 1 01/01/2016 Yes No 2 01/22/2016 No 01/25/2016 3 02/04/2016 No 02/23/2016 4 02/17/2016 No No ... ... ... ... ... ... ... ... n 04/23/2016 Yes No Observation Question Hypothesis Experiment Analysis Conclusion
  88. 88. Observation Question Hypothesis Experiment Analysis Conclusion Initial Metric Proposal Average time to hire for job seekers who were hired
  89. 89. But there’s a flaw
  90. 90. Job seeker First action Still active Hired 1 01/01/2016 Yes No 2 01/22/2016 No 01/25/2016 3 02/04/2016 No 02/23/2016 4 02/17/2016 No No ... ... ... ... ... ... ... ... n 04/23/2016 Yes No Observation Question Hypothesis Experiment Analysis Conclusion
  91. 91. Job seeker First action Still active Hired 1 01/01/2016 Yes No 2 01/22/2016 No 01/25/2016 3 02/04/2016 No 02/23/2016 4 02/17/2016 No No ... ... ... ... ... ... ... ... n 04/23/2016 Yes No Observation Question Hypothesis Experiment Analysis Conclusion
  92. 92. Job seeker First action Still active Hired 1 01/01/2016 Yes No 2 01/22/2016 No 01/25/2016 3 02/04/2016 No 02/23/2016 4 02/17/2016 No No ... ... ... ... ... ... ... ... n 04/23/2016 Yes No Observation Question Hypothesis Experiment Analysis Conclusion
  93. 93. Observation Question Hypothesis Experiment Analysis Conclusion Solution Estimate typical time to hire using Kaplan-Meier Estimate
  94. 94. Observation Question Hypothesis Experiment Analysis Conclusion
  95. 95. Time (t) Observation Question Hypothesis Experiment Analysis Conclusion Estimated time to hire
  96. 96. Observation Question Hypothesis Experiment Analysis Conclusion Mathematical criteria for metric evaluation 1 Bias 2 Variance 3 System complexity
  97. 97. Observation Question Hypothesis Experiment Analysis Conclusion Variance - a measure of data spread Low variance High variance
  98. 98. Observation Question Hypothesis Experiment Analysis Conclusion
  99. 99. Variance is fundamental for valid statistical inference Observation Question Hypothesis Experiment Analysis Conclusion
  100. 100. Science assumes “innocent until proven guilty” We often term this our null hypothesis (H0) Observation Question Hypothesis Experiment Analysis Conclusion
  101. 101. Proof required beyond reasonable doubt In order to reject the null hypothesis Observation Question Hypothesis Experiment Analysis Conclusion
  102. 102. Variance is your estimate of uncertainty, i.e. doubt Observation Question Hypothesis Experiment Analysis Conclusion
  103. 103. Observation Question Hypothesis Experiment Analysis Conclusion Note We often choose the Minimum Variance Unbiased Estimator (MVUE)
  104. 104. Not Always MVUE Occasionally you might trade bias for variance e.g. machine learning Low variance High variance HighbiasLowbias Observation Question Hypothesis Experiment Analysis Conclusion
  105. 105. Observation Question Hypothesis Experiment Analysis Conclusion Mathematical criteria 1 Bias 2 Variance 3 System complexity
  106. 106. System complexity
  107. 107. Product development isn’t linear Observation Question Hypothesis Experiment Analysis Conclusion
  108. 108. Observation Question Hypothesis Experiment Analysis Conclusion Sometimes there are multiple potential targets
  109. 109. Observation Question Hypothesis Experiment Analysis Conclusion Or the target is partially blocked
  110. 110. Observation Question Hypothesis Experiment Analysis Conclusion Or it keeps moving
  111. 111. It can become stressful
  112. 112. There is no catch-all mathematical formula to measure and account for system complexity Observation Question Hypothesis Experiment Analysis Conclusion
  113. 113. But that doesn’t mean you shouldn’t try to estimate it and factor it into decisions Observation Question Hypothesis Experiment Analysis Conclusion
  114. 114. Search Tap Apply Interview Offer “I need a job” Hire Observation Question Hypothesis Experiment Analysis Conclusion Covered extensively in Ketan’s talk
  115. 115. Observation Question Hypothesis Experiment Analysis Conclusion
  116. 116. A (strange) American staple
  117. 117. Which also involves prediction brackets
  118. 118. You predict a winner for each game and awarded points if correct 16 9 5 4 ✅ ✅ ✅ 1 9 5 4 ̶ my prediction ̶ actual result
  119. 119. If you predict an upset early, success/failure compounds 16 9 5 4 ✅ ✅ ✅ 1 9 5 4 ̶ my prediction ̶ actual result 9 1 4 ✅ 4 4 1
  120. 120. ● Downstream compounded loss ● Number of bracket participants ● Points awarded at each stage Observation Question Hypothesis Experiment Analysis Conclusion System complexity factors
  121. 121. How to win your NCAA pool Simulate the downstream effect of all potential decisions Check whether it increases/decreases your win probability Observation Question Hypothesis Experiment Analysis Conclusion
  122. 122. Reminder - How can you do it better? 1 Determine one or more hypothesis “Does extra whitespace between job cards help job seekers to navigate quicker.” 2 Agree on the data, metrics and acceptable trade-offs up front Metrics: (i) time to click, (ii) click rate, (iii) time to hire Observation Question Hypothesis Experiment Analysis Conclusion
  123. 123. Stage 4: Experiment
  124. 124. What did we do? Ran a single treatment experiment where we simultaneously changed four components Observation Question Hypothesis Experiment Analysis Conclusion
  125. 125. Observation Question Hypothesis Experiment Analysis Conclusion
  126. 126. Why was it wrong? Observation Question Hypothesis Experiment Analysis Conclusion Couldn’t disentangle the effects of the 4 different treatments
  127. 127. How can you do it better? Observation Question Hypothesis Experiment Analysis Conclusion Run a full factorial experiment
  128. 128. Observation Question Hypothesis Experiment Analysis Conclusion Full Factorial Experiment Suggestion A: Whitespace B: Orange text C: Salary range
  129. 129. Full Factorial Experiment Observation Question Hypothesis Experiment Analysis Conclusion Increased statistical power, and simultaneous testing of interaction effects
  130. 130. Full Factorial Experiment Observation Question Hypothesis Experiment Analysis Conclusion i.e. you’ll learn more and learn quicker
  131. 131. Stage 5: Analysis
  132. 132. What did we do? 1 Cobbled data together from different sources 2 Defined different metrics 3 Invested a lot of time analysing tests Observation Question Hypothesis Experiment Analysis Conclusion
  133. 133. To consult the statistician after an experiment is finished is often merely to ask her to conduct a post mortem examination. She can perhaps say what the experiment died of. R.A. Fisher
  134. 134. Why was it wrong? Observation Question Hypothesis Experiment Analysis Conclusion Opinion-driven, time sink, unsatisfying for all involved
  135. 135. How can you do it better? Observation Question Hypothesis Experiment Analysis Conclusion With correct setup, this should be trivial
  136. 136. Existing metric New metric Existing product New product Observation Question Hypothesis Experiment Analysis Conclusion
  137. 137. Existing metric New metric Existing product Uninteresting New product Observation Question Hypothesis Experiment Analysis Conclusion
  138. 138. Existing metric New metric Existing product Uninteresting Metric Innovation New product Observation Question Hypothesis Experiment Analysis Conclusion
  139. 139. Existing metric New metric Existing product Uninteresting Metric Innovation New product Product Innovation Observation Question Hypothesis Experiment Analysis Conclusion
  140. 140. Existing metric New metric Existing product Uninteresting Metric Innovation New product Product Innovation Uninformative Observation Question Hypothesis Experiment Analysis Conclusion
  141. 141. Never use new data or metrics to validate new products! Observation Question Hypothesis Experiment Analysis Conclusion
  142. 142. Stage 6: Conclusion
  143. 143. What did we do? Observation Question Hypothesis Experiment Analysis Conclusion Drew two different conclusions
  144. 144. Why was it wrong? 1 Didn’t learn anything 2 Lost team trust Observation Question Hypothesis Experiment Analysis Conclusion
  145. 145. How can you do it better? Observation Question Hypothesis Experiment Analysis Conclusion Should follow directly from analysis
  146. 146. The Goldilocks syndrome Observation Question Hypothesis Experiment Analysis Conclusion A/B test (-1%, 1%] (1%, 5%] (5%, ∞](-5%, -1%][-∞, -5%]Outcome Conclusion too cold too cold too cold Just right, declare victory too hot
  147. 147. Retain healthy skepticism Always look for bugs Check for repeatability via holdbacks
  148. 148. The Complete Scientific Method Observation Question Hypothesis Experiment Analysis Conclusion nano, micro, macro prioritize, implicit trade-offs bias & variance, 3 metrics full factorial design trivial, no data innovation Goldilocks syndrome, repeatability
  149. 149. Observation Question Hypothesis Experiment Analysis Conclusion nano, micro, macro prioritize, implicit trade-offs bias & variance, 3 metrics full factorial design trivial, no data innovation Goldilocks syndrome, repeatability
  150. 150. Data-driven can be disorientating in a world of abundant data Be science-driven, i.e. use the scientific method to add necessary structure Invest in the observation, question and hypothesis stages Parting Thoughts
  151. 151. ~ finn ~

×