Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Expert Estimation and Calibration Training - SIRAcon 2019

84 vues

Publié le

Slides from Tony Martin-Vegue's presentation at SIRAcon (Cincinatti, OH) on April 30, 2019

Abstract:
We make estimates every day in the process of performing risk assessments. We regularly estimate the probability of a data breach, effectiveness of awareness training or projected staffing levels for the next 5 years.

Here’s the problem: humans are horrible at making estimates! All sorts of bias cloud our judgement, making it difficult to make good security decisions. Here’s the good news: it is possible to overcome some of these inherent biases with many of the same techniques that professional bookmakers use to set odds when placing and taking bets. Attend this hands-on session to learn how to overcome your bias and become a better estimator. All attendees will take a test to determine estimation skills and will receive personalized feedback on what kind of bias is present. Bring a $20 bill to place bets – don’t worry, you’ll get it back!

Publié dans : Données & analyses
  • Soyez le premier à commenter

Expert Estimation and Calibration Training - SIRAcon 2019

  1. 1. #SIRAcon 1 Expert Estimation and Calibration Training
  2. 2. #SIRAcon About Me Tony Martin-Vegue www.tonym-v.com tony.martinvegue@gmail.com @tdmv • 20 years in Technology; last 10 in Cyber Risk • FAIR practitioner for about 8 years now • Reside in the Bay Area
  3. 3. #SIRAcon • What is this? • Quiz 1 • Calibration Training • Quiz 2 • Q&A / Wrap up 3 Agenda
  4. 4. #SIRAcon4
  5. 5. #SIRAcon 5 Quiz 1
  6. 6. #SIRAcon • Instructions: • Go to this link • This is not pub trivia : I don’t care if you get the answer right! • I only care if you think you got the answer right • Resist the urge to Google the answers or ask your neighbor! 6 http://bit.ly/2UVsP68 Quiz 1
  7. 7. #SIRAcon • Who are the superforecasters? • Why? • Are you surprised by your results? Results
  8. 8. #SIRAcon Calibration Training 8
  9. 9. #SIRAcon Expert Estimation • Interdisciplinary • Acquired knowledge • Predictive judgements
  10. 10. #SIRAcon
  11. 11. #SIRAcon •Data is not available, or not available to you •Data is too expensive (see “value of information” principles) •You have data, but it needs adjustment • You are forecasting some change in the landscape that just using historical data won’t reflect, or • Need an interpretation of incomplete, missing data or there’s a degree of uncertainty When to Use Expert Judgement
  12. 12. #SIRAcon Some Cognitive Biases
  13. 13. #SIRAcon Being Calibrated
  14. 14. #SIRAcon Calibration Test
  15. 15. #SIRAcon Name Question score Calibration Score Calibration Respondent 1 8/10 8.2 Perfectly calibrated Respondent 2 7/10 8.2 Slightly overconfident Respondent 3 8/10 8.6 Perfectly calibrated Respondent 4 7/10 7.4 Perfectly calibrated Respondent 5 8/10 7.7 Perfectly calibrated Respondent 6 6/10 7.2 Slightly overconfident Respondent 7 9/10 7.2 Underconfident Respondent 8 6/10 7.8 Overconfident Respondent 9 7/10 6.9 Perfectly calibrated Respondent 10 5/10 6.7 Overconfident Respondent 11 8/10 7.7 Perfectly calibrated Respondent 12 5/10 7.4 Overconfident Respondent 13 7/10 7.1 Perfectly calibrated Respondent 14 8/10 7.1 Perfectly calibrated Respondent 15 8/10 8.7 Perfectly calibrated An Example
  16. 16. #SIRAcon • Substitute for constant feedback • Questions do not need to be cyber related (pub trivia) • Simply training people to recognize bias Make forecast Observe event Receive feedback on prior forecast Why Does Trivia Work?
  17. 17. #SIRAcon Respondent Calibrated Min Mode Max Respondent 1 Yes 10 25 35 Respondent 2 No 27 32 34 Respondent 3 Yes 15 35 65 Respondent 4 Yes 1 5 36 Respondent 5 Yes 1 2 65 Respondent 6 No 20 25 40 Respondent 7 Yes 10 20 60 Respondent 8 Yes 1 50 100 Respondent 9 No 27 30 34 Respondent 10 No 25 31 35 Respondent 11 Yes 0 5 40 Respondent 12 No 5 10 20 Respondent 13 No 1 5 20 Respondent 14 No 5 35 80 Respondent 15 Yes 20 30 40 In Practice…
  18. 18. #SIRAcon Controlling for Bias
  19. 19. #SIRAcon Overconfident professionals sincerely believe they have expertise, act as experts and look like experts. You will have to struggle to remind yourself that they may be in the grip of an illusion. - Daniel Kahneman ” “
  20. 20. #SIRAcon Lesson 1: Accuracy versus Precision William Tell is forced by the tyrant Gessler to shoot an apple from his son's head
  21. 21. #SIRAcon 21 Accurate Precise Accuracy vs Prediction
  22. 22. #SIRAcon 22 How Much Does a Ford Pinto Weigh? 1,376.762 pounds is precise but not accurate 2,015–2,270 pounds accurate, but not precise Accuracy vs Prediction: An Example
  23. 23. #SIRAcon “Perfect is the enemy of good.” - Italian Proverb 23
  24. 24. #SIRAcon Lesson #2: Decomposition
  25. 25. #SIRAcon But I Don’t Know Anything About That • Start with the absurd • Apply anything that you know • Decompose it! • Can you bring the ranges in? • Remember the goal: accuracy, not precision 25
  26. 26. #SIRAcon What is the Wingspan of a 747? 26 • Start with the absurd • Apply anything that you know • Decompose it! • Can we bring the ranges in? • Goal: accuracy, not precision
  27. 27. #SIRAcon 27
  28. 28. #SIRAcon Lesson #3: Equivalent Bet Test
  29. 29. #SIRAcon Game Rules: • Place $20 down to play the game; house also places down $20 • I’m going to ask you to provide an estimate within a range in which you are 90% confident the answer is correct • You now have choice as to which game to play: see if your answer was right or spin the wheel How This Works
  30. 30. #SIRAcon How many credits does Steven Spielberg have as director? Game #1: Movie Trivia
  31. 31. #SIRAcon $20$20 WinLose SPIN Equivalent Bet Test Make your choice! Game 1: Spin the wheel Game 2: See if your answer was right
  32. 32. #SIRAcon Answer: 55 (source: imdb.com)
  33. 33. #SIRAcon The idea behind the equivalent best test is to test whether or not you are truly 90% confident about your estimation • If you choose option a, spin the wheel – you are less than 90% confident about your estimation (ranges are too tight) • If you choose option b, try your luck with your answer – you are more than 90% confident (ranges are too wide) • The perfect balance would be that you don’t care. Why This Works
  34. 34. #SIRAcon How tall is Steph Curry? Game #2: Sports Trivia
  35. 35. #SIRAcon $20$20 WinLose SPIN Equivalent Bet Test Make your choice! Game 1: Spin the wheel Game 2: See if your answer was right
  36. 36. #SIRAcon Source: User JeopardyTempest @ Sports Stack Exchange How Tall is Steph Curry? Answer: 6’3” (source: nba.com)
  37. 37. #SIRAcon Lesson #4: How might you be right or wrong?
  38. 38. #SIRAcon Why might you be wrong? • Anchoring • Blind spot bias • Confirmation bias • Dunning–Kruger effect Why might you be right? • You are an expert • Did you apply everything you already know? • Evaluated the options • Thought about the probability of being right Considering Pros and Cons
  39. 39. #SIRAcon Lesson 5: Probability “Kick-up at the Hazard Table” -- Painting by Thomas Rowlandson
  40. 40. #SIRAcon Lesson 5: Probability Source: https://www.mathsisfun.com/data/probability.html
  41. 41. #SIRAcon Question: •”Ramoray” is Chandler’s last name in the sitcom Friends. True or false? •Confidence that you are correct? (50%, 60%, 70%...) 41 Applying to Binary Questions
  42. 42. #SIRAcon 42 • Apply anything that you know • Decompose it! • Make a bet with yourself • Can you improve your confidence rating? (Is your guess better than a coin flip?) http://bit.ly/2VvLept Quiz 2
  43. 43. #SIRAcon More Calibration Quizzes • The Credence Calibration Game - http://acritch.com/credence-game/ • Calibrated Probability Assessments - http://calibratedprobabilityassessment.org/ Measurement • How to Measure Anything by Douglas Hubbard • How to Measure Anything in Cyber Risk by Douglas Hubbard and Richard Seiersen Cognitive Biases • Thinking Fast and Slow by Daniel Kahneman Expert judgement • The Wisdom of Crowds – by James Surowiecki • Superforecasters – by Philip Tetlock • The Good Judgement Project - https://goodjudgment.com/ Further Reading
  44. 44. #SIRAcon

×