Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

What it costs to be rubbish at risking

388 vues

Publié le

How to identify and quantify systematic biases in probabilistic assessments and how to assess the financial consequences of those biases

Publié dans : Données & analyses
  • Soyez le premier à commenter

What it costs to be rubbish at risking

  1. 1. DECISION RISK ANALYTICS confidence through insight What it costs to be rubbish at risking THE DISUTILITY OF POOR PROBABILISTIC PREDICTION
  2. 2. Faithful assessment of subsurface uncertainty What does good look like? Faithful assessments converge on aggregate ● Only way to reduce uncertainty enough to say be able to anything meaningful about the quality of assessments is to aggregate a large number of outcomes ● Most of the detail of the uncertainty of the individual prospects is lost. ● Only the expected value and the variance is preserved from the individual distributions to the aggregation ● So we can’t really audit the detail of individual distributions Single prospect 100 prospects One prospect, Chance of success 20% Two plausible outcomes 100 prospects Chance of success 20% 2100 outcomes Some of which fairly unlikely One discovery, P50: 30, P10/P90: 20 100 discoveries P50: 60, P10/P90: 1.5
  3. 3. Derived assessment of economic uncertainty Faithful assessment of subsurface uncertainty What does good look like? Portfolio decisions made on faithful assessments maximize value Assessments of economic uncertainty Portfolio aggregation Maximize reward for risk Expectation and variance dominate results of subsurface assessments Portfolio aggregation (mostly) washes out everything but expectation and variance Faithful assessments converge on aggregate
  4. 4. Bayesian bias Bias Bayes’ theorem Biassed assessment of subsurface uncertainty Faithful assessment of subsurface uncertainty Portfolio decisions made on faithful assessments maximize value Faithful assessments converge on aggregate ● Impossible to find individual errors in individual prospects, so look for systematic biases ● What can go wrong? ● Use Bayes theorem to model these biases in a mathematically consistent way Probabilities Volume distributions Expectation Low Low Expectation low High High Expectation high Variance Low Polarized Ranges too narrow High Centred Ranges too wide
  5. 5. Motivation Biassed predictions Portfolio decisions based on biassed assessments What do consistent biases look like when you compare prediction with outcomes. Can we elicit biases? How much money do we expect to lose when we make decisions based on biased assessments? Lookback Disutility Portfolio decisions made on faithful assessments maximize value Faithful assessments converge on aggregate Bias Bayes’ theorem Biassed assessment of subsurface uncertainty Faithful assessment of subsurface uncertainty
  6. 6. Pangloss Oil and Gas No oil left behind Small specialist exploration company. Humble with respect to uncertainty, but biased after their first, lucky, large discovery. Probabilities and P50s are consistently lifted relative to true distributions Arete Resources The beginning of wisdom is the awareness of ignorance Envy of the industry for their faithful predictions Predictions are the probabilities from which the outcomes are sampled The Aleatoric Gulf Eeyore Enterprises Every silver lining has its cloud Lucky to survive a run of failures and non-commercial discoveries, Eeyore Enterprises are consistently conservative. Probabilities and P50s are consistently depressed relative to true distributions Hubris Industries World class geoscience Small exploration company made from ex-chiefs from large companies. Solid predictions on average very sure of themselves. Probabilities are polarized relative to average and ranges are narrowed Sybil Better vaguely right than exactly wrong Culture of deciding by committee and can’t agree on anything. All probabilities close to base rate and very large P10/P90 ratios. Probabilities are centred on average and ranges are widened
  7. 7. Risk biases 20-40 40-600-20 80-10060-80 Unbiased probability Biased probability Unbiased probability Biased probability Unbiased probability Elicited bias Actual bias Elicited bias Actual bias
  8. 8. Volume biases Log-probit plot of samples from unbiased and biased lognormal distribution If all volume distributions had the same parameters, the effects of optimism / pessimism and overconfidence / vagueness, could be seen on log-probit plots Empirical distribution percentile plot To deal with the multiplicity of distributions, we look at the distribution of the realized percentiles. Error bars are crucial to see whether the deviations are significant given the number of data Probit(rank / N+1) log(V) Optimism and pessimism shift the line vertically where it crosses the P50 Overconfidence flattens the line, reducing the P10/P90 ratio rank / N+1 Realizedpercentile P75 P50 P100 P0 P25
  9. 9. Volume optimism and overconfidence 20-40 40-600-20 80-10060-80 Unbiased uniform distribution (with ranges) Unbiased uniform distribution (with ranges) Unbiased uniform distribution (with ranges) Unbiased uniform distribution (with ranges) Predicted biased distribution Elicited distribution Empirical distribution Predicted biased distribution Predicted biased distribution Elicited distribution Empirical distribution Predicted biased distribution P20 P60 P100 P0 P80 P40 P20 P60 P100 P0 P80 P40 P20 P60 P100 P0 P80 P40 P20 P60 P100 P0 P80 P40
  10. 10. Making money in the Aleatoric gulf Exploration well outcome Decision to drill Discovered volume p 1-p V 0 -C NPV(V)-C V NPV(V) MCFS Expected value in the case of a discovery: NPV(V)-C Expected value in the case the well is drilled: p NPV(V)-C Drill if p NPV(V)-C > 0
  11. 11. Losing money in the Aleatoric gulf Exploration well outcome Decision to drill Discovered volume p 1-p V 0 -C NPV(V)-C V NPV(V) MCFS Expected value in the case of a discovery: NPV(V)-C Expected value in the case the well is drilled: p NPV(V)-C Drill if p NPV(V)-C > 0 Expectation taken using biased volume distribution Biased probability
  12. 12. Losing money in the Aleatoric gulf Exploration well outcome Decision to drill Discovered volume p 1-p V 0 -C NPV(V)-C Expected value in the case of a discovery: NPV(V)-C Expected value in the case the well is drilled: p NPV(V)-C Drill if p NPV(V)-C > 0 Expectation taken using biased volume distribution Biased probability ● Lose a positive EMV if bias leads you to drop a well you would have drilled without bias ● Gain a negative EMV if bias leads you to drill a well you would have dropped without bias ● How much you lose is obviously strongly dependent on the prospect: ○ Dead certain and certainly dead aren’t affected ● To assess overall value erosion, we need to look at a portfolio of prospects with a range of different p, MCFS, p, C, etc.
  13. 13. ● Portfolio based on global historical averages ● Note asymmetry in probability bias Fortune favours the brave ● Results remarkably indifferent to risk profile ● Most sensitive to exploration well costs (relatively) ● Volume over-confidence is consistently the biggest eroder of value
  14. 14. ● Introducing some random error into unbiased assessments ● In most cases, value erosion is unaffected by random noise for large bias parameters ● However (predictably) probability confidence is made substantially worse (magnifying errors) ● Volume confidence (rather less predictably) is relatively unaffected
  15. 15. Outcome Volume Optimism Probabilities consistently too high. Visible on standard cumulative sequence and sliding window plots with ranges P50 volumes too high Visible on standard cumulative sequence and sliding window plots with ranges (discoveries only) Pessimism Probabilities consistently too low. P50 volumes too low Overconfidence Probabilities too polarized. Hard to spot without quite a lot of data Confidence parameter elicitation Ranges too narrow Empirical distribution plot shows this clearly. Underconfidence Probabilities too close to baseline. Ranges too wide A taxonomy of systematic bias ● Can use methods presented here to elicit bias and confidence parameters
  16. 16. DECISION RISK ANALYTICS confidence through insight Back up
  17. 17. Bayes’ theorem Prior ignorance: Expectation Variance Datum Bayes’ theorem Posterior ignorance Expectation Variance true false P(A) 1-P(A) P(A) Variance Datum B is an observation related to A ● Doesn’t have to be binary or even discrete ● For example, test result, score (1-5), index P(B | A true) > P(B | A false) P(B | A true) < P(B | A false) Amount probability moves depends where it starts Low prior probability, positive evidence more likely to be false positive High prior probability, negative evidence more likely to be false negative ExpectationExpectation Variance Estimate Datum most consistent with higher mean Datum most consistent with lower mean Uncertainty Datum found close to prior mean Datum found far from prior mean
  18. 18. Bayesian bias* Bias Bayes’ theorem true false P(A) 1-P(A) P(A) Optimism Probabilities raised in a way consistent with the observation of a particular piece of (spurious) positive evidence Pessimism Probabilities lowered in a way consistent with the observation of a particular piece of (spurious) negative evidence Overconfidence Probabilities moved further from the centre. High probabilities are raised, low probabilities are lowered. Choose to centre at 0.5 or mean Vagueness Probabilities moved closer to the centre. High probabilities are lowered, low probabilities are raised. Choose to centre at 0.5 or mean Optimism (Pessimism) Mean shifted to the right (left) in a way consistent with the observation of a particular piece of (spurious) positive evidence (P50 shifted by a constant factor) Overconfidence (Vagueness) Variance increased (decreased) as if there were additional consistent data. P10/P90 ratio increased (decreased) by a constant factor *Bias understood in the statistical sense, i.e. a systematic distortion not accounted for in derivation Variance ExpectationExpectation Variance Prior ignorance: Expectation Variance Posterior ignorance Expectation Variance

×