Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Distilling the Crowd
Jon Puleston
VP Innovation
28th Nov 2016
The next evolutionary step in crowd
prediction techniques
This is the story of the development of a research methodology
Market research is not so good at predicting certain
things…
Will you buy
this product?
Does this
ad work?
Who will win
th...
Political
Polling
• The US election poll error = 2%
• EU referendum error = 4%
• UK election error = 5%
Show me a brand that would not be ha...
• What respondents say and what they do is so often different – we are often not
very good at observing our own behaviour
...
• Well, for the last few years we have been very interested in prediction protocols
THE WISDOM OF THE CROWDS
• Instead of ...
• 100 year old evidence to show that when used in the right way, it can be
uncannily accurate
• Accesses a different part ...
• We have had some success at using this approach
• It can be really effective for ad testing…
WHAT WE LIKE ABOUT PREDICTI...
• They can miss fire ~ 15% of the time
• Highly sensitive to a wide array of cognitive biases
WHAT WE DON’T LIKE ABOUT PRE...
35% LIKE BEST
9% PREDICT THIS IS
MOST POPULAR
25% PREDICT THIS MOST POPULAR
Prediction error:
FAMILIARITY DOES NOT EQUAL L...
PREDICT IF THIS A GOOD AD?
“Its an Apple ad and
everything they do is great,
right!”
some people find it easier to put the...
• Crowds are particularly rubbish at predicting things like elections!
• Our predictions for many people are really proxie...
• But its more complicated than that
…we now all live in social bubbles, so even if we are able to step back from what we
...
ME!
The minority illusion
More social people have more connections but not necessarily the same
points of view as the crowd
This person is a
socialite has lots of
dinner parties. She
also goes to London
a lot where blue is
the most fashionable
co...
SOLUTIONS?
Isolating the good
predictors
Philip Tetlock
THE GOOD JUDGEMENT PROJECT
PHILIP TETLOCK’S GOOD JUDGEMENT PROJECT HAS DEMONSTRATED THAT
SOME PEOPLE ARE A LOT BETTER AT PREDICTING THAN OTHERS
By as...
LEARNING FROM THE GOOD JUDGEMENT PROJECT
• Those with an actively open mind-set are better at making predictions
• Knowled...
Are researchers good predictors?
Kantar prediction experiment….examples of what we asked them to predict:
Competitive Brand performance
Global economic gro...
48%
25%
Kantar staff Consumer
Close Predictions
Who will win the US election: Kantar Staff prediciton
80%
20%
Clinton Trump
How many superforecasters predicted the US election?
Even if they are brilliant at predicting, if the raw data they are wo...
Solution: Find the people with the best
vantage point on what is going on
C
C
A
B
Predicting what coloured bathrooms people have…
C
C
A
B
Person A, has a blue bathroom and so do all their friends so predicts
blue
C
C
A
B
Person B, has an orange bathroom and so do most of their friends
so predicts orange
C
C
A
B
Solution = Ask the plumbers
ME!
Research people
in this territory
• Find a group of people with good judgement but also with a good vantage point!
Then combine these 2 bits of thinking…
Go...
Predict the polls experiment
• Regular monthly survey in the 5 months leading up to the UK election
• Each month a group o...
DISTILLING THE EXPERTS
Scored each respondent each month for the prediction accuracy
Isolated the top 100 predictors and a...
It worked…sort of
EU Referendum forecasting leave…
Issue
F#@k me it is a lot of work
to find, calibrate and keep these people!
Issue 2
We had our suspicions that we were jus...
Could we apply this technique to something that
does not occur once in a blue moon?
Distilling a crowd who could predict the success of movies
Predicting box office takings…
Results of experiments
• Recruited 1,500 participants in the UK & USA
• Survey experiment ran for 5 months with adapted ve...
Could we switch to something that
was easier to calibrate?
Price testing experiment
48
A simple test taking 6 minutes
Ad testing experiment
50
51
Source: Based on an average of 4 large scale experiments
= 90% better
Crowd accuracy (Assuming no network errors)
Accuracy of individual predictors
100
people
50
people
20
people
10
people
5
p...
• Crowd distillation techniques I believe is the next evolutionary step in crowd prediction methodology
• The maths of exp...
Distilling the crowd: the next evolutionary step in crowd wisdom
Distilling the crowd: the next evolutionary step in crowd wisdom
Distilling the crowd: the next evolutionary step in crowd wisdom
Distilling the crowd: the next evolutionary step in crowd wisdom
Distilling the crowd: the next evolutionary step in crowd wisdom
Prochain SlideShare
Chargement dans…5
×

Distilling the crowd: the next evolutionary step in crowd wisdom

646 vues

Publié le

Jon Puleston
Vice-president of innovation, Lightspeed

  • Get Automated Computer NFL,MLB,Soccer picks [$127,999 profit verified] ▲▲▲ http://scamcb.com/zcodesys/pdf
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • 80% Win Rate? It's Not a BUG? [Proof Inside] ◆◆◆ http://scamcb.com/zcodesys/pdf
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • Soyez le premier à aimer ceci

Distilling the crowd: the next evolutionary step in crowd wisdom

  1. 1. Distilling the Crowd Jon Puleston VP Innovation 28th Nov 2016 The next evolutionary step in crowd prediction techniques
  2. 2. This is the story of the development of a research methodology
  3. 3. Market research is not so good at predicting certain things… Will you buy this product? Does this ad work? Who will win the election?
  4. 4. Political Polling
  5. 5. • The US election poll error = 2% • EU referendum error = 4% • UK election error = 5% Show me a brand that would not be happy with a 2-5% error margin on volumising their market? BEFORE WE BEAT OURSELVES UP TOO MUCH ABOUT THIS LET’S CONSIDER… 5 No other method not reliant traditional market research polling information gets anywhere near doing any better: • Social media analytics directionally helpful but proven completely unreliable at predicting • Betting markets have not only miss-called all 3 votes, they have magnified the error
  6. 6. • What respondents say and what they do is so often different – we are often not very good at observing our own behaviour • Often asking people questions they don’t know the answer themselves to, or are too difficult to work out • Not always asking the right question in the right way to get the right answer • Sometimes compounded by not asking the right people and the difficulties of accessing representative samples DO I NEED TO EXPLAIN TO A ROOM FULL OF RESEARCHERS WHY PREDICTION IS SO DIFFICULT? 6
  7. 7. • Well, for the last few years we have been very interested in prediction protocols THE WISDOM OF THE CROWDS • Instead of asking people what they will do, will they buy a product, you turn then tables and ask them to predict the outcome – who will win, will the product be successful • We have conducted about 3 years’ worth of experiments looking at the differences between traditional and predictive research approaches SO WHAT ARE THE ALTERNATIVES? 7
  8. 8. • 100 year old evidence to show that when used in the right way, it can be uncannily accurate • Accesses a different part of the brain “we think” • You can sculpt a different type of survey using these techniques, that is more fun • With a prediction you can feed back the answer and turn it into a game WHAT WE LIKE ABOUT PREDICTION PROTOCOLS 8
  9. 9. • We have had some success at using this approach • It can be really effective for ad testing… WHAT WE LIKE ABOUT PREDICTION PROTOCOLS 9 0.89 correlation 5x differentiation
  10. 10. • They can miss fire ~ 15% of the time • Highly sensitive to a wide array of cognitive biases WHAT WE DON’T LIKE ABOUT PREDICTION PROTOCOLS 10
  11. 11. 35% LIKE BEST 9% PREDICT THIS IS MOST POPULAR 25% PREDICT THIS MOST POPULAR Prediction error: FAMILIARITY DOES NOT EQUAL LIKING
  12. 12. PREDICT IF THIS A GOOD AD? “Its an Apple ad and everything they do is great, right!” some people find it easier to put themselves in someone else's shoes, some people find it easier to step back and look at things more objectively.
  13. 13. • Crowds are particularly rubbish at predicting things like elections! • Our predictions for many people are really proxies for what they want to happen… WHAT WE HAVE LEARNT… Predict who will win the election
  14. 14. • But its more complicated than that …we now all live in social bubbles, so even if we are able to step back from what we want to happen and look out and all we see we see is that nearly everyone has the same opinions as us WHAT HAVE WE LEARNT? 15
  15. 15. ME! The minority illusion
  16. 16. More social people have more connections but not necessarily the same points of view as the crowd
  17. 17. This person is a socialite has lots of dinner parties. She also goes to London a lot where blue is the most fashionable colour Thesr 2 people are also more social and aware that blue is the trend colour The rest of this crowd are less social don’t have many dinner parties so have not seen many other peoples bathrooms The minority illusion: predict what coloured bathrooms people have? The crowd predicts blue!
  18. 18. SOLUTIONS?
  19. 19. Isolating the good predictors
  20. 20. Philip Tetlock THE GOOD JUDGEMENT PROJECT
  21. 21. PHILIP TETLOCK’S GOOD JUDGEMENT PROJECT HAS DEMONSTRATED THAT SOME PEOPLE ARE A LOT BETTER AT PREDICTING THAN OTHERS By asking people to make ongoing predictions you can begin to calibrate them and cream off the good predictors into panels of super predictors
  22. 22. LEARNING FROM THE GOOD JUDGEMENT PROJECT • Those with an actively open mind-set are better at making predictions • Knowledge and understanding of the topic is important • For more complex predictions, numeracy and intelligence are important • Each participant needed to make 20 predictions before you could tell if they were any good or not
  23. 23. Are researchers good predictors?
  24. 24. Kantar prediction experiment….examples of what we asked them to predict: Competitive Brand performance Global economic growth figures TV audience figures Global warming estimates Assessment of fashion trends Performance of teams in the Olympics Outcome of a variety of political events
  25. 25. 48% 25% Kantar staff Consumer Close Predictions
  26. 26. Who will win the US election: Kantar Staff prediciton 80% 20% Clinton Trump
  27. 27. How many superforecasters predicted the US election? Even if they are brilliant at predicting, if the raw data they are working from is flawed they will simply amplify the error (the minority illusion) Some poll agitators e.g. Polypoll were predicting Clinton victory with 99% certainty
  28. 28. Solution: Find the people with the best vantage point on what is going on
  29. 29. C C A B Predicting what coloured bathrooms people have…
  30. 30. C C A B Person A, has a blue bathroom and so do all their friends so predicts blue
  31. 31. C C A B Person B, has an orange bathroom and so do most of their friends so predicts orange
  32. 32. C C A B Solution = Ask the plumbers
  33. 33. ME! Research people in this territory
  34. 34. • Find a group of people with good judgement but also with a good vantage point! Then combine these 2 bits of thinking… Good Judgement Plumber Plumbers with good judgement
  35. 35. Predict the polls experiment • Regular monthly survey in the 5 months leading up to the UK election • Each month a group of 1,500 randomly recruited panellists were asked to predict the opinion polls
  36. 36. DISTILLING THE EXPERTS Scored each respondent each month for the prediction accuracy Isolated the top 100 predictors and aggregated their predictions Compared their predictions to the average from the 4 leading polls
  37. 37. It worked…sort of
  38. 38. EU Referendum forecasting leave…
  39. 39. Issue F#@k me it is a lot of work to find, calibrate and keep these people! Issue 2 We had our suspicions that we were just lucky. The only way to validate the method would be to test out on 20 or so similar 50/50 elections which come round once in a blue moon (we thought!)
  40. 40. Could we apply this technique to something that does not occur once in a blue moon?
  41. 41. Distilling a crowd who could predict the success of movies
  42. 42. Predicting box office takings…
  43. 43. Results of experiments • Recruited 1,500 participants in the UK & USA • Survey experiment ran for 5 months with adapted versions in each market • 70% of participants completed all 5 waves • Collected a total of 38 film predictions • The average participant correctly predicted the box office performance (+/-20%) with 40% accuracy • Isolated a group of the top 100 participants who predicted box office takings (+/-20%) with an average accuracy of 60% 45
  44. 44. Could we switch to something that was easier to calibrate?
  45. 45. Price testing experiment 48 A simple test taking 6 minutes
  46. 46. Ad testing experiment 50
  47. 47. 51 Source: Based on an average of 4 large scale experiments = 90% better
  48. 48. Crowd accuracy (Assuming no network errors) Accuracy of individual predictors 100 people 50 people 20 people 10 people 5 people 50% 50% 50% 50% 50% 50% 55% 86% 80% 74% 74% 60% 60% 98% 94% 85% 82% 68% 65% 100% 99% 95% 91% 79% 70% 100% 100% 98% 95% 84% 75% 100% 100% 99% 99% 91% 80% 100% 100% 100% 99% 94% 85% 100% 100% 100% 100% 97% Sample Prediction maths
  49. 49. • Crowd distillation techniques I believe is the next evolutionary step in crowd prediction methodology • The maths of expert crowd prediction work on smaller numbers - whether or not the market is ready for a expert micro-sampling techniques is another matter • It takes work to distil a crowd – finding and maintaining relationships with good predictor is not easy • Creativity has been require to think how to apply this methodology within the Market research arena Summary & learnings 54

×