Sentiment is Shite (2014)
•
13 j'aime•4,283 vues
Slide deck from a talk given at the BreakingNews.ie Measurement Conference on 10th September 2014. It's an updated version of a talk I gave (only once) in 2011. My understanding (and the technology) have developed since then.
Recommandé
Recommandé
Contenu connexe
En vedette
En vedette (13)
Similaire à Sentiment is Shite (2014)
Similaire à Sentiment is Shite (2014) (20)
Plus de Mat Morrison
Plus de Mat Morrison (20)
Dernier
Dernier (20)
Sentiment is Shite (2014)
- 1. NEGATIVE SENTIMENT another grumbly presentation from @mediaczar
- 2. TWITTER VOLUME 15 10 5 0 1 Jun 2014 1 Jul 2014 Thousands “facebook” AND (“newsfeed” OR “news feed”) June 1 – July 31 2014 Data: Netbase
- 3. WHAT HAPPENED HERE? aND HOW DOES THAT MAKE YOU FEEL? Topic Analysis Emotions Data: Brandwatch, Netbase
- 4. DELOITTE: BALANCE OF SENTIMENT AFFECTS SALES The balance of sentiment in Tweets is a more powerful driver of sales than reach (or volume) alone, with positive Tweets having generally a higher impact than negative Tweets. Therefore, to gain the most out of the online word-of- mouth embodied by Tweets, companies would be best served by addressing the balance of sentiment about their games through increasing the number of positive Tweets. 6.10% 3.30% 1.60% 30% more positive Tweets 30% fewer negative Tweets 30% more non-Twitter advertising Deloitte, “Tweets for Sales, Gaming” (2013)
- 5. “SENTIMENT” HAS BECOME A MARKETING OBJECTIVE We want to increase the positive buzz around Your brand
- 6. “SENTIMENT” HAS BECOME A MARKETING OBJECTIVE Real case study. Name obscured to protect the innocent.
- 7. THIS IS HONEST ABE Public sentiment is everything. With public sentiment, nothing can fail. Without it, nothing can succeed.
- 8. I’M GOING TO BE EVEN MORE HONEST SENTIMENT ANALYSIS IS SHITE! DTEOLNL'T U HSO WLDH ABTA CYKO.U. .REALLY THINK!
- 9. THE PROMISE OF SOCIAL INTELLIGENCE AN ALMOST INFINITE SOURCE OF QUAL QUANT DATA! FINALLY THEY WILL REVEAL WHAT THEY REALLY THINK!
- 10. RELEVANCE VOLUME AUTHORITY TOPICS SENTIMENT
- 11. WEIGHTED SENTIMENT, MRS BROWN’S BOYS, JULY 2014 14.8% 12.2% 11.4% 7.3% -4.1% -5.2% -6.3% -8.5% POSITIVE NEGATIVE 4 DIFFERENT TOOLS GIVE 4 DIFFERENT MEASURES OF SAME DATA
- 12. METHODOLOGICAL PROBLEMS LEXICAL ANALYSIS CLASSIFICATION MANUAL INPUT
- 13. LEXICAL ANALYSIS visit: http://www.wjh.harvard.edu/~inquirer/spreadsheet_guide.htm e.g. Harvard General Enquirer: 11.8K categorised, tagged words
- 14. QUICK DIRTY DIY SENTIMENT ANALYSIS TOOL
- 16. CAN’T HANDLE IDIOMS, SYNONYMS (OR IRONY) WELL, THAT'S JUST GREAT
- 17. FACEBOOK USED LEXICAL APPROACH “Posts were determined to be positive or negative if they contained at least one positive or negative word” Visit: http://www.pnas.org/content/111/24/8788.full LIWC: http://www.liwc.net/ 15 10 5 0 Thousands
- 18. CLASSIFIERS SUPERVISED LEARNING… TRAIN MODEL TAGGED TEST MODEL DATA TRAINING DATA TEST DATA POSITIVE NEUTRAL NEGATIVE
- 19. THIS PRESENTATION IS GOOD visit: text-processing.com/demo/sentiment CLASSIFIER SAYS “POSITIVE”
- 20. THIS PRESENTATION IS BAD CLASSIFIER SAYS “NEGATIVE” visit: text-processing.com/demo/sentiment
- 21. THIS PRESENTATION IS NOT GOOD CLASSIFIER SAYS “NEGATIVE” visit: text-processing.com/demo/sentiment
- 22. THESE ARE BOTH POSITIVE analysed analysed
- 23. THIS IS NEUTRAL analysed
- 24. WHAT ABOUT THIS?
- 25. DOES IT UNDERSTAND WHAT IT’S READING? 2. 1. Take the original text Randomise the word order 3. Re-test recipe: http://stackoverflow.com/questions/17825945/generating-a-list-of-random-words-in-excel-but-no-duplicates
- 26. WORD ORDER MAKES NO DIFFERENCE
- 27. WORD ORDER MAKES NO DIFFERENCE
- 28. WORD ORDER MAKES NO DIFFERENCE
- 29. DOMAIN SPECIFIC Models trained on one set of data may not work well on other sets
- 30. WHAT ABOUT HUMAN MARKERS?
- 31. FAIRLY EASY TO ASSESS ENTERTAINMENT CATEGORY
- 32. EXPERIMENT: IS GUINNESS GOOD FOR YOU? Selected 50 positive and 50 negative tweets as scored by classifier. Passed these tweets to human markers. Each tweet scored 3 times (5 point scale) Average score compared to classifier.
- 33. MECHANICAL TURKS! Visit: http://www.crowdflower.com/
- 34. RESULTS: IS GUINNESS GOOD FOR YOU? NEG NEUT POS CLASSIFIER 50 0 50 MANUAL 23 30 47 AGREEMENT 40% 0% 72% (Agreement based on #tweets with matched judgments)
- 35. HUMANS DON’T ALWAYS AGREE… Advert used to say #Guinness is good for you but I think it is not acceptable to say that these days, but in moderation I thrive on it at 70 It's called a rotten apple #twobeersonecup #Guinness #angryorchard #delicious http://t.co/GUFxrYoF6i
- 36. CLASSIFY THIS… THERE ARE HUGE LINES AT THE APPLE STORE TODAY OR
- 37. BIG ENOUGH NUMBERS If the sample is large enough, won’t these problems get ironed out?
- 38. SAMPLE BIAS (1936 US PRESIDENTIAL ELECTION) SURVEY SIZE ROOSEVELT LITERARY DIGEST 2,400,000 43% GALLUP 50,000 54% ACTUAL 61% See: Tim Harford, “Big Data: are we making a big mistake?” (FT Magazine, 28 March 2008) I THINK YOU'll find iT'S 48 times bigger ALF LANDON
- 39. WIN 30 MINS OF FREE CONSULTANCY (VALUE £750) I'm loving #breakingnewsconf, @mediaczar. IT’S FAR TOO EASY TO GAME “POSITIVE SENTIMENT” METRICS
- 40. RECOMMENDATIONS LIFE ISN’T A POPULARITY CONTEST DON’T LET SENTIMENT BECOME A KPI MAKE SENTIMENT A TOOL FOR MORE COMPLEX RESEARCH PUT GREAT TECH TO MORE MEANINGFUL USE
- 41. THANK YOU! PLEASE DON’T ASK ME ANY TRICKY QUESTIONS THAT WILL MAKE ME LOOK STUPID I’M @MEDIACZAR ON TWITTER FEEL FREE TO COME AND TALK TO ME AFTERWARDS
Notes de l'éditeur
- "respondents who returned their questionnaires represented only that subset of the population with a relatively intense interest in the subject at hand, and as such constitute in no sense a random sample... it seems clear that the minority of anti-Roosevelt voters felt more strongly about the election than did the pro-Roosevelt majority.” 50x bigger sample. Wrong answer.