I used to believe that sentiment analysis was one of the most important tools in a smart marketer's box.
The rise of social media offered (I thought) a huge, free, and above all, reliable source of both qualitative and quantitative data about how real people really feel about brands, and how our marketing contributed to those feelings.
Over the years, though, I've become increasingly disappointed in both the reality and the promise; to the point at which I'm actively recommending that our clients avoid using social media sentiment in their models or as a performance metric.
It's time for us all to stop kidding ourselves, 'fess up, and end the conspiracy. In this short talk, I hope to explain why sentiment analysis doesn't function as a useful social media metric, now or EVER.
21. Turk THis!
RT @BangsandaBun: According to
some, I wasn't deserving/grateful/
cool enough to be there, but I
blogged about my Glastonbury
experience http://t.co/eEInrnu
Tuesday, 5 July 2011
22. Turk THis!
RT @BangsandaBun: According to
some, I wasn't deserving/grateful/
cool enough to be there, but I
blogged about my Glastonbury
experience http://t.co/eEInrnu
Robot says Negative
Tuesday, 5 July 2011
23. Turk THis!
I know the bands for Glastonbury
were great this year, but I still
don't think you could have paid me to
be part of that mental crowd! You?
Tuesday, 5 July 2011
24. Turk THis!
I know the bands for Glastonbury
were great this year, but I still
don't think you could have paid me to
be part of that mental crowd! You?
Robot says positive
Tuesday, 5 July 2011
25. SENTIMENT scoring using
Amazon Mechanical Turk
disagreed with 16/40 (40%)
robot judgments.
Only 13/40 (32.5%) of
robotic judgments were
passed by all three human
Judges.
Raw Data available at http://mczar.me/mkue1p
Tuesday, 5 July 2011
26. But Are Humans
Reliable?
??? Emotional State/Mood
Priming
Cultural bias
Cognitive Bias
Famously not.
Tuesday, 5 July 2011
27. It Gets Worse...
The Same Human Scorer,
when presented with the
same statement on
multiple occasions, may
score it differently each
time.
cf. A Guerilla Experiment in Sentiment Analysis http://mczar.me/lrp7Yi
Tuesday, 5 July 2011
37. Why might marketers
want to perform
Sentiment analysis?
To guide customer service
interactions
As a Performance indicator
To qualify social media buzz
To identify fans
Tuesday, 5 July 2011
38. Is sentiment
actionable?
Product Query
Is it a...
Product Complaint
???
Product endorsement
Product review
Product COmparison
...I need to know more
Tuesday, 5 July 2011
39. THE KPI Problem
!"#"$%&'#
'##$
&"#$
&##$
%"#$ =)*;>$?)@2<$
A2.-4.@)3>$?)@2<$
%##$ B;3C>,;<<$?)@2<$
"#$
#$
()*+,-+.$ /+0+,-+.$ 12342.5$ 6+-.42.5$ 72.08$ 9:.;<$ 725$
!"#$
Massive (and unrepresentative) spike in Net Positive
Sentiment caused by specific campaign activity
Should we be allowed to mark our own homework?
Tuesday, 5 July 2011