Aspect and Sentiment Unification Model

Aspect and Sentiment Uniﬁcation Model
ACM Web Search and Data Mining 2011

Yohan Jo & Alice Oh
alice.oh@kaist.edu
Users & Information Lab
KAIST

December 2010

1

Wednesday, December 1, 2010

Our Research

• KAIST: major research and undergrad/graduate education in Korea

• KAIST CS has 49 full-time tenure-track faculty

• Research at Users & Information Lab

• Topic modeling: LDA, HDP and their variants

• Sentiment analysis of reviews, Twitter, and other user-generated contents

• We welcome collaborations and discussions: email alice.oh@kaist.edu


Problem: Unstructured reviews

4


These aspects and aspect-speciﬁc sentiments are available on
some Web sites for some of the products.

5


Can we automatically ﬁnd and analyze the relevant attributes and
the aspect-speciﬁc sentiments?

6


Overview of Talk

• Introduction to Topic Models

• LDA: Latent Dirichlet Allocation

• Aspect and sentiment in review data

• ASUM: Aspect and Sentiment Uniﬁcation Model

• Experiments and results

• Review data

• Twitter data
8


Topic Models

Slides from David Blei (Princeton University)
http://www.cs.princeton.edu/~blei/blei-meetup.pdf

A great tutorial by David Blei on videolectures.net
http://videolectures.net/mlss09uk_blei_tm/


http://www.cs.princeton.edu/~blei/blei-meetup.pdf


Latent Dirichlet Allocation
Blei, Ng, and Jordan, JMLR 2003

1. Basic Assumption
2. Generative Process
3. Inference
4. Graphical Representation


http://www.nytimes.com/2010/08/09/sports/autoracing/09nascar.html?hp

nascar, races, track, raceway, race, cars, fuel, auto, racing
economic, slowdown, sales, recession, costs, spending, save
fans, spectators, sports, leagues, teams, competition
13


Topics: multinomial over words

Topic Distributions Topics: multinomial over words

http://www.nytimes.com/2010/08/09/sports/autoracing/09nascar.html?

Topic Distributions Topics: multinomial over words

Graphical Representation of LDA

Topic Distributions

Topics

sales xxx slowdown
nascar, races, track, raceway, race, cars, fuel, auto, racing recession cars races
spending xxx save
costs fuel
15


Input to LDA

16


Input to LDA

http://www.nytimes.com/2010/08/09/sports/autoracing/09nascar.html?

16


Topics Discovered by LDA

nascar 0.12 spending 0.09 sports 0.12
races 0.10 economic 0.07 team 0.11
cars 0.10 recession 0.06 game 0.10
racing 0.09 save 0.05 player 0.10
track 0.08 money 0.05 athlete 0.09
speed 0.06 cut 0.04 win 0.07
... ... ...
money 0.002 speed 0.003 nascar 0.001

Topics: multinomial over vocabulary
17


Topic Distributions of Documents in the Corpus

http://www.nytimes.com/2010/08/09/sports/

Topic distributions for each document in the corpus

Topic 18


Graphical View

19


Graphical View

Observed
sales xxx slowdown
recession cars races
spending xxx save
costs fuel
19


Graphical View

Discovered

Topic Distributions

Topics
Observed
Discovered sales xxx slowdown
nascar, races, track, raceway, race, cars, fuel, auto, racing recession cars races
spending xxx save
costs fuel
19


ASUM: Aspect Sentiment Uniﬁcation Model

to uncover the intertwined semantic structure of aspects and sentiments in reviews

Yohan Jo and Alice Oh
WSDM 2011

20


Problem

21


Aspect

• This thing is small, and it's light, too.

• Start up and turn off time is fast.

• The low light performance is best in class, period

• The one thing I don't get is the 640X480 movie mode.

22


Sentiment





23


Sentiment Words

• affective words: love, satisﬁed, disappointed

• general evaluative words: best, excellent, bad

• aspect-speciﬁc evaluative words: small, cold, long

24


Sentiment Words

• affective words: love, satisﬁed, disappointed

• general evaluative words: best, excellent, bad

• aspect-speciﬁc evaluative words: small, cold, long

This camera is small. The LCD is small.
Beer was cold. Pizza was cold.
The wine list is long The wait is long.

24


SLDA: Sentence LDA
ASUM: Aspect Sentiment Uniﬁcation Model

automatically discover aspects and the corresponding sentiments in reviews

24,184 amazon reviews

7 product categories

27,458 yelp reviews

4 cities

320 restaurants

12 sentences per review (ave)

25


Observation





26


Observation





One sentence describes one aspect

26


Observation





One sentence describes one aspect
LDA assumption: each word represents one aspect

26


α

θ

β z β

φ w φ

T N
M
T
D S

(a) SLDA

Figure 2: Graphical represent
LDA vs SLDA ASUM. A node represents a r
edge represents dependency, an
replication. A shaded node is ob
shaded node is not observable.

27
and a sentiment. For ASUM, in cont

e l e c t r o n i c s restaurants
camera iso window keyboard laptop park beer
hand card vista pad ram street wine
feel raw softwar button processor valet drink
grip imag mac kei graphic cash glass
weight camera instal mous netbook lot select
size shoot os touch drive meter bottl
fit nois xp trackpad core across martini
solid file run finger game car tap
small print program touchpad batteri find mojito
bodi pictur driver scroll hp free margarita

α: 0.1
β: 0.001

Aspects found by SLDA product-speciﬁc details of reviews

28


Aspect-Sentiment Uniﬁcation Model
α α γ
Table 1: M
θ θ π els
D
M
β z β z s N
T
w
φ w φ S
V
T N
M
T N
M
D S D w
(a) SLDA (b) ASUM z
s
φ
topic
Figure 2: Graphical representation of SLDA and
ASUM. A node represents(LDA)a random variable, an θ
aspect (SLDA)
edge represents dependency, and a plate represents π
replication. A shaded node is observable and an un- α(k)
{sentiment, aspect}
shaded node is not observable. (ASUM) β(w) , βj
γ(j)
29

and a sentiment. For ASUM, in contrast, a pair of topic and
zi

results, which
Table 3: Full list of sentiment seed words in tion. We also tr
PARADIGM and PARADIGM+. For each word set,
Sentiment Seed Words the first line is the positive words, and the second
they do not re
use symmetric
line is the negative words. The words’ order does Some examples
not mean anything. Table 4.
good, nice, excellent, positive, fortunate, cor- From Elect
Paradigm
rect, superior
specific to the
bad, nasty, poor, negative, unfortunate,
wrong, inferior aspects such as
ered seven aspe
good, nice, excellent, positive, fortunate, cor- als, battery life
Paradigm+
rect, superior, amazing, attractive, awesome,
best, comfortable, enjoy, fantastic, favorite, Table 4(a). Ea
fun, glad, great, happy, impressive, love, per- laptop. The asp
fect, recommend, satisfied, thank, worth and features of
bad, nasty, poor, negative, unfortunate, cuss in laptop r
wrong, inferior, annoying, complain, disap- the 50 aspects f
pointed, hate, junk, mess, not good, not like, product categor
not recommend, not worth, problem, regret, aspects that SL
sorry, terrible, trouble, unacceptable, upset,
waste, worst, worthless applications su
and retrieval.
We compared
built into the model by setting asymmetric priors by LDA, and Ta
the negative and Gibbs from the sentence. Previous work
sentiment sampling initialization
has proposed several approaches for this problem including found by SLDA
flipping the sentiment of a word when the word is located as “grip” and “l
closely behind “not” [7]. We use simple rules to express the about a camera
negation by prefixing “not” to a word that is modified by aspects, but ra
negating words, as is done in [6]. “brands” and “c
30
assumption bui

Sentiment Seed Words in the Model

β is different for positive φ and negative φ
α γ α: 0.1

Table 1: Meaning
θ π
β: els0 for negative
sentiment seed
words Dpositive
in th
senti-aspects
M th
β z s 0 for positive
N th
sentiment seed
T
words in negative th
w senti-aspects
φ S th
0.001 for all other words
V th
T N
M
D S D w w
(b) ASUM z 31
as
Wednesday, December 1, 2010 s se

positive senti-aspects negative senti-aspects
worth screen easi monei fingerprint
monei color light save glossi
penni bright carri notwast magnet
extra clear weight wast screen
well video lightweight yourself show
everi displai suction notbui finger
price crisp small awai finish
dollar great around spend print
spend resolut vacuum notworth smudg
pai qualiti power stai easili

Senti-Aspects discovered
contain both aspect words and
by ASUM sentiment words
32


positive senti-aspects negative senti-aspects
flavor music dry loud cash
tender night bland tabl onli
crispi group too convers card
sauc crowd salti hear credit
meat loud tast music downsid
juici bar flavor nois park
soft atmospher meat talk take
perfectli peopl chicken sit accept
veri dinner bit close bring
moist fun littl other wait

Senti-Aspects discovered
contain both aspect words and
by ASUM sentiment words
33


aspect.
Common Words Sentiment Words
screen color clear great pictur sound movi beauti good
bright displai hd imag size watch rai nice crystal
crisp qualiti glossi glare light reflect matt edg macbook
sharp kei black bit peopl notlik minor
music song radio listen fm movi record easi convert
player video podcast album audio book librari watch
download itun problem updat driver vista system xp
zune file firmwar disk mac hard run microsoft appl
our us server water glass refil wine attent friendli
waiter tabl she brought sat veri arriv plate help staff nice
he waitress ask said me want card get tell if would gui bad
minut seat could rude pai becaus walk then w
r

“crust”. To express negative sentiment, they use words such 6
as “dry”, “bland”, and “disappointed”. These two aspects
aspect-specific sentiment
were discovered in ASUM but not in SLDA, and the reason
without using sentiment c
discovered these aspects
words labels
is that people express their sentiment toward
d
very clearly. In SLDA the words that convey a sentiment 34
v
toward the quality of meat appear in various cuisine-type

I was so excited about this product.
I’d tasted the coffee and it was pretty good and easy and quick to make.
However, this machine makes the most awful, LOUD sound while heating
water.
It’s disturbing to hear in the morning, while others are sleeping especially!
Keurig’s customer service is terrible too!

The restaurant is really pretty inside and everyone who works there looks
like they like it.
The food is really great.
I would recommend any of their seafood dishes.
Come during happy hour for some great deals.
The reason they aren’t getting five stars is because of their parking
situation.
They technically don’t “make” you use the valet but there’s only a half
dozen spots available to the immediate left.

senti-aspects assigned to
sentiments shown in greeen (p),
sentences pink (n)
35


Parking (A46, Negative)
park, street, valet, lot, there, free, can, ﬁnd, onli, if, valid, car, get, meter, your, block,
hour, spot

• Parking is only validated for 3 hours.
• This place is a lol hard to see coming from 10th street and parking is
limited.

• They don’t have a lot/any designated parking/complimentary valet.
• Apparently since it’s Friday the valets charge $5 to park, which I found
really annoying and just found a spot on the street.

same aspect from different reviews
sentences
36


Coffeemaker Easy (A10, Positive)
coffee, hot, maker, brew, cup, great, caraf, pot, good, fast, keep, hour, love, like,
machin, warm, time, thermal, easi

• Makes coffee fast and hot
• It took us several uses to understand how much coffee to use
• And easy to use programmer for morning coffee
• Very convenient
• Guests always comment on how nice it looks and how easy it is to use

same aspect from different reviews
sentences
37


0.85! 0.85! 0.9! 0.9!
0.8! 0.8! 0.85! 0.85!
0.75! 0.75! 0.8! 0.8!

0.7! 0.75! 0.75!
0.7!
Accuracy!

Accuracy!
Accuracy!

Accuracy!
0.7! 0.7!
0.65! 0.65!
0.65! 0.65!
0.6! 0.6!
0.6! 0.6!
0.55! 0.55!
0.55! 0.55!
0.5! 0.5! 0.5! 0.5!
0.45! 0.45! 0.45! 0.45!
0.4! 0.4! 0.4! 0.4!
30! 30! 50! 50! 70! 70! 100! 100! 30! 30! 50! 50! 70! 70! 100! 100!
Number ofNumber of Topics!
Topics! Number ofNumber of Topics!
Topics!
ASUM! ASUM+! ASUM+!
ASUM! JST+! TSM+!
JST+! TSM+! ASUM! ASUM+! ASUM+!
ASUM! JST+! TSM+!
JST+! TSM+!

(a) Electronics
(a) Electronics (b) Restaurants
(b) Restaurants

3: Sentiment classification results. Three unified models (ASUM, JST, JST, TSM) are compared
gure 3: Sentiment classification results. Three unified models (ASUM, TSM) are compared in th
with with seed Sentiment Topic Model, andand He, CIKM09
ures two Joint seed word Paradigm Lin Paradigm+ (“+” (“+” indicates Paradigm+). error err
JST: two word sets sets Paradigm and Paradigm+ indicates Paradigm+). The The ba
nt the standard deviation after after multiple trials.
present the standard deviation multiple trials.
TSM: Topic Sentiment Mixture, Mei et al., WWW07

ndition of unigrams. The baseline with only seed seed
me condition of unigrams. The baseline with only
rforms quite well, but ASUM performs even better.
rds performs quite well, but ASUM performs even better.
al, the accuracy increases as the as the number of aspects
general, the accuracy increases number of aspects
Sentiment Classification
because the models better fit the data. However,
creases because the models better fit the data. However,
ease slows slows down for ASUM, as the additional in-
e increase down for ASUM, as the additional in- among generative models, ASUM
the number of aspects becomes no longer effective. Comparison
ease of the number of aspects becomes no longer effective.
T had performance on movie movie reviews in the original
great great performance on reviews in the original performs best
38
ut did not perform well on our data. data. is not is not 1
per, but did not perform well on our TSM TSM I was so was so excitedthis product. {A24, p}
1 I excited about about this product. {A24, p}
ended for sentiment classification,sentiment words words
for sentiment classification, and and sentiment

. (3) Once the author decides which topic the word
out, the author will further decide whether the word
d to describemodel p(w)neutrally,LDA
Language the topic like φ in positively, or nega-
(Document-independent)
. (4) Let the topic picked in step (2) be the j-th topic
The author would ﬁnally sample a word using θj , θP
Like topic in LDA
, according to the decisionzin step(3). This generation
TSM
ss is illustrated (Document-Specific)
in Figure 2.
A theme itself is not a language model

!! Generation Process of word w
"!./0./1
1. Decide whether w is generated from B or themes.
!"#$%&'

!"
""./0./1 1
!! If B, then choose w according to p(w|B).
#0!
# !/3 $, else
'()*)+

2
!"
!$
"$./0./1 #0" - 2. Choose a theme j from which w is generated.
# 3. Decide whether w is generated from θj, θP, or θN.
"2./0./%
k
!$
#0$ 4. Choose w from the selected θ.
()*+$+,"

!%
θP and θN are theme-independent (i.e., shared by all
$,

0 themes)
!"-&$+,"

!&
"2./0./& ,
• They should cover as many sentiment words as
possible to be applied to all themes
• This is problematic because it requires special effort
(unlike general sentiment words)
• This
reLanguage model p(w|B) process of the topic- model can’t find theme-specific sentiment words
2: The generation
ment mixture model (e.g., function words)
B: Background words

now formally present the Topic-Sentiment Mixture
l and the estimation of parameters based on blog data.

JST

(a) (b)
Generation Process of word w
1. Choose a sentiment l.
2. Choose a topic label z based on l.
Figure 1: (a) LDA model; (b) JST model; (c) Ty
3. Choose w from φzl.
Same β for all φ’s
• There 1(a), is one in the positive φ and negative models based
in Figure is no differenceof β for most popular topicφ θ for each indivi
• Effect assumption that documents away
upon the of Gibbs sampling initialization fadesare mixture of topics, in JST is assoc
where a no sentencea probability aspect discovery over words [2,
There is topic is layer required for distribution topic-document
18]. The LDA model is eﬀectively a generative model from a sentiment lab
which December 1, 2010document can be generated in a predeﬁned
Wednesday,
a new feature essential

associated with each aspect, because in many cases at least one word in the n-gram is assigned to th

will not be the ones where this aspect is discussed.
MAS
the most predictive fragments for each aspect rating associated aspect topic (r = loc, z = a).
Instead of having a latent variable yov ,6 we use
Our proposal is to estimate the distribution of pos- similar model which does not have an explicit no
sible values of an aspect rating on the basis of the tion of yov . The distribution of a sentiment rating y
overall sentiment rating and to use the words as- for each rated aspect a is computed from two score
signed to the corresponding topic to compute cor- The first score is computed on the basis of all the n
rections for this aspect. An aspect rating is typically grams, but using a common set of weights indepen
correlated to the overall sentiment rating5 and the dent of the aspect a. Another score is computed onl
fragments discussing this particular aspect will help using n-grams associated with the related topic, bu
to correct the overall sentiment in the appropriate di- an aspect-specific set of weights is used in this com
rection. For example, if a review of a hotel is gen- putation. More formally, we consider the log-linea
distribution:
erally positive, but it includes a sentence “the neigh- ya = {p(1-star), p(2-stars), ..., p(5-stars)}

borhood is somewhat seedy” then this sentence is P (y = y|w, r, z) ∝ exp(ba+ J +pa J a ),
Assumptions (a)
predictive of rating for an aspect location being be-(b)
a y f,y f,r,z f,y
f ∈w
•lowsentence is covered by several sliding windows a = explicitly MG-aspectof all the words in a docu
A other ratings. This rectifies the aforementioned
Figure 3: (a) MG-LDA model. (b) An extension ofz rated where w, r, are vectors
(ψds is a window distribution of sentence s in
LDAIn theobtain MAS.experiments all three aspect rat- f = 6n-gram feature
5 to
document d) used in our
dataset Preliminary experiments suggested that this is also a feas
ings are equivalent for 5,250 reviews out of 10,000.
Jf,y = common somewhatfor f computationally expensive
ble approach, but
weight more
Jaf,y = aspect-specific weight for f
Generation Process of word w in sentence s
paf,r,z = fraction of words in f assigned r = loc
• The formal vdefinition of the model =with K gl
Choose a window from ψds 311
Decide whether w is chosen from global topics or z a
•
globaltopics (πK loc local topics is as follows: aspect-specific sentiment words
local and v = {p(gl), p(loc)}) There is no First,
• If r =K gl word distributions for global topics ϕgl user-rated training data
gl, choose topic z from ϑgl
draw if r = loc, choose topic z from ϑloc This model requires
else z
from a Dirichlet prior Dir(β gl ) and K loc word dis-
• Choose w from z
tributions for local topics ϕloc - from Dir(β loc ).

ASUM
Generation Process
For each sentence β is different for positive φ and negative φ
• Choose a sentiment s
• Choose a topic label z
• Choose words from φzs α α γ

Current Limitations
• θ would be different for different θ θ π
sentiments as in JST
• If a sentence is too short (1 or 2
words), the topic assigned is almost z
β β z s
random (because there is no clue)
• It does not model well the sentences w
that have multiple aspects
φ w φ

T N
M
T N
M
D S D

(a) SLDA (b) ASUM

Results on Twitter Data

• 1.3 million tweets

• 50k words in vocabulary

• What would happen when we apply this model to Twitter

• Many more and wide variety of aspects (topics)

• Different notions of “sentiment”

• Review data: polarity (like vs. dislike)

• Twitter: feelings (happy vs. unhappy)


Seed Words

:) :(
:-) :-(
:] :[
:^) :-[
:D :'(
:-D :/
=) :-/
=] =(
=D =[


Positive Senti-Aspects

vote cream morn happi idol dinner
milei ic good dai adam home
jona chocol everyon birthdai american :)
cyru eat :) mother kri had
demi cake night father lambert hous
award cooki dai hope allen famili
too butter hello all vote fun
lovato mm hope thank watch night
teen yum world mom danc then
taylor peanut twitter bless talent friend
selena yummi afternoon dad paula lunch
song chip all :) win hang
choic strawberri happi easter susan birthdai
who breakfast great love boyl parti
brother coffe how great he tonight


Positive Senti-Aspects

love god obama ##dollar## app ever
:) bless #tcot dress window movi
thank prai health sale iphon seen
smile lord palin new googl funni
ya prayer presid bought instal watch
your jesu vote wear mac wa
keep our mccain design firefox best
up thank he shop os ve
ll your care shoe tweetdeck thing
all ##time## senat art chrome funniest
we we republican bag beta hilari
good christ reform vintag download laugh
make he bill black us video
lol prais tax gift version love
alwai love elect paint desktop saw


Negative Senti-Aspects

hurt ##percent## tire monei jackson twitter
feel ##dollar## blah ##dollar## michael facebook
pain market :( make mj try
:( stock sleep guarante rip why
sore price feel onlin farrah work
throat trade bore earn di upload
headach rate sick month sad how
sick sale bed free fawcett can
stomach forex im twitter dead figur
doctor profit sleepi 24 death updat
flu bank work home tribut tweetdeck
ey report soo incom billi comput
cough rise realli hour he link
teeth oil ugh your memori anyon
but billion want start micheal pictur


Negative Senti-Aspects
flight quiz game obama rain #iranelect
airport class plai health snow iran
plane exam tonight senat weather avatar
drive test watch state storm support
home homework laker presid sun add
hour school footbal new wind iranian
back tomorrow win bill cold democraci
wait done night court outsid protest
traffic finish readi tax dai 1-click
train paper tiger #tcot thunder #iran
bu math let vote degre green
trip :( wait law sunni #gr88
from work season fund cloud tehran
car final some blog but overlai
delai start wing govern here #twibbon


ASUM: uncovering the hidden semantic structure of aspects and sentiments

Alice Oh alice.oh@kaist.edu
Yohan Jo yohan.jo@kaist.ac.kr
http://uilab.kaist.ac.kr


Aspect and Sentiment Unification Model

Recommandé

Recommandé

Contenu connexe

Dernier

Dernier (20)

En vedette

En vedette (20)

Aspect and Sentiment Unification Model