1. When Relevance is not Enough:
Promoting Diversity and Freshness in
Personalized Question
Recommendation
IDAN SZPEKTOR,YOELLE MAAREK,DAN PELLEG
YAHOO!RESEARCH
2. ABSTRACT
a good question recommendation system
1.
designed around answerers, rather than exclusively for askers
2.
Scale to many questions and users and be fast enough
3.
Relevant to his or her interests
4.
diversity
4. INTRODUCTION
relevance: to what degree the question matches the user’s tastes
diversity and freshness needs
Three requirements:
1. questions need to be recommended for all types of users
2. questions have to be diverse
3. recommendations need to be fresh and be served fast
a) serve questions as recommendations immediately
b) instantly adapting to users’ changes in taste
7. Framework
Question profile:
1. LDA model
2. Lexical model
3. Category model
User profile:
Question recommendation
Matching question and user profiles
Proactive diversification
Recommendation merging
8. QUESTION PROFILE
Split it according to the 26 top categories in Yahoo! Answers
Two Advantage:
1.
2.
represent disjoint users’ interests.
word sense disambiguation
1.
question textual content(title and body)
2.
category
9. QUESTION PROFILE
Build profile, which is represented by three vectors:
1.
a Latent Dirichlet Allocation (LDA) topic vector
2.
a lexical vector
3.
a category vector
10. LDA Model
1. Initial training: a random sample
of up to 2 million resolved
questions
2. Incremental learning: a random
sample of up to half a million
questions per top category
3. Inference: at least10% of the
probability mass
11. Lexical Model
a unigram bag-of-words representation of a question
tf·idf score / L1 normalized
a probability distribution
Category Model
a probability of 1 to the category in which the question was posted
12. USER PROFILE
the questions answered in the past
the user representation is generated by aggregating signals over these
questions
user profile: a probability tree
13. 1. Aggregating the profiles of the questions the user answered
2. Update
14. the first and third tree levels:
a decaying factor on past questions
the second level:
1. Measure the similarity between the feature distribution of each model in the
question and the corresponding feature distribution in the user profile
2. Normalized to a probability distribution
15. QUESTION RECOMMENDATION
Matching Question and User Profiles
A list of open questions ranked by a relevance score, which is calculated for the pair {question
profile , user profile}
For question profiles:
1.
Turn the three vectors forming the question profile into a single vector, multiply the
probability of each feature by 1/3 before storing it in the index
2.
Index every question vector and build an inverted index
16. QUESTION RECOMMENDATION
For user profile:
associate with each user feature a score that consists of the product of each probability score
on the tree path that led to this feature
Ranking:
Similarity: a simple dot-product
17. QUESTION RECOMMENDATION
Proactive Diversification
thematic sampling:
1.
For each user vector u , we generate N query vectors u 1 ;u 2 ;…;u N
2.
N ranked lists
3.
Blending them together results in a final diverse list
Two types of thematic constraints:
specific top category: randomly select top categories as constraints by sampling without repetition
based on their distribution in the root node of the user’s probability tree
spefic LDA topic: randomly sample LDA topics without repetition from the user profile by traversing
the probability tree
18. QUESTION RECOMMENDATION
Recommendation Merging
blending algorithm
1.
Each list being associated with a probability score
2.
Sampling an intermediate list, based on the assigned probabilities
3.
Removing one recommendation from the sampled list to be added at the end of the final
list.
4.
Repeat