17. Recap: Inverted Index
Terms doc_ids ttf
fifa 1 1
call 2 1
of 2 1
duty 2 1
god 3 1
war 3 1
pes 4 1
doodle 5 1
Doc Id Title
1 Fifa
2 Call of Duty
3 God of War
4 PES
5 Doodle God
2,3 2
3,5 2
18. {
"title": “Call of Duty®: Black Ops 4",
"image": "rs-137178-883f5fe955b2745cd539.jpg",
"description": "<p>Digital Standard Edition includes: - 1,100
Call of Duty® Points* - Digital Edition Bonus Items: --
Specialist Outfit for all Specialists -- Gesture -- Calling Card,
Emblem, Sticker and Tag inspired by the iconic Call of Duty®:
Black Ops 4 skull. Black Ops is back! Featuring gritty,
grounded, fluid Multiplayer combat, the biggest Zombies
offering ever with three full undead adventures at launch, and
Blackout, where the universe of Black Ops comes to life in
one massive battle royale experience.</p>",
"rating": 4.5,
"numberOfRatings": 279,
"vendor": "Activision",
"price": 59.99,
"releaseDate": “2018-12-10“,
"id": 350640
}
25. Learning To Rank
Learning to rank or machine-learned ranking is the application of machine
learning, typically supervised, semi-supervised or reinforcement learning, in the
construction of ranking models for information retrieval systems.
~ Wikipedia
http://bit.ly/ltr-wp
28. Model Evaluation Types (Errors)
• Binary relevance (MAP, Precision)
• Graded relevance, position based (DCG, NDCG)
• Only discounts based on relevance
• Graded relevance, cascade based (ERR)
• Discounts based on user interaction with the results
http://bit.ly/eval-metric
29. MAP using Binary relevance
23
9
88
33
45
YI Average Precision
1
0.67
0.6
fifa
X
relevant
not relevant
= (1 + 0.67 + 0.6) / 3
= 0.76
MAP
30. DCG using Graded relevance
23 rel=3
9 rel=2
88 rel=4
33 rel=1
45 rel=0
YI
Discounted
Cumulative
Gain
3
5 + 4/log2(3) = 7.57
8.07
fifa
X
Documents ranked on 0 - 4 relevance scale
3 + 2/log2(2) = 5
7.57 + 1/log2(4) = 8.07
32. Learning to rank - Model
X
Model
Y
Y’
Algorithm
Parameters
Cost function
33. LTR: Steps to take
1. Create Judgement List (Ground Truth)
2. Define features for the model
3. Log features during usage
4. Training and testing the model
5. Deploying and using the model
6. Feedback loop
40. Judgement List: Implicit Feedback
• Log user behaviour
• Compare actual clicks versus expected clicks
• A click is not a relevance judgement
41. Judgement List: Implicit Feedback
• Use as a signal to the ranking algorithm -> Feature
• Use as Label to train the model -> Ground truth
42. Using the LTR Plugin and the
python scripts
https://github.com/o19s/elasticsearch-learning-to-rank
43. Our Judgement List
# grade (0-4) queryid docId title
#
# Add your keyword strings below, the feature script will
# Use them to populate your query templates
#
# qid:1: fifa
# qid:2: football
# qid:3: call of duty
# qid:4: marvel
# qid:5: basketball
# qid:6: god
#
4 qid:1 # 1538781503000 FIFA 18
2 qid:1 # 1536187840000 EA SPORTS FIFA 16
3 qid:1 # 1538107776000 FIFA 19 Ultimate Edition
4 qid:1 # 1538694141000 FIFA 19
3 qid:1 # 1536293937000 EA SPORTS FIFA 17 Standard Edition
1 qid:1 # 1536022097000 FIFA 15
3 qid:2 # 1538509479000 PRO EVOLUTION SOCCER 2018
4 qid:2 # 1535257488000 Pro Evolution Soccer 2018 FC Barcelona Edition
1 qid:2 # 1536293937000 EA SPORTS FIFA 17 Standard Edition
1 qid:2 # 1538781636000 2MD: VR Football
44. 2. features for the model
• Raw Term Statistics
• Document Frequency
• Total Term Frequency
• Also max, min, sum (in case of multiple terms, fields)
• Elasticsearch queries
45. 2. features for the model
{
"query": {
"match": {
"title": "{{keywords}}"
}
}
}
{
"query": {
"match": {
"description”: "{{keywords}}"
}
}
}
51. 4. Train and test model
• Making use of Ranklib
• Can specify separate train, validation and test set
• Can normalise feature sets
52. Models using Ranklib
MART
Multiple Additive Regression Trees, a gradient boosting machine. Can be used for
regression as well as classification.
RankNet
Compare two feature vectors using stochastic gradient descent with the help of a cost
function.
RankBoost
Based on AdaBoost, combining many weak rankings into a single highly accurate ranking.
Is pairwise comparison
AdaRank
Combines a number of weak learners in a linear way. Builds on AdaBoost, but directed
more at ranking.
Coordinate Ascent
Optimises one parameter at a time, keeping the other constant. Done iteratively until
some convergence criteria is met.
LambdaRank
Optimisation of RankNet that only looks at the gradients represented by arrows
indicating how much they need to move up or down
LambdaMART Combines using the gradients of LambdaRank and the use of MART
ListNet
Uses a list wise loss function, a neural network and gradient descent. Similar to RankNet,
only difference is List versus Pair loss functions.
Random Forests
Number of trees to vote for the most popular class for a vector of features. One tree
would not be better than a random choice, but a forest is
Linear Regression
Most of the times to simplistic for the learning to rank problem with lots of features, but
good to have available to at least try
53. Evaluation metrics
MAP Mean Average Precision:The average of all P@k
DCG@k Discounted Cumulative Gain:Add relevance of all documents discounted
by the position of the document making the 1st document more important
NDCG@k Normalised DCG:A DCG with a value between 0 and 1, normalised by the
highest score.
P@k Percentage of relevant documents of this top K
RR@k Reciprocal Rank: 1/K where K is the first relevant document
ERR@k Expected Reciprocal Rank: discounts documents that are below a very
relevant document.
58. 6. Feedback Loop
• Register clicks by users (and other events)
• Use click data for predicting labels
59. Click models
• Random Click Model -> Every document has the same chance of being
clicked
• Click Through Rate Model -> Uses the fact that the first document is
clicked far more than the second document
• Cascade Model -> A click in the third item also tells us something about
the first two items. Only one click per session is assumed.
• Dynamic Bayesian Network Model -> Supports multiple clicks in a search
session and the difference in actual relevance of a document.
http://bit.ly/click-model
60. Dynamic Bayesian Network
Ei-1 Ei Ei+1
Ci
Ai Si
au su
http://bit.ly/dbn-clickmodel
Ei - Did the user examine the url
Ai - Was the user attracted by the url
Ci - Did the user click the url
Si - Was the user satisfied with the landing page
au - Probability of being attracted by the url
su - Probability of being satisfied by landing page