This document discusses using clickstream data to improve search ranking and facet optimization in SOLR. It describes compiling relevance feedback from click data, indexing click signals to SOLR to boost document ranking, and reordering facets based on engagement data learned from clickstream. The approach led to improvements in key metrics like CTR, conversion rates, and sales. Ongoing work includes learning to rank models and discovering relevant facets for queries to better guide users' product discovery.

1. Integrating Clickstream Data in SOLR
for Ranking and Dynamic Facet Optimization
Ilayaraja Prabakaran
Lead Engineer, Search
Target

2. Agenda
• Search Ranking using clickstream
Compiling relevance feedback
Indexing click signals to Solr
Re-ranking using clicks signals
• Facet Optimization using clickstream
Compiling facet engagement data
Re-ordering of facets based on engagement
Discovery of relevant facets per query

3. 01
Learning From Implicit Feedback
Query Product list
Q1 Px, Py, Pz….
Q2 Pi, Pj, Pk….
Qn …....
Events
UserId SessionId Time stamp Action Meta-Data
xyz ab2n2n.. 20161008-233554 Search Search term
xyz ab2n2n.. 20161008-233601 Click Query, ProdId,
pos
xyz ab2n2n.. 20161008-233801 CartAdd Query, ProdId,
pos
MAP REDUCE JOBS
(Aggregation over
sessions and actions)
Item List of user queries and importance
X <Q1,score>, <Q2,score>,….
Y <Qi,score>,<Qj,score>,….
SOLR
Search Service
Doc: Title
Brand
Click_term1,Click_val1
Click_term2,Click_val2
….
Boosting for
matching
click_terms …

4. 01
Rank Score
The final weighted score is calculated as below:
Weighted Score = ( W1 * ClickRate ) + ( W2 * CartRate )
Currently weights are :
W1 = 0.25
W2 = 0.75
• Query to Item Clicks (ClickRate)
• Query to Item added to cart (CartRate)

5. 5
public static double slotweight(int page,int slotinpage) {
try {
int expo = (page - 1) * (24) + slotinpage;
Double score = 1 / (1 + Math.exp(-0.2 * expo));
return score;
} catch (Exception e) {
e.printStackTrace();
}
return 0;
}
0
0.2
0.4
0.6
0.8
1
1.2
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
Slot
Weight
Positional Bias

6. 6
Days dayDiff = Days.daysBetween(eventDate,DateTime.now());
double decay=0;
if(dayDiff.getDays() <= 365){
decay = 1/((dayDiff.getDays()/60.0) + 1);
}
Time Decay
0
0.2
0.4
0.6
0.8
1
1.2
1
7
13
19
25
31
37
43
49
55
61
67
73
79
85
91
97
103
109
115
121
127
133
139
145
151
157
163
169
175
181
187
193
199
205
211
217
223
229
235
241
247
253
259
265
271
277
283
289
295
301
307
313
319
325
331
337
343
349
355
361
Time
Decay

7. 7
Data File
ITEM_ID SEARCH_QUERY SCORE
16981621 bandit belly 14.82
16981621 band belly 13.87
16981621 clothing maternity 13.60
16981621 maternity 13.45
16981621 belt postpartum 12.58
15179653 patrol paw 5.83
15179653 patrol paw toys 5.36
15179653 patrol paw skye 3.78
15179653 patrol paw sky 2.77
15179653 patrol paw skye toys 2.58

8. 8
Integration with Solr – Indexing Time
• schema.xml: Include below dynamic fields
<dynamicField name="click_term*" type="string" indexed="true" multiValued="false" />
<dynamicField name="click_val*" type="double" indexed="true" multiValued="false" />
Index the search terms (click_term1, click_term2..) as a new field in the solr index
along with corrosponding click scores (click_val1, click_val2...).

9. 9
Integration with Solr – Search Time
Boost the match on click_term(s):
Including the boost query for top N terms to the original solr query, for
example:
bq=(click_term1:"nespresso") AND _val_:"max(1,product(click_val1,1000))” &
bq=(click_term2:"nespresso”) AND _val_:"max(1,product(click_val2,1000))"& ….
Reference: Implementing Click-through Relevance , Andrzej Białecki

10. 10
Impact
Before
A(er

11. 11
Impact
Before
A(er

12. 12
Metrics & Measurement
Offline:
NDCG/Human judgments
Click-Rank
Cart-Rank
Online: A/B test
Primary metrics: CTR, Conversion
Secondary metric: Demand Sales
Overall Impact: 350K queries
10% up in CTR-Top10
6% up in CTR-Top5
2% up in $search demand

13. 13
WIP - Learning To Rank
TargetFeatureStore:
OriginalScoreFeature – originalScore
SolrFeature
productTitleMatchQuery
productBrandMatchQuery
productItemTypeMatchQuery
productCategoryNamesMatchQuery
productRecencyFeature
…..
Learned Model: (prototype)
weights: {
"originalScore":0.0,
"productTitleMatchQuery":-0.022,
"productBrandMatchQuery":0.0241,
"productItemTypeMatchQuery":0.022,
"productCategoryNamesMatchQuery":0.0182,
"productKeywordsMatchQuery":0.030,
"productMetaKeywordsMatchQuery":0.017
}

14. 14
Facet Optimization
Ø Problem: Rule based faceting treatment is static and limited.
Ø Solution: Build learning model for driving facets dynamically.
• Solution 1: Facet Ranking
• Solution 2: Facet Discovery
Ø How will this affect our guest?
• Bringing relevant refinements for a given search query or category (browse)
• Help guests to narrow down their product discovery through “smart” facets

15. 15
Ranking Implementation
Ø Computing query to facet association:
• Will go through each session record and calculate number of clicks for each facet.
• Calculate search impression by getting distinct search terms for a facet type and
summing them up.
• Facet Engagement Rate = Facet clicks/ facet impression.
Ø Computing query to product click association
• For each session, find attribution of product clicks to each facet.
• Total sum of clicks calculated for <query, facet> combination.
• Facet Click Rate = Normalized by dividing with search impression.
Ø Computing query to product cart add association
• For each session, find attribution of product cart adds to each facet.
• Total sum of cart adds calculated for <query, facet> combination.
• Facet Cart Rate = Normalized by dividing with search impression.

16. 16
Facet Ranking
The final weighted score is calculated as below:
Score = Sum { W1 * Facet Engagement Rate,
W2 * Facet Click Rate,
W3 * Facet Cart Rate }

17. 17
Facet Discovery
• Query to Category (breadcrumb) mapping
• Compute weighted score based click/cart signals for browse with breadcrumb
as the key.
• Select the optimal breadcrumb(s) by ranking based on above score for the
breadcrumbs at a search term level.
• Search and browse facets are aggregated.

18. 18
Facet Discovery & Ranking

19. 19
Before
Discovery
0
10000
20000
30000
40000
50000
60000
70000
80000
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
Impact
on
Search
No.
of
“Unique
Guest
Queries”
No. of “newly discovery filters”
Discovery of “Search Filters”
A(er Discovery

20. Thank You