1. The document discusses how to efficiently test hypotheses and eliminate ideas that do not improve recommendations through both offline and online testing methods.
2. It describes Retail Rocket's use of Scala and Spark notebooks to significantly shorten offline testing from days to hours by building an offline evaluation framework.
3. Key recommendations are to use offline testing for minor changes if an accurate offline metric is established, but to always do online experiments for major changes or new algorithms to get an accurate assessment of impact.
11. Offline framework
• Scala on Spark
• Deals with existing web logs
• Implicit feedback
• Major metrics:
o Recall, Diversity, Recall with NN, Empty Recs
• Minor metrics:
o Serendipity, Novelty, Coverage
• Different types of events sequences
• Different definitions of users’ sessions
• Personalised / Non-personalised recommendations
• Adjustable TOP of viewable recommendations
• Test panel of sites from different domains
16. Recall with Nearest Neighbours (NN)
Top 4 recs
0.8 0.7 0.5 0.5
0.8 0.7 0.5 0.5
0.6 0.5 0.4
0.9 0.8 0.3 0.5
Content based similarity
(Nearest neighbours)
Real item
0.5
Indirect hit
1.0
Direct hit
No hit
0.0
Metric = Average over all sessions