Real-time Recommender systems

1. Real-Time Recommender Systems
Bay Area Search Meetup at eBay
April 25, 2012
Balu Rajagopal

2. Goal of Recommenders INSTANT INTELLIGENCE
1. Increase number of items sold
2. Cross-Sell, Up-Sell diverse items
3. Increase Customer Satisfaction
4. Build Loyalty
5. Improve User Experience
Cetas Software Inc. – Copyright © 2012– CONFIDENTIAL – DO NOT DISTRIBUTE 2

3. Recommendations INSTANT INTELLIGENCE
USERS
Search Recommendations
Products
Web sites
Social networks
ITEMS
Blogs
News
….
Cetas Software Inc. – Copyright © 2012– CONFIDENTIAL – DO NOT DISTRIBUTE 3

4. Two Challenges INSTANT INTELLIGENCE
 Make a Personalized Recommendation
– Multi-Dimensional Data
– Streams: Social, Activity, Apps, Tweets, Actions, …
– Demographic
– Temporal, Spatial
 Do it in real-time
– Query to Analysis to Visualization
– User Experience (UX)
– System Constraints – Network, Capacity, SLA
Cetas Software Inc. – Copyright © 2012– CONFIDENTIAL – DO NOT DISTRIBUTE 4

5. Problem Space INSTANT INTELLIGENCE
Cetas Instant Intelligence Framework
Secs or Less
Large
RESPONSE TIME TO USER
DATA DIMENSIONS
Minutes
Medium
Hours
Small
Gigabytes Terabytes Petabytes
ANALYSIS VOLUME
Cetas Software Inc. – Copyright © 2012– CONFIDENTIAL – DO NOT DISTRIBUTE 5

6. Real-time Recommender System INSTANT INTELLIGENCE
Inputs Terabytes of Multi-Dimensional data
Preprocessing Reduction
@ Scale
@ Speed
Analysis Classifying, Clustering
Output Prediction, Recommendation
Cetas Software Inc. – Copyright © 2012– CONFIDENTIAL – DO NOT DISTRIBUTE 6

7. Real-time Recommender System INSTANT INTELLIGENCE
• Spatial
Inputs • Temporal
• Demographic
• Personal • Psychographic
• Behavioral
Preprocessing Reduction
Analysis Classifying, Clustering
Output Predictions, Recommendations, Patterns
Cetas Software Inc. – Copyright © 2012– CONFIDENTIAL – DO NOT DISTRIBUTE 7

8. Real-time Recommender System INSTANT INTELLIGENCE
• Spatial
Inputs • Temporal
• Demographic
• Personal • Psychographic
• Behavioral
• Distance Measures
Preprocessing • Sampling
• PCA
• Dimensionality Reduction
• SVD
Analysis Classifying, Clustering
Output Predictions, Recommendations, Patterns
Cetas Software Inc. – Copyright © 2012– CONFIDENTIAL – DO NOT DISTRIBUTE 8

9. Real-time Recommender System INSTANT INTELLIGENCE
• Spatial
Inputs • Temporal
• Demographic
• Personal • Psychographic
• Behavioral
• Distance Measures
Preprocessing • Sampling
• PCA
• Dimensionality Reduction
• SVD
• Predictors • Classification
Analysis
• Descriptors • Association
• Clustering
Output
Cetas Software Inc. – Copyright © 2012– CONFIDENTIAL – DO NOT DISTRIBUTE 9

10. Real-time Recommender System INSTANT INTELLIGENCE
• Spatial
Inputs • Temporal
• Demographic
• Personal • Psychographic
• Behavioral
• Distance Measures
Preprocessing • Sampling
• PCA
• Dimensionality Reduction
• SVD
• Predictors • Classification
Analysis
• Descriptors • Association
• Clustering
• Predictions
Output • Recommendations
• Patterns
Cetas Software Inc. – Copyright © 2012– CONFIDENTIAL – DO NOT DISTRIBUTE 10

11. Big Data Analytics – eCommerce
INSTANT INTELLIGENCE
Input data Clustering Closed-loop Action
User
transactions
live stream Product placement
decision
Demographics
data stream
Category, sub-
category sorting
Online app
events stream
New product
Ad placement offering
stream
Other streams
… Other actions …
Cetas Software Inc. – Copyright © 2012– CONFIDENTIAL – DO NOT DISTRIBUTE 11

12. Real-time Stream Processing INSTANT INTELLIGENCE
Billions of Events
I
n
d
e
x CEP
RAM
Cache Joins
RAM Disk Aggregates
HBase
HDFS
Cetas Software Inc. – Copyright © 2012– CONFIDENTIAL – DO NOT DISTRIBUTE 12

13. Wrap-up INSTANT INTELLIGENCE
 Personalized Recommendation Engine
– Non-trivial
– Focus on Specific Use Case
 Real-time
– Distributed Indexing
– Pre-computation
– Compact store (in memory, on disk)
– Parallelization
Cetas Software Inc. – Copyright © 2012– CONFIDENTIAL – DO NOT DISTRIBUTE 13

14. References INSTANT INTELLIGENCE
 Mining Massive Datasets
– Free eBook – Anand Rajaraman, Jeff Ullman
– cs246.stanford.edu
 Introduction to Data Mining
– Tan, Steinback, Kumar
 Introduction to Recommender Systems
Handbook
– Ricci, Rokach, Shapira
Cetas Software Inc. – Copyright © 2012– CONFIDENTIAL – DO NOT DISTRIBUTE 14

15. INSTANT INTELLIGENCE
Cetas Software Inc. – Copyright © 2012– CONFIDENTIAL – DO NOT DISTRIBUTE 15