Abstract:- In this talk, we propose a generalized machine learning framework for e-commerce businesses. The framework is responsible for over 30 different user-level predictions including lifetime value, recommendations, churn predictions, engagement and lead scoring. These predictions provide a vital layer of intelligence for a digital marketer. Kinesis is used to capture browsing information from over 120M users across 100 companies (both in-app and web). A data processing and feature engineering layer is build on Apache Spark. These features provide inputs to predictive models for business applications. Different models each for Churn, Lifetime value, Product recommendation and search are written on Spark. These models can be plugged into any marketing campaign for any integrated e-commerce company leading to a generalized system. We finally present a monitoring system for machine learning called RS Sauron. This system provides more than 200 objective metrics measuring the health of predictive models, and depicts KPIs for model accuracy in a continual setting.
6. The Challenge
• Many Clients
• Dirty Data
• Sparse Datasets
• Custom Attributes
• Various Industries
Clean
PredictionsModel Layer!
C1 C2 C3 C4
7. What Kind of Predictions?
Purchase Probability: High-likelihood
Lifecycle Stage: Ready to Buy
Churn Time: 300 days
Customer Future Value: $925
Contact Frequency: Every 3 days
Optimal Time to Engage: Thursday 7-9PM
Optimal Incentive/Discount: Dollars Off
Product Recommendations: Based on interest
Optimal Subject Line: Individual preference
Optimal Template: Individual preference
8. Our Approach & Learnings
1. Robust Ingestion Pipeline
2. Common Feature Engineering Layer
3. “Plug-in” Architecture for Models
4. Evaluation / AB Testing
5. Robust Monitoring & Visualization
9. 1. Robust Ingestion Pipeline
10K+ Actions Per Second
auto-scaling!
auto-scaling
lambdas!
• Abstraction Layer: Data
Ingestion
• Do not compromise for
clean data
• Auto-scaling everywhere
• High confidence in
upstream data
Flume
Kinesis
10. 2. Common Feature Engineering Layer
• Abstraction: Feature Layer
• Allow custom features
• Handle feature selection
• Modelers know what to
expect
Raw Data
User Behavior
Features
Product
Features
User Sta4c
Features
Timing Model CLV Model Recommender
11. 3. Model Plug-in Architecture
C6
C3
C4
C5
C7
• Plug-in Architecture
• Tune model hyper-parameters
• A/B test models per client
C1
C8 C2
Client’s Model Execution Plan
13. 4. Model Evaluation / Fast Feedback
A/B Framework
M1 M2 M3
• Start Simple
• Collect feedback data
• Skip long production cycle
• Unbiased policy
generation is important
M1
Campaign Predic4ons
14. 5. Robust Model Monitoring and Visualization
“Sauron”
(LOTR)!
Monitor,
monitor,
monitor!!