Open-source technologies allow developers to build microservices framework to build myriad real-time applications. One such application is building the real-time model scoring. In this session, we will showcase how to architect a microservice framework, in particular how to use it to build a low-latency, real-time model scoring system. At the core of the architecture lies Apache Spark’s Structured Streaming capability to deliver low-latency predictions coupled with Docker and Flask as additional open source tools for model service. In this session, you will walk away with: * Knowledge of enterprise-grade model as a service * Streaming architecture design principles enabling real-time machine learning * Key concepts and building blocks for real-time model scoring * Real-time and production use cases across industries, such as IIOT, predictive maintenance, fraud detection, sepsis etc.

1. A Microservices Framework
for
Real time Model Scoring
using
Structured Streaming
Vedant Jain
October 2018
#SAISStreaming1

2. 2
About me
• Solutions Architect @ Databricks
• x-Hortonworks, JPMC

3. 3
What will we talk about?
§ Machine learning
§ Model scoring
§ Microservices
§ Structured Streaming
§ Demo

4. 4
Machine Learning
“…what we want is a machine that can learn from experience.”

5. Types of Machine Learning
• Unsupervised Learning

6. X Y Z
0.5 -3.4 9.2
-2.3 2.9 3.2
1.3 2.3 -4.5
• Supervised Learning
Types of Machine Learning
Activity
Biking
Standing
Sitting
Labels
X Y Z
0.5 -3.4 9.2

7. X Y Z
0.5 -3.4 9.2
-2.3 2.9 3.2
1.3 2.3 -4.5
• Supervised Learning
Types of Machine Learning
Activity
Biking
Standing
Sitting

8. Continuous Learning
Machine Learning Process
Define the
problem
Connect data
source
Acquire Data
Enrich, Transform
Data
Feature
Selection
Build/Train
Model
Serve/Score
Model

9. 9
What will we talk about?
§ Machine learning
§ Model scoring
§ Microservices
§ Structured Streaming
§ Demo

10. Model Scoring
Propensity: Likelihood of a user to
commit a certain action
Affinity: How similar are two products or
users etc.
Lead: How closely matched lead is to
target profile
Attrition/Churn: Likelihood of a
customer to drop a service and/or start
using a competitor’s service
Credit: Ability of the user to keep
promise if granted access
Anomaly Detection: Identification of
rare or invalid transaction

11. 11
Model Scoring on “Big Data”
Scale Streaming Dynamic

12. Machine Learning Pipeline
12
• Drop null values
• Calculate moving
averages on 10 minute
window
• Convert to single
parquet
• OneHotEncoder
• RF Classifier
• Cross Tabulation
• K-Means Clustering
• Hyperparameter Tuning
• Pipelines
• Deploy Pipelines to
Docker
• Serve the models using
Python Flask
• Phone/Watch
• Gyroscope/Accelerometer
• CSV

13. 13
What will we talk about?
§ Machine learning
§ Model scoring
§ Microservices
§ Structured Streaming
§ Demo

14. Microservices
● Properties:
Ø Decomposition: Logic is broken down into multiple independent components
Ø Isolation: Component services are deployed and maintained independently of one another
● Benefits:
ü Reduced Regression Testing time
ü Organizational Autonomy
ü Cloud/On-premise Agnostic
ü Scalability/Reusability

15. Microservices
Example:
Company A tracks user’s activity through smart
devices and wants to provide tailored content to the
users based on their behavior.

16. Microservices
X Y Z Activity
0.5 -3.4 9.2 Biking
-2.3 2.9 3.2 Standing
1.3 2.3 -4.5 Sitting
‘Activity’
score
User Bike Stand Sit Cluster
A 0.1 0.3 0.6 0
B 0.0 0.3 0.7 1
Actionable Insights
Web services
Target
content
Affinity Score

17. Microservices

18. Microservices
Isolation

19. Microservices
Scalability
• Consistent
• Scalable
• Fault tolerant
• Complex Event
Processing

20. Microservices

21. 21
What will we talk about?
§ Machine learning
§ Model scoring
§ Microservices
§ Structured Streaming
§ Demo

22. Structured Streaming
• High-level streaming API
built on Spark SQL engine
• Runs the same computation
as batch queries in
Datasets/DataFrames
• Event time, windowing,
sessions, sources & sinks
• End-to-end exactly once
semantics
• Late Data Handling

23. 23
ML Limitations in Streaming
• Many models/transformers/estimators are not supported
• Limited to only models built in Spark MLLib
• Not ideal for Continuous Learning
23

24. 24
What will we talk about?
§ Machine Learning
§ Model scoring
§ Microservices
§ Structured Streaming
§ Demo
https://github.com/vedantja/eu_summit_demo