Pipelines have become ubiquitous, as the need for stringing multiple functions to compose applications has gained adoption and popularity. Common pipeline abstractions such as “fit” and “transform” are even shared across divergent platforms such as Python Scikit-Learn and Apache Spark. Scaling pipelines at the level of simple functions is desirable for many AI applications, however is not directly supported by Ray’s parallelism primitives. In this talk, Raghu will describe a pipeline abstraction that takes advantage of Ray’s compute model to efficiently scale arbitrarily complex pipeline workflows. He will demonstrate how this abstraction cleanly unifies pipeline workflows across multiple platforms such as Scikit-Learn and Spark, and achieves nearly optimal scale-out parallelism on pipelined computations. Attendees will learn how pipelined workflows can be mapped to Ray’s compute model and how they can both unify and accelerate their pipelines with Ray.

1. Scaling and Unifying
Scikit Learn and Spark
Pipelines using Ray
Raghu Ganti
Principal Research Staff Member
IBM T J Watson Research Center
Team (IBM & Red Hat):
Michael Behrendt, Linsong Chu, Carlos
Costa, Erik Erlandson, Mudhakar Srivatsa

2. So many pipelines…
And many more…

3. Ray.IO
§ Can we do pipelines on
Ray?
§ Can we scale popular
AI/ML pipelines on Ray?
§ Can we unify scikit learn
and Spark pipelines?

4. Current pipeline API
• Focus on scikit learn and Spark pipelines
• Scikit learn missing scaling; Spark focus on data parallel
scaling
Transform
Fit
X
X
y
X’
Fitted model

5. Scaling Pipelines: I/O as List of Objects
Transform
Fit
[X1, X2, … XN]
[X1, X2, … XN]
[y1, y2, … yN]
[X1’, X2’, …, XN’]
[FM1, FM2, … FMN]

6. Scaling Pipelines: AND/OR Graphs
And node
X1
X2
XN
X1’
X2’
XM’
Or node
X
Step1
Step2
StepN
X’
X’
X’

7. Key Features
▪ Python function as
unit of compute
▪ Intuitive for data
scientist
▪ Follows transformer
APIs
▪ MPI-style scaling
▪ Object references
as I/O for unit of
compute
▪ Sharing of objects
using Plasma store
▪ Enables zero-copy
object sharing
• List of objects as I/O
• Function as unit of
compute
▪ Scikit learn typically
in Python
▪ Ray.IO with RayDP
enables efficient
data exchange
• Cross environment
▪ Enriched DAGs from
plain pipelines
▪ OR nodes for fan-
out expressions
▪ AND nodes for
arbitrary lambdas
• AND/OR Graphs

8. Illustrative Example
8
Preprocess
Random
Forest
Gradient
Boost
Decision
Tree
Sample Pipeline
Scikit learn Pipeline
Our Pipeline

9. Pipelines Galore…
Airflow Kubeflow Scikit learn
Spark
Pipeline
Our
pipeline
Task
parallelism
✓ ✓ ✗ ✓ ✓
Data
parallelism
✗ ✗ ✗ ✓ ✓
And/Or Graphs ✓ ✓ ✗ ✗ ✓
Computational
unit
Container Container
Python
function
Python/Java
function
Python/Java
function
Mutability of
DAG
✗ ✗ ✓ ✓ ✓

10. What to expect?
• Execution strategies based on graph traversals
• Early stopping criteria
• Mutability of execution pipelines
• Current status: Proposal discussion with Ray and OSS
community

11. Q&A
Contacts:
Raghu Ganti (rganti@us.ibm.com)
Michael Behrendt (michaelbehrendt@de.ibm.com)
Linsong Chu (lchu@us.ibm.com)
Carlos Costa (chcost@us.ibm.com)
Erik Erlandson (eerlands@redhat.com)
Mudhakar Srivatsa (msrivats@us.ibm.com)

12. Feedback
Your feedback is important to us.
Don’t forget to rate and review the sessions.