"Machine learning (ML) models are deployed for production use cases with ever increasing pace, driving the growing need for machine learning operations (MLOps) for the deployment, monitoring, and explainability of ML models at scale. With the rise of the data-centric AI movement, businesses are seeking solutions that will provide them with highly discoverable and available data for monitoring, governance, and compliance. In this talk, we identify dataflow architectural principles to address these demands and discuss their application in an open-source ecosystem. We show how to create a decentralized dataflow engine underpinned by Kafka and the Kafka Streams client library, and how this can be leveraged for building flexible data processing pipelines on-the-fly. We explore the challenges faced in creating such a dataflow engine and reflect on our journey with the Kafka ecosystem. We consider managing dynamically-created Kafka Streams topologies, multiplexing hundreds or even thousands of these topologies onto individual JVM instances, and the integration between Kafka and Kotlin, amongst other things."

1. Dataflows for machine learning operations
Andrei Paleyes and Alex Rakowski

2. Part I – Dataflow and Streams

3. Dataflows for machine learning operations
What is ML@CL group?
3
Machine
Learning
@
Computer
Lab
3

4. Dataflows for machine learning operations
Dataflow research at ML@CL
4 4

5. Dataflows for machine learning operations
Open Source: Seldon Core
5 5

6. Dataflows for machine learning operations
Inference graph
6 6

7. Dataflows for machine learning operations
Pipelines
7
apiVersion: mlops.seldon.io/v1alpha1
kind: Pipeline
metadata:
name: road-counter
spec:
steps:
- name: vectorize
- name: faulty_image_filter
inputs:
- vectorize.outputs
- name: object_detection
inputs:
- vectorize.outputs
- faulty_image_filter.outputs
. . . . .
output:
steps:
- counter
7

8. Core V1 - Central orchestration
8 8
Dataflows for machine learning operations

9. Dataflow architecture
9 9
Dataflows for machine learning operations
Dean, J. & Ghemawat, S. (2008). MapReduce: simplified data processing on large clusters. Communications of the ACM, 51(1), 107-113.

10. Dataflow vs control flow
10
“In control flow, the processor follows
explicit order, executing instructions one
after another. In dataflow, by contrast, an
instruction is ready to execute as soon as
all its inputs are available.”
M. Schwarzkopf, The Remarkable Utility of Dataflow Computing
10
Dataflows for machine learning operations

11. Dataflow features
11
Separation of data
and operations
Data as a first
class citizen
Dataflow graph of
the entire system
Decentralisation
11
Dataflows for machine learning operations

12. Dataflows for machine learning operations
Dataflow and inference graph
12 12

13. Role of streaming in dataflow
13 13
Dataflows for machine learning operations

14. Control plane
14 14
Dataflows for machine learning operations

15. Part II – Implementation

16. Tech Stack — Constraints
16
OSS Platform agnostic Lightweight
16
Dataflows for machine learning operations

17. Tech Stack — Contenders
17 17
Dataflows for machine learning operations

18. Pub/Sub – Multiplexing Pipelines
18 18
Dataflows for machine learning operations

19. Pub/Sub – Extending Pipelines
19 19
Dataflows for machine learning operations

20. Pub/Sub – Repeatable Streams
20 20
Dataflows for machine learning operations

21. Tech Stack — Choices
21 21
Dataflows for machine learning operations

22. Anatomy of a Topic Name
22
<prefix>.<namespace>.<pipeline|model>.<name>.<inputs|outputs>
seldon.default.pipeline.foo.inputs
seldon.recommendations.model.bar.outputs
<prefix>.<namespace>.errors.errors
ml.default.errors.errors
22
Dataflows for machine learning operations

23. Anatomy of an Inference Message
23 23
Dataflows for machine learning operations

24. Open Inference Protocol
24 24
Dataflows for machine learning operations

25. Open Inference Protocol – Batching
25 25
Dataflows for machine learning operations

26. Dataflow Operations – Topic Chaining
26 26
Dataflows for machine learning operations

27. Dataflow Operations – Tensor Projection
27 27
Dataflows for machine learning operations

28. 28
Dataflow Operations – Batching
Dataflows for machine learning operations

29. 29 29
Dataflow Operations – Inner Join
Dataflows for machine learning operations

30. Dataflow Operations – Outer Join
30 30
Dataflows for machine learning operations

31. 31 31
Dataflow Operations – Any Join
Dataflows for machine learning operations

32. Dataflow Operations – Triggers
32 32
Dataflows for machine learning operations

33. Core v2 Pipeline = KStream
Topology
Topologies
33 33
Dataflows for machine learning operations
apiVersion: mlops.seldon.io/v1alpha1
kind: Pipeline
metadata:
name: road-counter
spec:
steps:
- name: vectorize
- name: faulty_image_filter
inputs:
- vectorize.outputs
- name: object_detection
inputs:
- vectorize.outputs
- faulty_image_filter.outputs
...
val builder = StreamsBuilder()
builder
.stream(inputTopic.topicName, consumerSerde)
.filterForPipeline(inputTopic.pipelineNam
e)
...
val topology = builder.build()

34. Pipeline Allocation – 1 Topology = 1 Engine
34 34
Dataflows for machine learning operations

35. Pipeline Allocation – Shared Engines
35 35
Dataflows for machine learning operations

36. Pipeline Allocation — Threads
36 36
Dataflows for machine learning operations

37. How Many Is Too Many?
37 37
Dataflows for machine learning operations

38. How Many Is Too Many?
38 38
Dataflows for machine learning operations
KSQL Limitations in Confluent Cloud
Kafka Summit 2022 - Modular Topologies
KIP-809 (Modular Topologies)

39. Advanced ML Applications
39
Dataflows for machine learning operations
39

40. Challenges
40 40
Dataflows for machine learning operations
Dynamism Scalability Variety

41. Thanks for listening!
41
Dataflows for machine learning operations
41

42. 42
The Seldon Core v2 Team
hello@seldon.io
Core v2 on GitHub Team
Clive Cox
CTO
Rafal Skolasinski
Machine Learning Engineer
Alex Rakowski
Software Engineer
Sherif Akoush
MLOps Engineer
Adrian Gonzalez-Martin
Machine Learning Engineer
Andrei Paleyes
MLOps Researcher
Contact us: