This is a quick overview of LinkedIn's stream processing tech stack.

1. Stream Processing @Scale in
LinkedIn
Yi Pan
Data Infrastructure
Samza Team @LinkedIn
Databus

2. • What is Stream Processing?
• What is Samza?
• Stream Processing @LinkedIn
• Upcoming features
Overview

3. • What’s stream processing
– Input: an unbounded sequence of events
• E.g. web server logs, user activity tracking events,
database changelogs, etc.
– Latency: near real-time
• From milliseconds to minutes, instead of hours to days
– Output: an unbounded sequence of changes to
the derived dataset
• The derived dataset is usually the final or partial
analytic results that can either be in another stream, or
a serving data store
Stream Processing

4. Response latency
Milliseconds to minutes
Synchronous Later. Possibly much later.
0 ms
Stream Processing

5. • What are the application requirements?
– Scalable, fast, stateful stream processing
– What scale should we operate at?
• Traffic Volume: 1.4 Trillion events/day
• Intermediate State Size: multi TB / colo (*)
– Why is it expensive to run stream processing at
scale?
• Intermediate data set needs to be stored to allow low
latency processing
• Large volume of data needs to be pulled and pushed
via network
Stream Processing

6. • What is Stream Processing?
• What is Samza?
• Stream Processing @LinkedIn
• Upcoming features
Overview

7. • Samza is a distributed Turing machine
– Single Task Samza Job is a stateful Turing
machine
What’s Samza
Samza Task
Input stream Output stream
State
changelog
checkpoint

8. – Scaling a Samza job: partition the streams
What’s SamzaInputstreamA
partition 0
partition 1
partition 2
partition 3
partition n

9. – Scaling a Samza job: partition the streams
What’s SamzaInputstreamB
partition 0
partition 1
partition 2
partition 3
partition n

10. – Scaling a Samza job: replicating the state
machine
What’s Samza
shared checkpoint
Job

11. • Samza Execution in Yarn
What’s Samza
Host 1 Host 2 Host 3
Application
Master
Samza container Samza container
Samza container
Deploy Samza job

12. • States in Samza
– Checkpoints
• Offsets per input stream partitions
– State Stores
• In-memory or on-disk (RocksDB) derived data set
What’s Samza
Samza Task
Output stream partitions
State
changelogpartitions
checkpoint Host 1

13. • States in Samza
– Checkpoints and local state stores are backed by
distributed logs
What’s Samza
Samza Task
Output stream partitions
State
changelogpartitions
checkpoint Host 1

14. • What is Stream Processing?
• What is Samza?
• Stream Processing @LinkedIn
• Upcoming features
Overview

15. Stream Processing @
LinkedIn
WebServers
WebServers
WebServers
WebServers
WebServers
WebServers
WebServersMonitor
Servers
Oracle
Espresso
Kafka Databus
Tracking
events
Metrics
changelog
changelog
Samza
Jobs
Samza
Jobs
Samza
Jobs
Samza
Jobs
bootstrap
bootstrap
Voldemort
Derived
Data Derived
Data

16. Stream Processing @
LinkedIn
• Tracking aggregate/analysis (ACG)

17. Stream Processing @
LinkedIn
• Content standardization w/ adjunct data
set
Member
Profile DB
Bootstrap
Job
Databus
Kafka
Content
Standardization
Kafka
Kafka

18. Stream Processing @
LinkedIn
• Kafka Deployment
– 1.1 Trillion messages / day
• Databus Deployment
– 300 Billion messages / day
• Samza Deployment
– multiple colos
– 10+ Yarn clusters
– 200+ nodes
– 100+ Jobs in production

19. • What is Stream Processing?
• What’s Samza
• Stream Processing @LinkedIn
• Upcoming features
Overview

20. • New features
– Local state store improvements
• RocksDB TTL support
• Fast recovery
– Dynamic configuration
– Easier deployment w/ standalone jobs
– High-level query language for faster
development
Upcoming Features

21. Contact Us / Get Involved
• Open Source
–Documentation: samza.apache.org
–Mailing list: dev@samza.apache.org
–JIRA:
https://issues.apache.org/jira/browse/SA
MZA