We have seen tremendous growth in near real-time ("nearline") processing at LinkedIn in recent years. LinkedIn now uses Apache Samza to process well over a Trillion messages every day across thousands of applications. Apache Samza serves as the foundation for several application platforms at LinkedIn, spanning a wide variety of use cases like security, notifications, machine learning, monitoring, search, and more. In this talk we will explore various features of Apache Samza that provide the flexibility and scalability to we need to power stream processing at massive scale.
2. Agenda
● Stream Processing at LinkedIn
○ Scale at LinkedIn
○ Scenarios at LinkedIn
● Apache Samza
○ Processing Model
○ Stateful Processing
○ Processing APIs
○ Deployment Model
3. Apache Kafka
5 Trillion+ messages ingested
per day
1.5+ PB data per day
100k+ topics, 5M+ partitions
Brooklin
2 Trillion+ messages moved
per day
10k+ topics mirrored
2k+ change capture streams
Apache Samza
1.5 Trillion+ messages
processed per day
3k+ jobs in production
500 TB+ local state
Scale at LinkedIn
4. Scenarios at LinkedIn
DDoS prevention,
bot detection, access
monitoring
Security
Email and Push
notifications
Notifications
Topic tagging, NER in
news articles, image
classification
Classification
Site speed and
health monitoring
Site Speed
Monitoring
inter-service
dependencies and
SLAs
Call Graphs
5. Scenarios at LinkedIn
Tracking ad views
and clicks
Ad CTR
Tracking
Pre-aggregated
real-time counts by
dimensions
Business
Metrics
Standardizing titles,
companies,
education
Profile
Standardization
Updating search
indices with new
data
Index
Updates
Tracking member
page views,
dwell-time, sessions
Activity
Tracking
6. Hardened
at Scale
In production at
LinkedIn, Slack, Intuit,
TripAdvisor, VMWare,
Redfin, etc.
Processing events from
Kafka, Brooklin,
Kinesis, EventHubs,
HDFS, DynamoDB
Streams, Databus, etc.
Apache Samza
Incremental checkpoints
for large local state and
instant recovery.
Local state that works
seamlessly across
upgrades and failures.
APIs for simple and
efficient remote I/O
Best In Class
Stateful Processing
Stream and batch
processing without
changing code.
Convenient High-level
DSLs and a powerful
Low-level API.
Universal
Processing APIs
Write once, run
anywhere.
Run on a multi-tenant
cluster or as an
embedded library.
Flexible
Deployment Model
8. ● Parallelism across tasks by increasing the number of containers.
○ Up to 1 container per task.
● Parallelism across partitions by increasing the number of tasks.
○ Up to 1 task per partition.
● Parallelism within a partition for out of order processing.
○ Any number of threads.
Scaling a Samza Application
9. Hardened
at Scale
In production at
LinkedIn, Uber, Slack,
Intuit, TripAdvisor,
VMWare, Redfin, etc.
Processing events from
Kafka, Brooklin, Kinesis,
EventHubs, HDFS,
DynamoDB Streams,
Databus, etc.
Apache Samza
Incremental checkpoints
for large local state and
instant recovery.
Local state that works
seamlessly across
upgrades and failures.
APIs for simple and
efficient remote I/O
Best In Class
Stateful Processing
Stream and batch
processing without
changing code.
Convenient High-level
DSLs and a powerful
Low-level API.
Universal
Processing APIs
Write once, run
anywhere.
Run on a multi-tenant
cluster or as an
embedded library.
Flexible
Deployment Model
10. • State is used for performing lookups
and joins, caching data,
buffering/batching data, and writing
computed results.
• State can be local (in-memory or on
disk) or remote.
Samza
Local Store I/O
Samza
Why State Matters
and
Remote DB I/O
11. Why Local State Matters: Throughput
on disk w/ caching comparable with in memory changelog adds minimal overhead
remote state
30-150x worse than
local state
Terminology
Disk Type: SSD
Max-Net: Max network bandwidth
CLog: Kafka changelog
ReadOnly: read only workloads (lookups)
ReadWrite: read - write workloads (counts)
Shadi A. Noghabi et al. Samza: stateful scalable stream processing at LinkedIn. Proc. VLDB Endow. 10, 12 (August 2017), 1634-1645.
12. Why Local State Matters: Latency
on disk w/ caching comparable with in memory changelog adds minimal overhead
> 2 orders of magnitude slower compared to
local state
Shadi A. Noghabi et al. Samza: stateful scalable stream processing at LinkedIn. Proc. VLDB Endow. 10, 12 (August 2017), 1634-1645.
13. Optimizations for Local State
Task-1
Container-1
Samza Application Master
Durable Container ID – host mapping
1. Log state changes to a Kafka compacted
topic for durability.
2. Catch up on only the delta from the
change log topic on restart.
Task-2
Container-2
14. Optimizations for Local State
1. Host Affinity
2. Parallel Recovery
3. Bulk Load Mode
4. Standby Containers
5. Log Compaction
Task-1
Container-1
Samza Application Master
Durable Container ID – host mapping
Task-2
Container-2
15. Why Remote I/O Matters
• Data is only available in the remote store (no change capture).
• Need strong consistency or transactions.
• Data cannot be partitioned but is too large to copy to every container.
• Writing processed results for online serving.
• Calling other services to handle complex business logic.
17. Hardened
at Scale
In production at
LinkedIn, Uber, Slack,
Intuit, TripAdvisor,
VMWare, Redfin, etc.
Processing events from
Kafka, Brooklin, Kinesis,
EventHubs, HDFS,
DynamoDB Streams,
Databus, etc.
Apache Samza
Incremental checkpoints
for large local state and
instant recovery.
Local state that works
seamlessly across
upgrades and failures.
APIs for simple and
efficient remote I/O
Best In Class
Stateful Processing
Stream and batch
processing without
changing code.
Convenient High-level
DSLs and a powerful
Low-level API.
Universal
Processing APIs
Write once, run
anywhere.
Run on a multi-tenant
cluster or as an
embedded library.
Flexible
Deployment Model
18. Example Application
Count number of "Page Views" for each member in a 5 minute window
18
Page View
Page View Per
Member
Repartition
by member id
Window Map SendTo
Intermediate Stream
19. High Level API
● Complex Processing Pipelines
● Easy Repartitioning
● Stream-Stream and Stream-Table Joins
● Processing Time Windows and Joins
20. High Level API
public class PageViewCountApplication implements StreamApplication {
@Override
public void describe(StreamApplicationDescriptor appDescriptor) {
KafkaSystemDescriptor ksd = new KafkaSystemDescriptor("tracking");
KafkaInputDescriptor<PageViewEvent> pageViews = ksd.getInputDescriptor("PageView", serde);
KafkaOutputDescriptor<PageViewCount> pageViewCounts = ksd.getOutputDescriptor("PageViewCount", serde);
appDescriptor.getInputStream(pageViews)
.partitionBy(m -> m.memberId, serde)
.window(Windows.keyedTumblingWindow(m -> m.memberId, Duration.ofMinutes(5),
initialValue, (m, c) -> c + 1))
.map(PageViewCount::new)
.sendTo(appDescriptor.getOutputStream(pageViewCounts));
}
}
21. Apache Beam
● Event Time Processing
● Multi-lingual APIs (Java, Python, Go*)
● Advanced Windows and Joins
22. Apache Beam
public class PageViewCount {
public static void main(String[] args) {
...
pipeline
.apply(LiKafkaIO.<PageViewEvent>read()
.withTopic("PageView")
.withTimestampFn(kv -> new Instant(kv.getValue().header.time))
.withWatermarkFn(kv -> new Instant(kv.getValue().header.time - 60000))
.apply(Values.create())
.apply(MapElements
.into(TypeDescriptors.kvs(TypeDescriptors.strings(), TypeDescriptors.integers()))
.via((PageViewEvent pv) -> KV.of(String.valueOf(pv.header.memberId), 1)))
.apply(Window.into(TumblingWindows.of(Duration.standardMinutes(5))))
.apply(Count.perKey())
.apply(MapElements
.into(TypeDescriptors.kvs(TypeDescriptors.strings(), TypeDescriptor.of(Counter.class)))
.via(newCounter()))
.apply(LiKafkaIO.<Counter>write().withTopic("PageViewCount")
pipeline.run();
}
}
24. Samza SQL
● Declarative streaming SQL API
● Managed service at LinkedIn
● Create and deploy applications in minutes using SQL Shell
25. Samza SQL
INSERT INTO kafka.tracking.PageViewCount
SELECT memberId, count(*) FROM kafka.tracking.PageView
GROUP BY memberId, TUMBLE(current_timestamp, INTERVAL '5' MINUTES)
27. Hardened
at Scale
In production at
LinkedIn, Uber, Slack,
Intuit, TripAdvisor,
VMWare, Redfin, etc.
Processing events from
Kafka, Brooklin, Kinesis,
EventHubs, HDFS,
DynamoDB Streams,
Databus, etc.
Apache Samza
Incremental checkpoints
for large local state and
instant recovery.
Local state that works
seamlessly across
upgrades and failures.
APIs for simple and
efficient remote I/O
Best In Class
Stateful Processing
Stream and batch
processing without
changing code.
Convenient High-level
DSLs and a powerful
Low-level API.
Universal
Processing APIs
Write once, run
anywhere.
Run on a multi-tenant
cluster or as an
embedded library.
Flexible
Deployment Model
28. Samza on a Multi-Tenant Cluster
• Uses a cluster manager (e.g. YARN) for resource management,
coordination, liveness monitoring, etc.
• Better resource utilization in a multi-tenant environment.
• Works well for large number of applications.
29. Samza as an Embedded Library
• Embed Samza as a library in an application. No cluster manager dependency.
• Dynamically scale out applications by increasing or decreasing the number of
processors at run-time.
• Supports rolling upgrades and canaries.
30. ● Uses ZooKeeper for leader election and liveness monitoring for processors.
● Leader JobCoordinator performs work assignments among processors.
● Leader redistributes partitions when processors join or leave the group.
Samza as a Library
ZooKeeper Based Coordination
Zookeeper
StreamProcessor
Samza
Container
Job Coordinator
StreamProcessor
Samza
Container
Job
Coordinator
StreamProcessor
Samza
Container
Job
Coordinator…
Leader
31. Apache Samza
• Mature, versatile, and scalable processing framework
• Best-in-class support for local and remote state
• Powerful and flexible APIs
• Can be operated as a platform or used as an embedded library
