If you’re trying to process financial market data, monitor IoT sensor metrics or run real-time fraud detection, you’ll be thinking of stream processing. Stream processing sounds wonderful in concept, but scaling and debugging stream processing frameworks on distributed systems can be a nightmare. In clustered environments, your logs are scattered across many different computers making errors and strange behaviors are hard to trace. On frameworks like Apache Storm, the many layers of abstraction make it difficult to predict performance and do capacity planning. In micro batching frameworks like Spark Streaming, stateful aggregations can be a hassle. Moreover, in most of the existing frameworks, changing a single line of code requires a full topology redeploy causing operational strain. Concord strives to solve all the challenges above. In this talk, you’ll learn how Concord differs from other stream processing frameworks and how Concord can provide flexibility, simplicity, and predictable performance with help from Apache Mesos.
https://databythebay2016.sched.org/event/6EPy/concord-simple-amp-flexible-stream-processing-on-apache-mesos
Presentation on how to chat with PDF using ChatGPT code interpreter
Concord: Flexible Stream Processing on Apache Mesos
1. Concord: Simple & Flexible
Stream Processing on Apache Mesos
Shinji Kim
Co-founder, Concord Systems
@concord
@databythebay #datagrid
2. Overview
• What is Stream Processing?
• Today’s Stream Processing
• Introducing Concord
1. Concepts & API
2. Job Topology Management
3. Operations, Toolings, Performance
4. Message Delivery Guarantees
• Future Development Plans
Page 2
3. What is stream processing?
Page 3
• Processing Data in motion
• Sits between message queues and databases
• Used for faster:
– Data enrichment
– Aggregation
– Filtering / deduplication
4. Today’s Stream Processing
• Faster MapReduce jobs à ends up running core
business logic on top
– Fradulent click detection
– Real-time budget updates
– Trigger-based trading
• Your stream processing jobs are more like microservices
• Need support for services / application management:
Cluster mgmt, Monitoring, Debuggability
Page 4
5. Introducing Concord
Concord is a distributed stream processing framework
built in C++ on top of Apache Mesos, designed for
high-performance, real-time applications that require
flexibility & control.
Page 5
7. Pub / Sub Operator Model
• Composable jobs by Metadata
A
B
words
Metadata(
Name=‘A’,
istreams=[],
ostreams=[‘words’])
Metadata(
Name=‘B’,
istreams=[‘words’,
StreamGrouping.GROUP_BY],
ostreams=[])
Page 7
8. Pub / Sub Operator Model
• Composable jobs by Metadata
A
B
words
Metadata(
Name=‘A’,
istreams=[],
ostreams=[‘words’])
Metadata(
Name=‘B’,
istreams=[‘words’,
StreamGrouping.GROUP_BY],
ostreams=[])
Page 8
C
Metadata(
Name=‘C’,
istreams=[‘words’,
StreamGrouping.SHUFFLE],
ostreams=[])
9. Simple API in Multiple Languages
• ProcessRecord, ProduceRecord, ProcessTimer
• GetState, SetState backed by Rocksdb
• API available in Python, Ruby, Go, Java/Scala, C++
B
Metadata(
Name=‘C’,
istreams=[‘words’,
StreamGrouping.GROUP_BY],
ostreams=[‘wordcount’])
Page 9
words
wordcount
Key
Value
Corgi
2
Chiwawa
4
Dashhound
5
10. Useful for multiple teams to consume the same
streaming data in real-time
Page 10
20. Concord performs well at scale
• Word count benchmark (1.13B msgs)
– Concord: 500K QPS/node at 10ms/event
– Storm: 16K QPS/node at 100ms/event
– Spark Streaming: 100K QPS/node at 1s batch window
• Server log processing (29G server log, ~260M msgs)
– 4 nodes, 8 vCPU, 32GB RAM each
– Concord: 1M – 1.8M QPS
– Spark Streaming: 72K – 2M QPS
• Consistent performance
Page 20
21. Concord is designed for Predictability
• As you scale, JVM reconfiguration and GC pauses are
inevitable (Framework GC vs. Application GC)
• Cluster abstracted as CPU, Memory, Disk numbers à
cluster optimization & overall runtime
• Fast Compile à Test à Deploy cycle without downtime
Page 21
22. Message Delivery Guarantees
Today: Fast > Complete or Perfect
• Best-effort / at-most-once processing
– When operator or node crashes, the local cache goes away
– Automatically retries the failed operator (number of retries is
configurable)
– Recommends implementing check mechanisms in operators
(e.g., Concord Kafka consumer)
Page 22
23. Message Delivery Guarantees
Soon: Fast + Complete > Perfect
• In development for at-least-once with Kafka
– Kafka acts as a message bus between operators
– Kafka replays data from checked offset (data duplication)
Eventually: Fast + Complete + Perfect
• Transactional datastore in design phase
Page 23
24. Future plans
• “At least once” guarantee support with Kafka
• DC/OS integration
• More data source / data sink connector support
• Higher level DSL
Page 24
25. Concord: Simple & Flexible streaming application
framework on Apache Mesos
Page 25
• Operator model that you can use multiple languages
26. Concord: Simple & Flexible streaming application
framework on Apache Mesos
Page 26
• Operator model that you can use multiple languages
à Fast development and iteration time for multiple
teams using the same data
27. Concord: Simple & Flexible streaming application
framework on Apache Mesos
Page 27
• Operator model that you can use multiple languages
à Fast development and iteration time for multiple
teams using the same data
• Dynamic topology, run-time deployment and scaling
28. Concord: Simple & Flexible streaming application
framework on Apache Mesos
Page 28
• Operator model that you can use multiple languages
à Fast development and iteration time for multiple
teams using the same data
• Dynamic topology, run-time deployment and scaling
à Decoupled development & dev ops work
29. Concord: Simple & Flexible streaming application
framework on Apache Mesos
Page 29
• Operator model that you can use multiple languages
à Fast development and iteration time for multiple
teams using the same data
• Dynamic topology, run-time deployment and scaling
à Decoupled development & dev ops work
• High performance at scale
30. Concord: Simple & Flexible streaming application
framework on Apache Mesos
Page 30
• Operator model that you can use multiple languages
à Fast development and iteration time for multiple
teams using the same data
• Dynamic topology, run-time deployment and scaling
à Decoupled development & dev ops work
• High performance at scale
à Predictable system for real-time applications
31. Concord: Simple & Flexible streaming application
framework on Apache Mesos
Page 31
• Low-latency / Real-time applications:
– Real-time fraud detection
– Financial market data processing for real-time risks and triggers
– Real-time campaign management for real-time bidding (RTB)