This document discusses Apache Flink and its integration with Apache Mesos and DC/OS. It provides an overview of Flink's stream processing capabilities and common use cases. It then describes how Flink runs on Mesos using Fenzo for resource allocation and task scheduling. A demo is shown of a financial data pipeline using Kafka for ingestion and output, and Flink for real-time stream processing on Mesos via DC/OS. In conclusion, Flink is a modern stream processor that can run on Mesos through its integration, and DC/OS provides an easy-to-use package for deploying Flink.

1. Till Rohrmann
trohrmann@apache.org
@stsffap
Apache Flink® Meets
Apache Mesos® and DC/OS

4. MapReduce is
crunching Data

5. We need to turn faster!

6. FMACK Stack
EVENTS
Ubiquitous data
streams from
connected devices
INGEST
Apache
Kafka
STORE
Apache
Flink
ANALYZE
Apache
Cassandra
ACT
Akka
Ingest millions of
events per second
Distributed & highly
scalable database
Real-time and batch
process data
Visualize data & build
data driven apps
Mesos/ DC/OS
Sensors
Devices
Clients

7. Datacenter

8. Naive Approach
Typical Datacenter
siloed, over-provisioned servers,
low utilization
Industry Average
12-15% utilization
mySQL
microservice
Cassandra
Flink
Kafka

9. © 2017 Mesosphere, Inc. All Rights Reserved. 9

10. Apache Mesos
Typical Datacenter
siloed, over-provisioned servers,
low utilization
Industry Average
12-15% utilization
mySQL
microservice
Cassandra
Flink
Kafka
Mesos
automated schedulers, workload multiplexing
onto the same machines

12. Original creators of Apache
Flink®
Providers of the
dA Platform, a supported
Flink distribution

13. Apache Flink in a Nutshell
Event-driven applications
(event sourcing, CQRS)
Stateful, event-driven,
event-time-aware processing
Batch Processing
(data sets)
Stream Processing / Analytics
(data streams, windows, …)

14. Programming Model
Computation
Computation
Computation
Computation
Source Source
Sink
Sink
Transformation
state
state
state
state

15. What is Flink Good For?

16. Detecting fraud in real time
As fraudsters get better,
need to update models
without downtime
Live 24/7 service
Credit card
transactions
Notifications
and alerts
Evolving fraud
models built by
data scientists
@

17. ▪ Athena X (https://eng.uber.com/athenax/)
▪ Streaming analytics platform
▪ SQL as abstraction layer
Streams from
Hadoop,
Kafka, etc
SQL, thresholds,
actions
Analytics
Alerts
Derived
streams
@

18. ▪ Blink based on Flink
▪ A core system in Alibaba Search
• Machine learning, search, recommendations
• A/B testing of search algorithms
• Online feature updates to boost conversion rate
@

19. @
Complete social network
Implemented using event sourcing and
CQRS (Command Query Responsibility
Segregation)
https://data-artisans.com/blog/drivetribe-cqrs-apache-flink

20. Apache Flink & Apache Mesos

21. Why Apache Mesos?
▪ Mesos offers full functionality to implement fault
tolerant and elastic distributed applications
▪ 30% of survey respondents were running Flink
on Mesos (prior to proper Mesos support,
September 2016)

22. Flink’s Mesos Integration
Apache Flink Framework
Mesos Master
Mesos App Master
Flink Mesos
ResourceManager
JobManager
Mesos Task
TaskManager
Mesos Task
TaskManager
Allocate
Resources
Launch Mesos
tasks
Register
Execute Job

23. Resource Manager Components
▪ Monitors connection to Mesos
Connection Monitor Launch Coordinator
▪ Resource offer processing and task
scheduling
▪ Gathers offers and matches them to
tasks using Fenzo
Task Monitor
Reconciliation
Coordinator
▪ Monitors Mesos tasks
▪ Triggers reconciliation
▪ Makes sure tasks are properly killed
▪ Reconciles tasks view between
ResourceManager and Mesos Master

24. Component Interplay
ResourceManager
Connection
Monitor
Launch
Coordinator
Task MonitorReconciliation
Coordinator
Mesos Master
Resource offers
Launch
tasks
Monitor tasks
Status
messages
Trigger
reconciliation
Status messages
Mesos Task
Reconcile tasks
Start
TaskManagers
Recover tasks
Kill task

26. DC/OS
Datacenter Operating System (DC/OS)
Distributed Systems Kernel (Mesos)
Big Data + Analytics EnginesMicroservices (in containers)
Streaming
Batch
Machine Learning
Analytics
Functions &
Logic
Search
Time Series
SQL / NoSQL
Databases
Modern App Components
Any Infrastructure (Physical, Virtual, Cloud)

27. Demo Time
Generator
▪ Financial data generated by generator
▪ Written to Kafka topics
▪ Kafka topics consumed by Flink
▪ Flink pipeline operates on Kafka data
▪ Results written back into Kafka

28. Conclusion

29. TL;DL
▪ Apache Flink modern stream processor for
real-time processing and event-driven
applications
▪ Apache Flink runs on Mesos using Fenzo
▪ DC/OS offers easy to use Flink package

30. 30
Thank you!
@stsffap
@ApacheFlink
@dataArtisans

31. We are hiring!
data-artisans.com/careers