Presented by Jamie Grier, Data Artisans

1. 1
Jamie Grier
@jamiegrier
data-artisans.com
Building Advanced Streaming Applications:
The Latest and Greatest of Flink and Kafka

2. 2
Original creators of Apache
Flink®
Providers of the
dA Platform, a supported
Flink distribution

3. The Latest Features - Quick Overview
▪ Rescalable State
▪ Async I/O Support
▪ Streaming SQL (Dynamic Tables)
▪ Flexible Deployment Options
▪ Enhanced Security

4. The Latest Features
▪ ProcessFunction API
▪ Queryable State API
▪ Excellent support for advanced applications that are:
▪ Flexible
▪ Stateful
▪ Event Driven
▪ Time Driven

5. Rescalable State
▪ Separates state parallelism from task
parallelism
▪ Enables autoscaling integrations while
maintaining stateful computations
▪ Handled efficiently via key groups

6. Rescalable State
Map Filter Window
State State State
Source Sink
State is partitioned by key

7. Rescalable State
Source Sink
Map Filter Window
State State State
Map Filter Window
State State State
State is partitioned by key

8. Rescalable State
Source Sink
Map Filter Window
State State State
Map Filter Window
State State State
Map Filter Window
State State State
Map Filter Window
State State State
State is partitioned by key

9. Rescalable State
Source Sink
Map Filter Window
State State State
Map Filter Window
State State State
Map Filter Window
State State State
Map Filter Window
State State State
Map Filter Window
State State State
Map Filter Window
State State State
Map Filter Window
State State State
Map Filter Window
State State State
State is partitioned by key

10. Asynchronous I/O Support
▪ Make aynchronous calls to external
services from streaming job
▪ Efficiently keeps configurable number of
asynchronous calls in flight
▪ Correctly handles failure scenarios -
restarts failed async calls, etc

11. Asynchronous I/O Support
Async
I/O
User
Code

12. Asynchronous I/O Support
Little’s Law:
throughput = occupancy / latency

13. Asynchronous I/O Support
a
b
c
d
b
c
d
x
a
a
b
b
a
Sync. I/O Async. I/O
database database
sendRequest(x) x receiveResponse(x) wait

14. Asynchronous I/O Support

15. Asynchronous I/O Support
// create the original stream
val stream: DataStream[String] = ...
// apply the async I/O transformation
val resultStream: DataStream[(String, String)] =
AsyncDataStream.unorderedWait(
input = stream,
asyncFunction = new AsyncDatabaseRequest(),
timeout = 1000,
timeUnit = TimeUnit.MILLISECONDS,
concurrentRequests = 100)

16. Asynchronous I/O Support
class AsyncDatabaseRequest extends AsyncFunction[String, (String, String)] {
override def asyncInvoke(str: String, asyncCollector: AsyncCollector[(String, String)]): Unit = {
// issue the asynchronous request, receive a future for the result
val resultFuture: Future[String] = client.query(str)
// set the callback to be executed once the request by the client is complete
// the callback simply forwards the result to the collector
resultFuture.onSuccess {
case result: String => asyncCollector.collect(Iterable((str, result)));
}
}
}

17. Streaming SQL: Querying Dynamic Tables

18. Streaming SQL: Stream to Table
Append Mode

19. Streaming SQL: Stream to Table
Update Mode

20. Streaming SQL: Queries

21. Streaming SQL: Windowed Queries

22. Streaming SQL: Table to Stream
Redo Stream

23. Streaming SQL: Table to Stream
Undo / Redo Stream

24. Flexible Deployment Options
▪ YARN
▪ Mesos
▪ Docker Swarm
▪ Kubernetes

25. Flexible Deployment Options
▪ DC/OS
▪ Amazon EMR
▪ Google Dataproc

26. Enhanced Security
▪ SSL
▪ Kerberos
▪ Kafka
▪ Zookeeper
▪ Hadoop

27. Advanced Event-Driven Applications
▪ ProcessFunction API
▪ Queryable State API
▪ Excellent support for advanced applications that are:
▪ Flexible
▪ Stateful
▪ Event Driven
▪ Time Driven

28. Example: FlinkTrade
▪ Overall Requirements:
▪ Consume “starting position” and “quote” streams
▪ Process complex, time-oriented, trading rules
▪ Trade out of positions to our advantage if possible
▪ Provide a dashboard of currently held positions to traders and asset managers
▪ Complex Rules:
▪ We only make trades where the Bid Price is above our current Ask Price
▪ When a trade is made we increase our Ask Price — looking to optimize our profits
▪ Positions have a set time-to-live until we try to trade out of them more aggressively
by decreasing the Ask Price over time

29. Example: FlinkTrade
Quotes
Starting
Positions
Trading
Engine
Trades
Positions
SYMBOL SHARES BUY PRICE ASK PRICE LAST
TRADE
PRICE
Profit
AAPL 10,000 140.40 140.50 140.40 $10,921.00
GOOG 20,000 846.81 846.91 846.81 $12,021.00
TWTR 8,000 15.12 15.22 15.12 $4,032.00
● Event Driven Processing
● Complex Trading Rules
● Time Based Logic
Trader Dashboard

30. Example: FlinkTrade
Quotes
Starting
Positions
Trades
Trading
Engine
Positions
Trading
Engine
Positions
Trading
Engine
Positions
Trading
Engine
Positions
Trading
Engine
Positions
Trading
Engine
Positions
Trading
Engine
Positions
Trading
Engine
Positions
Trading
Engine
Positions
Trading
Engine
Positions
Trading
Engine
Positions
Trading
Engine
Positions
SYMBOL SHARES BUY PRICE ASK PRICE LAST
TRADE
PRICE
Profit
AAPL 10,000 140.40 140.50 140.40 $10,921.00
GOOG 20,000 846.81 846.91 846.81 $12,021.00
TWTR 8,000 15.12 15.22 15.12 $4,032.00

31. Example: FlinkTrade
Let’s look at
the code

32. We are hiring!
data-artisans.com/careers
@jamiegrier
@ApacheFlink
@dataArtisans