Kafka Streaming Windowed Count

@loicmdivad
SAK Meetup
Kafka Tutorials
Window Final Result
Stockholm Apache Kafka Meetup

@loicmdivad
SAK Meetup
Photo by Kelly Sikkema
on Unsplash
2

@loicmdivad
SAK Meetup 3
Loïc DIVAD
Software Engineer at Spotify
@loicmdivad

@loicmdivad
SAK Meetup 4
Loïc DIVAD
Software Engineer at Spotify
@loicmdivad

@loicmdivad
SAK Meetup 5
@loicmdivad
SAK Meetup

@loicmdivad
SAK Meetup @loicmdivad
@loicmdivad
SAK Meetup
Collect data over time
7
Photo by Aron Visuals
on Unsplash

@loicmdivad
SAK Meetup 8
1. The Use Case
2. The Implementation
3. The what’s next for 2021?
on Unsplash

@loicmdivad
SAK Meetup 9
1. The Use Case
a. Context and motivation
b. Emit a ﬁnal result from a time window
c. Kafka Tutorials presentation
a. Develop
b. Test
c. Deploy
a. New year resolution at massive scale!
on Unsplash

@loicmdivad
SAK Meetup
Chapter I: The Use Case
10

@loicmdivad
SAK Meetup
Window Final Result for real-time alerting
Let's be an automated drink order service:
- Captors are sending pressure alerts
11

@loicmdivad
SAK Meetup
- The pressure can be high
- But these are just warnings
- However, we don’t want them to be too
frequents
- It could indicate a damaged captors
12

@loicmdivad
SAK Meetup
frequents
- Instead of maintaining the overall values,
we’d like to emit the ﬁnal count
- This can be done by aggregating the
unbounded stream of pressure alerts
13

@loicmdivad
SAK Meetup
frequents
- Instead of maintaining the overall values,
we’d like to emit the ﬁnal count
- This can be done by aggregating the
unbounded stream of pressure alerts
14

@loicmdivad
SAK Meetup
Data Pipeline
15

@loicmdivad
SAK Meetup
Alerting Tumbling Windows
16
101
102

@loicmdivad
SAK Meetup
17
101
102

@loicmdivad
SAK Meetup
18
⚠☢
⚠☢
101
102

@loicmdivad
SAK Meetup
What if the captor were
laggy ??
19
In this exercise we will probably
face out-of-order elements and
late data arrivals.
We have to take this in account!
The solution could be to add a
extra period when late events can
still join there corresponding
aggregate and trigger a event.
It’s called a grace period!

@loicmdivad
SAK Meetup
102
Alerting Tumbling Windows (with grace period)
20
101

@loicmdivad
SAK Meetup
102
101
Alerting Tumbling Windows (with grace period)
21
⚠☢

@loicmdivad
SAK Meetup
The streaming app solving this has to:
● Continually consume and process
events
● Aggregate events over time
● Correctly integrate late events
● Emit an event at the closing of each
window
● Be easy to test and deploy
22

@loicmdivad
SAK Meetup
Kafka Tutorials
Kafka Tutorials is a collection of
common event streaming use
cases, with each tutorial
featuring an example scenario
and several complete code
solutions. It’s the fastest way to
learn how to use Kafka with
conﬁdence.
-- Michael Drogalis
24
from:
https://www.conﬂuent.io/blog/announci
ng-apache-kafka-tutorials/

@loicmdivad
SAK Meetup
WFR applied to
real-time alerting
● Step N°1: ﬁnd an issue you’d
like to tackle
25

@loicmdivad
SAK Meetup
WFR applied to
real-time alerting
like to tackle
● Step N°2: submit a PR
referencing the previous issue
26

@loicmdivad
SAK Meetup
WFR applied to
real-time alerting
like to tackle
● Step N°2: submit a PR
referencing the previous issue
● Step N°3: Negotiate during the
review to get your dirty code
accepted!
27

@loicmdivad
SAK Meetup
My ﬁrst contribution
28

@loicmdivad
SAK Meetup 29
1. The Use Case
a. Context and motivation
b. Emit a ﬁnal result from a time window
c. Kafka Tutorials presentation
a. Develop
b. Test
c. Deploy
a. New year resolution at massive scale!
on Unsplash

@loicmdivad
SAK Meetup
Chapter II: The Implementation
30

@loicmdivad
SAK Meetup
Develop
31

@loicmdivad
SAK Meetup 32
Project Structure

@loicmdivad
SAK Meetup
Dev environment and Docker Compose
33

@loicmdivad
SAK Meetup
Application Build
34
Plugins:
java, application, jib, shadow, avro
dependencies:
- kafka-streams
- kafka-streams-avro-serde
- kafka-streams-test-utils
- junit
source / target compatibility: 1.8

@loicmdivad
SAK Meetup
The PressureAlert Schema
35

@loicmdivad
SAK Meetup 36
1. Simple
2. Expressif
3. Declarative
The Kafka Streams DSL is built on top of the
Streams Processor API. It is the recommended for
most users… Most data processing operations can
be expressed in just a few lines of DSL code.

@loicmdivad
SAK Meetup
Event Time vs Processing Time
37

@loicmdivad
SAK Meetup 38
value
key
headers
Kafka Message in
value
key
headers
Kafka Message out
Null
Timestamp

@loicmdivad
SAK Meetup 39
…

@loicmdivad
SAK Meetup 40

@loicmdivad
SAK Meetup 41
…

@loicmdivad
SAK Meetup 42
Kafka Streams Topology

@loicmdivad
SAK Meetup 43

@loicmdivad
SAK Meetup 44

@loicmdivad
SAK Meetup 45

@loicmdivad
SAK Meetup 46
…

@loicmdivad
SAK Meetup 47

@loicmdivad
SAK Meetup 48
Suppress operator

@loicmdivad
SAK Meetup 49
Time window specification
1.
● Return a window definition with the
given window size
2.
● specifies by how much a window
moves forward relative to the
previous one
3.
● Reject late events that arrive after
the specified delay
of
advanceBy grace

@loicmdivad
SAK Meetup 50

@loicmdivad
SAK Meetup 51
Suppress Operator
Aggregated state
Alert Event

@loicmdivad
SAK Meetup 52
Suppress Operator
Aggregated state Suppressed state
Alert Event

@loicmdivad
SAK Meetup 53

@loicmdivad
SAK Meetup
Streaming Topology print
54

@loicmdivad
SAK Meetup 55
MERCI !(Thank you)
Photo by Loïc DIVAD
on Unsplash

@loicmdivad
SAK Meetup
Test
56

@loicmdivad
SAK Meetup
TopologyTestDriver
57
Processing Nodes
Input Nodes
Output Nodes

@loicmdivad
SAK Meetup
TopologyTestDriver
58
Processing Nodes
Input Nodes
Output Nodes
✂
✂ ✂

@loicmdivad
SAK Meetup
TopologyTestDriver
59
Processing Nodes
Input Nodes
Output Nodes

@loicmdivad
SAK Meetup
Topology should group over date-time windows
60
�� Kafka Streams
Topology
101/[T1-T2] N1
101/[T3-T4] N2
101/[T4-T5] N3
⚠ ⚠ ⚠

@loicmdivad
SAK Meetup 61

@loicmdivad
SAK Meetup 62
…
…

@loicmdivad
SAK Meetup
Topology should group by Id
63
��
Kafka Streams
Topology
101 / [T1-T2] N1
102 / [T1-T2] N2
103 / [T1-T2] N3
⚠ ⚠ ⚠
��

@loicmdivad
@loicmdivad
SAK Meetup 64

@loicmdivad
SAK Meetup 65
…
…

@loicmdivad
SAK Meetup
PDE should extract Timestamp
66
�� Kafka Streams
Topology
101/[T1-T2] TS1
101/[T3-T4] TS2
101/[T4-T5] TS3
⚠ ⚠ ⚠

@loicmdivad
SAK Meetup 67

@loicmdivad
SAK Meetup 68
…

@loicmdivad
SAK Meetup
Deploy
69

@loicmdivad
SAK Meetup
Package
as you are
70
● Build a jar with all the
dependencies and deploy
it on machine having a
JVM and where all the the
conﬁgs ﬁles are
automatically provided
Or
● Build a container, deploy
the container

@loicmdivad
SAK Meetup
Chapter III: The what’s next for 2021?
71

Kafka Streaming Windowed Count

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Kafka Streaming Windowed Count

Similaire à Kafka Streaming Windowed Count (20)

Plus de confluent

Plus de confluent (20)

Dernier

Dernier (20)

Kafka Streaming Windowed Count