Event logs and event streams can be found in software systems of very diverse kinds. Analyzing the wealth of information contained in these logs can serve multiple purposes. In this tutorial, the audience will learn about BeepBeep, a versatile Java library intended to make the processing of event streams (either offline or in realtime) both fun and simple. BeepBeep is the result of more than a decade of research led by a team at Laboratoire d’informatique formelle at Université du Québec à Chicoutimi (Canada). Over the past few years, BeepBeep has been involved in a variety of case studies, and provides built-in support for writing domain-specific languages. Recently, a complete textbook has been published on BeepBeep, testifying to the maturity that the system has acquired.
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Event Stream Processing with BeepBeep 3
1. Event Stream
Event Stream
SYLVAIN HALLÉ
Université du Québec
à Chicoutimi
SYLVAIN HALLÉ
Université du Québec
à Chicoutimi
Processing
Processing
with
with
CRSNG
NSERC
3. What is an event?
$
A unit of data produced by the execu�on or the
observa�on of a "system".
$$$$$
$$$$$
$$$$$
update of a stock price
sensor reading
link
state of a video game
user visi�ng a web page
parcel being delivered
/
move in a chess game
4. ...and an event stream?
An ordered sequence of events of a given type
...
5. ...and an event stream?
An ordered sequence of events of a given type
...
...
6. ...and an event stream?
An ordered sequence of events of a given type
...
...
...
7. ...and an event stream?
An ordered sequence of events of a given type
...
...
...
...
8. ...and an event stream?
An ordered sequence of events of a given type
...
...
...
...
called a log
when stored
for later use
9. ...and an event stream?
An ordered sequence of events of a given type
...
...
...
...
called a log
when stored
for later use
Events can be
composite: ...
11. Event stream processing
A calcula�on done on an event stream
what is the maximum
temperature of the last
100 readings?
12. Event stream processing
A calcula�on done on an event stream
what is the maximum
temperature of the last
100 readings?
who is the employee
who delivered the most
parcels?
13. Event stream processing
A calcula�on done on an event stream
what is the maximum
temperature of the last
100 readings?
who is the employee
who delivered the most
parcels?
is there a user who
accessed page A from
page B?
14. Event stream processing
A calcula�on done on an event stream
what is the maximum
temperature of the last
100 readings?
who is the employee
who delivered the most
parcels?
is there a user who
accessed page A from
page B?
$ has the stock decreased
three days in a row?
15. Event stream processing
A calcula�on done on an event stream
what is the maximum
temperature of the last
100 readings?
who is the employee
who delivered the most
parcels?
is there a user who
accessed page A from
page B?
$ has the stock decreased
three days in a row?
has Mario ever collided
with an enemy during
a jump?
16. Event stream processing
A calcula�on done on an event stream
what is the maximum
temperature of the last
100 readings?
who is the employee
who delivered the most
parcels?
is there a user who
accessed page A from
page B?
$ has the stock decreased
three days in a row?
has Mario ever collided
with an enemy during
a jump?
what is the move
taken most frequently
a�er e2 e4?
17. Event stream processing
A calcula�on done on an event stream
Similar to a database query, but:
o�en concerned with the order of occurrence
result is produced incrementally
what is the maximum
temperature of the last
100 readings?
who is the employee
who delivered the most
parcels?
is there a user who
accessed page A from
page B?
$ has the stock decreased
three days in a row?
has Mario ever collided
with an enemy during
a jump?
what is the move
taken most frequently
a�er e2 e4?
32. Introducing
BeepBeep is an open source library wri�en in Java
and developed at since 2015.
It can be used to perform stream processing in
any language that uses the JVM:
this presenta�on
h�ps://liflab.github.io/beepbeep-3
33. Introducing
A complete textbook is available under an open
access license
research papers
and case studies
downloads
Best Paper
Award
1000
1000
20
20
1
1
h�ps://bit.ly/beepbeep-book
Presses de l'Université du Québec
36. Design principles
Simple elementary building blocks
zero-configura�on
Intui�ve simple mental model
few rules of opera�on
Flexible composi�on
dozens of extensions
37. Design principles
Simple elementary building blocks
zero-configura�on
Intui�ve simple mental model
few rules of opera�on
Flexible composi�on
dozens of extensions
Lightweight low system requirements
core < 300 kb
38. Events
2
3
4
π abc
3 8 a
3 8 a
2 6 c
An event can be any (Java) object:
numbers Booleans strings tuples (key-value)
39. Events
2
3
4
π abc
3 8 a
3 8 a
2 6 c
An event can be any (Java) object:
numbers Booleans strings tuples (key-value)
+
⊇?
...but also...
sets func�ons plots colors
chess
moves
. . .
40. Processor
A processor is a computa�on unit that ingests
input events and produces output events
P
41. Processor
A processor is a computa�on unit that ingests
input events and produces output events
P
processor
42. Processor
A processor is a computa�on unit that ingests
input events and produces output events
P
processor
input pipe
43. Processor
A processor is a computa�on unit that ingests
input events and produces output events
P
processor
input pipe
event
44. Processor
A processor is a computa�on unit that ingests
input events and produces output events
P
processor
input pipe
event
output pipe
46. Arity
Some processors have more than one input pipe...
"2:1 processor"
"1:2 processor"
...others can have more than one output pipe
47. A source is a processor with no input pipe
Sources and sinks
3 1 4
4 1 5 9 2
queue source keyboard
48. A source is a processor with no input pipe
Sources and sinks
3 1 4
4 1 5 9 2
queue source keyboard
A sink is a processor with no output pipe
queue sink print black hole
49. A processor can have pipes with different types:
Pipe types
Each pipe receives / produces events of a specific
type. Color conven�ons are used for common types:
Booleans strings tuples
tables blobs
maps/sets
numbers
50. Basic processors
A set of "core" processors that perform elementary
opera�ons on streams.
n
P
{
f
f n n
Σ
f
n
k
n
fork filter window slice turn into
apply decimate trim cumulate prefix
f
P
56. ApplyFunction
Applies a func�on to events
of the input
f
f
f
f
√
4 9 16 1 9 1
25 2 3 1 3 1
4 5
f a b z
A
3
8
a
A B C
0
5
b
A B C
1
2
z
A B C
57. ApplyFunction
Applies a func�on to events
of the input
f
f
f
f
√
4 9 16 1 9 1
25 2 3 1 3 1
4 5
get a�ribute "A" from a tuple
tuple
f a b z
A
3
8
a
A B C
0
5
b
A B C
1
2
z
A B C
58. ApplyFunction
Applies a func�on to events
of the input
f
f
f
f
√
4 9 16 1 9 1
25 2 3 1 3 1
4 5
f a b z
A
3
8
a
A B C
0
5
b
A B C
1
2
z
A B C
59. ApplyFunction
Applies a func�on to events
of the input
f
f
f
f
√
4 9 16 1 9 1
25 2 3 1 3 1
4 5
f a b z
A
3
8
a
A B C
0
5
b
A B C
1
2
z
A B C
60. ApplyFunction
Applies a func�on to events
of the input
f
f
f
f
√
4 9 16 1 9 1
25 2 3 1 3 1
4 5
f a b z
A
3
8
a
A B C
0
5
b
A B C
1
2
z
A B C
f
4 0 21 3 1 0
-2 ⊤ ⊥ ⊤ ⊤ ⊥
⊤ ⊥
>
1 0
61. ApplyFunction
Applies a func�on to events
of the input
f
f
f
f
√
4 9 16 1 9 1
25 2 3 1 3 1
4 5
f a b z
A
3
8
a
A B C
0
5
b
A B C
1
2
z
A B C
f
4 0 21 3 1 0
-2 ⊤ ⊥ ⊤ ⊤ ⊥
⊤ ⊥
>
1 0
constant 0
the first input pipe
62. ApplyFunction
Applies a func�on to events
of the input
f
f
f
f
√
4 9 16 1 9 1
25 2 3 1 3 1
4 5
f a b z
A
3
8
a
A B C
0
5
b
A B C
1
2
z
A B C
f
4 0 21 3 1 0
-2 ⊤ ⊥ ⊤ ⊤ ⊥
⊤ ⊥
>
1 0
constant 0
the first input pipe
63. ApplyFunction
Applies a func�on to events
of the input
f
f
f
f
√
4 9 16 1 9 1
25 2 3 1 3 1
4 5
f a b z
A
3
8
a
A B C
0
5
b
A B C
1
2
z
A B C
f
4 0 21 3 1 0
-2 ⊤ ⊥ ⊤ ⊤ ⊥
⊤ ⊥
>
1 0
constant 0
the first input pipe
68. ApplyFunction
Applies a func�on to events
of the input
f
f
f
f
+
4 9 16 1 9 1
25
4 0 21 3 1 0
-2 8 9 37 1
23
4 10
f
⊤ ⊥ ⊤ ⊤ ⊥
⊤ ⊥
a b d f g
c e
A B D F G
C E
a B d f G
c E
69. ApplyFunction
Applies a func�on to events
of the input
f
f
f
f
+
4 9 16 1 9 1
25
4 0 21 3 1 0
-2 8 9 37 1
23
4 10
f
⊤ ⊥ ⊤ ⊤ ⊥
⊤ ⊥
a b d f g
c e
A B D F G
C E
a B d f G
c E
if-then-else
70. ApplyFunction
Applies a func�on to events
of the input
f
f
f
f
+
4 9 16 1 9 1
25
4 0 21 3 1 0
-2 8 9 37 1
23
4 10
f
⊤ ⊥ ⊤ ⊤ ⊥
⊤ ⊥
a b d f g
c e
A B D F G
C E
a B d f G
c E
71. ApplyFunction
Applies a func�on to events
of the input
f
f
f
f
+
4 9 16 1 9 1
25
4 0 21 3 1 0
-2 8 9 37 1
23
4 10
f
⊤ ⊥ ⊤ ⊤ ⊥
⊤ ⊥
a b d f g
c e
A B D F G
C E
a B d f G
c E
102. Filter
Discards events from a stream
based on a control signal
⊤ ⊥ ⊤ ⊤ ⊥
⊤ ⊥
a b d f g
c e
a c f
d
The Boolean stream does not need to be related
to the filtered stream
104. Cumulate
Applies func�on to current
input and last output, star�ng with
value
f
n
Σ
f
n
3 1 1 9 2
4 5 3 4 9
8 14
Σ
0
+
23 25
105. Cumulate
Applies func�on to current
input and last output, star�ng with
value
f
n
Σ
f
n
3 1 1 9 2
4 5 3 4 9
8 14
Σ
0
+
23 25
+
106. Cumulate
Applies func�on to current
input and last output, star�ng with
value
f
n
Σ
f
n
3 1 1 9 2
4 5 3 4 9
8 14
Σ
0
+
23 25
+
107. Cumulate
Applies func�on to current
input and last output, star�ng with
value
f
n
Σ
f
n
3 1 1 9 2
4 5 3 4 9
8 14
Σ
0
+
23 25
+
108. Cumulate
Applies func�on to current
input and last output, star�ng with
value
f
n
Σ
f
n
3 1 1 9 2
4 5 3 4 9
8 14
Σ
0
+
23 25
+
109. Cumulate
Applies func�on to current
input and last output, star�ng with
value
f
n
Σ
f
n
3 1 1 9 2
4 5 3 4 9
8 14
Σ
0
+
23 25
sum of all
events
110. Cumulate
Applies func�on to current
input and last output, star�ng with
value
f
n
Σ
f
n
3 1 1 9 2
4 5 3 4 9
8 14
Σ
0
+
23 25
sum of all
events
Σ
-∞
max
3 1 1 9 2
4 5 3 3 4
4 5 9 9
111. Cumulate
Applies func�on to current
input and last output, star�ng with
value
f
n
Σ
f
n
3 1 1 9 2
4 5 3 4 9
8 14
Σ
0
+
23 25
sum of all
events
Σ
-∞
max
3 1 1 9 2
4 5 3 3 4
4 5 9 9
max
112. Cumulate
Applies func�on to current
input and last output, star�ng with
value
f
n
Σ
f
n
3 1 1 9 2
4 5 3 4 9
8 14
Σ
0
+
23 25
sum of all
events
Σ
-∞
max
3 1 1 9 2
4 5 3 3 4
4 5 9 9
max
113. Cumulate
Applies func�on to current
input and last output, star�ng with
value
f
n
Σ
f
n
3 1 1 9 2
4 5 3 4 9
8 14
Σ
0
+
23 25
sum of all
events
Σ
-∞
max
3 1 1 9 2
4 5 3 3 4
4 5 9 9
max
114. Cumulate
Applies func�on to current
input and last output, star�ng with
value
f
n
Σ
f
n
3 1 1 9 2
4 5 3 4 9
8 14
Σ
0
+
23 25
sum of all
events
Σ
-∞
max
3 1 1 9 2
4 5 3 3 4
4 5 9 9
max of all
events
115. Cumulate
Applies func�on to current
input and last output, star�ng with
value
f
n
Σ
f
n
3 1 1 9 2
4 5 3 4 9
8 14
Σ
0
+
23 25
sum of all
events
Σ
-∞
max
3 1 1 9 2
4 5 3 3 4
4 5 9 9
max of all
events
⊥ ⊥ ⊥ ⊥ ⊤
⊥ ⊤ ⊥ ⊥ ⊥ ⊤ ⊤
⊥ ⊤
Σ
⊥
∧
116. Cumulate
Applies func�on to current
input and last output, star�ng with
value
f
n
Σ
f
n
3 1 1 9 2
4 5 3 4 9
8 14
Σ
0
+
23 25
sum of all
events
Σ
-∞
max
3 1 1 9 2
4 5 3 3 4
4 5 9 9
max of all
events
⊥ ⊥ ⊥ ⊥ ⊤
⊥ ⊤ ⊥ ⊥ ⊥ ⊤ ⊤
⊥ ⊤
Σ
⊥
∧
have we seen
⊤ so far?
117. Window
Runs processor on each
window of successive inputs
and returns the last event it produces
n
n
n
P
{
P
118. Window
Runs processor on each
window of successive inputs
and returns the last event it produces
n
n
n
P
{
P
n
3
{
Σ
0
+
3 1 1 9 2
4 5 8 6 10 15 16
119. Window
Runs processor on each
window of successive inputs
and returns the last event it produces
n
n
n
P
{
P
n
3
{
Σ
0
+
3 1 1 9 2
4 5 8 6 10 15 16
3 1 4 3 4 8
120. Window
Runs processor on each
window of successive inputs
and returns the last event it produces
n
n
n
P
{
P
n
3
{
Σ
0
+
3 1 1 9 2
4 5 8 6 10 15 16
1 4 1 1 5 6
121. Window
Runs processor on each
window of successive inputs
and returns the last event it produces
n
n
n
P
{
P
n
3
{
Σ
0
+
3 1 1 9 2
4 5 8 6 10 15 16
4 1 5 4 5 10
122. Window
Runs processor on each
window of successive inputs
and returns the last event it produces
n
n
n
P
{
P
n
3
{
Σ
0
+
3 1 1 9 2
4 5 8 6 10 15 16
1 5 9 1 6 15
123. Window
Runs processor on each
window of successive inputs
and returns the last event it produces
n
n
n
P
{
P
n
3
{
Σ
0
+
3 1 1 9 2
4 5 8 6 10 15 16
sum of 3 successive events
124. Window
Note that can be any
processor (not limited to
aggrega�ons or numerical inputs)
n
n
P
{
P
125. Window
Note that can be any
processor (not limited to
aggrega�ons or numerical inputs)
n
n
P
{
P
n
3
{
Σ
+
n
4
{
Σ
0
⊥
⊥ ⊤ ⊥ ⊥ ⊤
⊥ ⊥
∧
⊤ ⊤ ⊤
⊥
126. Window
Note that can be any
processor (not limited to
aggrega�ons or numerical inputs)
n
n
P
{
P
⊥ ⊤
⊥ ⊤ ⊤ ⊤
⊥ ⊤
n
3
{
Σ
+
n
4
{
Σ
0
⊥
⊥ ⊤ ⊥ ⊥ ⊤
⊥ ⊥
∧
⊤ ⊤ ⊤
⊥
127. Window
Note that can be any
processor (not limited to
aggrega�ons or numerical inputs)
n
n
P
{
P
⊥ ⊥
⊤ ⊥ ⊤ ⊤
⊤ ⊤
n
3
{
Σ
+
n
4
{
Σ
0
⊥
⊥ ⊤ ⊥ ⊥ ⊤
⊥ ⊥
∧
⊤ ⊤ ⊤
⊥
128. Window
Note that can be any
processor (not limited to
aggrega�ons or numerical inputs)
n
n
P
{
P
⊥ ⊥
⊥ ⊥ ⊥ ⊥
⊥ ⊥
n
3
{
Σ
+
n
4
{
Σ
0
⊥
⊥ ⊤ ⊥ ⊥ ⊤
⊥ ⊥
∧
⊤ ⊤ ⊤
⊥
129. Window
Note that can be any
processor (not limited to
aggrega�ons or numerical inputs)
n
n
P
{
P
⊥ ⊤
⊥ ⊥ ⊥ ⊤
⊥ ⊥
n
3
{
Σ
+
n
4
{
Σ
0
⊥
⊥ ⊤ ⊥ ⊥ ⊤
⊥ ⊥
∧
⊤ ⊤ ⊤
⊥
130. Window
Note that can be any
processor (not limited to
aggrega�ons or numerical inputs)
n
n
P
{
P
true if a window of 4 events
contains at least one ⊤
n
3
{
Σ
+
n
4
{
Σ
0
⊥
⊥ ⊤ ⊥ ⊥ ⊤
⊥ ⊥
∧
⊤ ⊤ ⊤
⊥
135. Pipelines
Complex calcula�ons can be achieved through
composi�on
passing the output of a processor
to the input of another
Doing so creates processor chains or "pipelines".
Example:
2
f
x2
136. Pipelines
Complex calcula�ons can be achieved through
composi�on
passing the output of a processor
to the input of another
Doing so creates processor chains or "pipelines".
Example:
2
f
x2
2
1 3 4 5 9
1 25
137. Pipelines
Complex calcula�ons can be achieved through
composi�on
passing the output of a processor
to the input of another
Doing so creates processor chains or "pipelines".
Example:
2
f
x2
2
1 3 4 5 9
1 25
the square of every
other number
140. Pipelines
Any output can be connected to any input, as long
as they have the same type
many types can occur in the same pipeline
141. A few examples
BeepBeep's textbook comes with 150+ illustrated
code examples (in Java) available online
h�ps://liflab.github.io/beepbeep-3-examples
147. f
Σ
0
+
Σ
0
+
÷
1
What does this pipeline calculate?
A few examples
1
3 2 6 3
4
3 6 12 15
2
1 3 4 5
2
3 2 3 3
the running average of the input stream
150. 3
{
f
Σ
0
+
Σ
0
+
÷
1
What does this pipeline calculate?
A few examples
the running average
of the input stream
1
3 2 6 3 2 4 3 3
2 3
32
3 32
3 32
3
151. 3
{
f
Σ
0
+
Σ
0
+
÷
1
What does this pipeline calculate?
A few examples
the running average
of the input stream
1
3 2 6 3 2 4 3 3
2 3
32
3 32
3 32
3
2
3 2
1
3 2
152. 3
{
f
Σ
0
+
Σ
0
+
÷
1
What does this pipeline calculate?
A few examples
the running average
of the input stream
1
3 2 6 3 2 4 3 3
2 3
32
3 32
3 32
3
1
1 3
2
1 6
1
2
153. 3
{
f
Σ
0
+
Σ
0
+
÷
1
What does this pipeline calculate?
A few examples
the running average
of the input stream
1
3 2 6 3 2 4 3 3
2 3
32
3 32
3 32
3
2
6
2 3 4 32
3
154. 3
{
f
Σ
0
+
Σ
0
+
÷
1
What does this pipeline calculate?
A few examples
the running average
of the input stream
1
3 2 6 3 2 4 3 3
2 3
32
3 32
3 32
3
6
3
6 2 32
3
41
2
155. 3
{
f
Σ
0
+
Σ
0
+
÷
1
What does this pipeline calculate?
A few examples
the running average
of the input stream
on each window of
3 successive events
1
3 2 6 3 2 4 3 3
2 3
32
3 32
3 32
3
156. What does this pipeline calculate?
A few examples
f
foo
1
Σ
0
+
157. What does this pipeline calculate?
A few examples
f
foo
1
Σ
0
+
read lines
from a file
158. What does this pipeline calculate?
A few examples
f
foo
1
Σ
0
+
split line into
word list
159. What does this pipeline calculate?
A few examples
f
foo
1
Σ
0
+
output each
word as an event
160. What does this pipeline calculate?
A few examples
f
foo
1
Σ
0
+
create a slice
for each word
161. What does this pipeline calculate?
A few examples
f
foo
1
Σ
0
+
count
162. What does this pipeline calculate?
A few examples
f
foo
1
Σ
0
+
keep last
event
163. What does this pipeline calculate?
A few examples
f
foo
1
Σ
0
+
the number of occurrences
of each word in a file
brown
fox
jumped
lazy
dog
quick
the
over
1
1
5
2
3
2
12
4
186. Push vs. pull mode
The default state of a processor
is to sleep.
ZZ
Z
187. Push vs. pull mode
The default state of a processor
is to sleep.
ZZ
Z
It springs into ac�on when either:
188. Push vs. pull mode
The default state of a processor
is to sleep.
ZZ
Z
It springs into ac�on when either:
we give it a new input event to ingest
!
1
189. Push vs. pull mode
The default state of a processor
is to sleep.
ZZ
Z
It springs into ac�on when either:
! may in turn give events
to downstream
processors: push mode
we give it a new input event to ingest
1
190. Push vs. pull mode
The default state of a processor
is to sleep.
ZZ
Z
It springs into ac�on when either:
! may in turn give events
to downstream
processors: push mode
we give it a new input event to ingest
1
we ask it for new output events to produce
2
?
191. Push vs. pull mode
The default state of a processor
is to sleep.
ZZ
Z
It springs into ac�on when either:
! may in turn give events
to downstream
processors: push mode
we give it a new input event to ingest
1
we ask it for new output events to produce
2
may in turn ask events
to upstream
processors: pull mode
?
237. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
238. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
?
239. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
?
?
240. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
?
3
241. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
?
?
3
242. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
?
2
3
243. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
?
3
2
244. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
5
245. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
246. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
?
247. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
?
?
248. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
?
1
249. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
?
?
1
250. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
?
7
1
251. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
?
1
7
252. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
8
253. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
254. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
?
255. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
?
?
256. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
?
4
257. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
?
?
4
258. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
?
1
4
259. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
?
4
1
260. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
5
261. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
262. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
?
263. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
?
?
264. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
?
1
265. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
?
?
1
266. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
?
8
1
267. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
?
1
8
268. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
9
269. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
270. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
?
271. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
?
?
272. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
?
5
273. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
?
?
5
274. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
?
×
5
275. 3 1 1 9 2
f
+
4 5
2 7 8
1
Synchronous operation
Example in pull mode
5
×
276. ZZ
Z
A processor needs an event in each input pipe
to execute a computa�on step.
Synchronous operation
277. A processor needs an event in each input pipe
to execute a computa�on step.
Synchronous operation
278. A processor needs an event in each input pipe
to execute a computa�on step.
Synchronous operation
When this is the case, it can produce an
output event...
279. A processor needs an event in each input pipe
to execute a computa�on step.
Synchronous operation
When this is the case, it can produce an
output event... or more than one output event...
280. A processor needs an event in each input pipe
to execute a computa�on step.
Synchronous operation
When this is the case, it can produce an
output event... or more than one output event...
or nothing at all
281. ZZ
Z
A processor needs an event in each input pipe
to execute a computa�on step.
Synchronous operation
Otherwise, incoming events are stored into internal
buffers un�l they can be consumed.
282. ZZ
Z
A processor needs an event in each input pipe
to execute a computa�on step.
Synchronous operation
Otherwise, incoming events are stored into internal
buffers un�l they can be consumed.
283. ZZ
Z
A processor needs an event in each input pipe
to execute a computa�on step.
Synchronous operation
Otherwise, incoming events are stored into internal
buffers un�l they can be consumed.
284. ZZ
Z
A processor needs an event in each input pipe
to execute a computa�on step.
Synchronous operation
Otherwise, incoming events are stored into internal
buffers un�l they can be consumed.
285. ZZ
Z
A processor needs an event in each input pipe
to execute a computa�on step.
Synchronous operation
Otherwise, incoming events are stored into internal
buffers un�l they can be consumed.
286. ZZ
Z
A processor needs an event in each input pipe
to execute a computa�on step.
Synchronous operation
Otherwise, incoming events are stored into internal
buffers un�l they can be consumed.
287. A processor needs an event in each input pipe
to execute a computa�on step.
Synchronous operation
Otherwise, incoming events are stored into internal
buffers un�l they can be consumed.
318. BeepBeep programs
Crea�ng a BeepBeep program is made of 3 steps:
instan�a�ng the processors
1
connec�ng them
2
pushing/pulling events
3
Prerequisites
Java 8 (or higher)
Groovy 4.0 (or higher)
to run CLI scripts
325. A simple program
@Grab("io.github.liflab:beepbeep-3:0.11")
import ca.uqac.lif.cep.Connector
import ca.uqac.lif.cep.functions.*
import ca.uqac.lif.cep.io.*
import ca.uqac.lif.cep.tmf.*
import ca.uqac.lif.cep.util.*
dc = new CountDecimate(2)
load BeepBeep
0
2
dc
326. A simple program
@Grab("io.github.liflab:beepbeep-3:0.11")
import ca.uqac.lif.cep.Connector
import ca.uqac.lif.cep.functions.*
import ca.uqac.lif.cep.io.*
import ca.uqac.lif.cep.tmf.*
import ca.uqac.lif.cep.util.*
dc = new CountDecimate(2)
sq = new ApplyFunction(Numbers.square)
load BeepBeep
0
2
f
x2
dc sq
327. A simple program
@Grab("io.github.liflab:beepbeep-3:0.11")
import ca.uqac.lif.cep.Connector
import ca.uqac.lif.cep.functions.*
import ca.uqac.lif.cep.io.*
import ca.uqac.lif.cep.tmf.*
import ca.uqac.lif.cep.util.*
dc = new CountDecimate(2)
sq = new ApplyFunction(Numbers.square)
pr = new Print()
load BeepBeep
0
2
f
x2
dc sq pr
328. A simple program
@Grab("io.github.liflab:beepbeep-3:0.11")
import ca.uqac.lif.cep.Connector
import ca.uqac.lif.cep.functions.*
import ca.uqac.lif.cep.io.*
import ca.uqac.lif.cep.tmf.*
import ca.uqac.lif.cep.util.*
dc = new CountDecimate(2)
sq = new ApplyFunction(Numbers.square)
pr = new Print()
load BeepBeep
instan�ate
processors
1
0
2
f
x2
dc sq pr
329. A simple program
@Grab("io.github.liflab:beepbeep-3:0.11")
import ca.uqac.lif.cep.Connector
import ca.uqac.lif.cep.functions.*
import ca.uqac.lif.cep.io.*
import ca.uqac.lif.cep.tmf.*
import ca.uqac.lif.cep.util.*
dc = new CountDecimate(2)
sq = new ApplyFunction(Numbers.square)
pr = new Print()
Connector.connect(dc, sq)
load BeepBeep
instan�ate
processors
1
0
2
f
x2
dc sq pr
330. A simple program
@Grab("io.github.liflab:beepbeep-3:0.11")
import ca.uqac.lif.cep.Connector
import ca.uqac.lif.cep.functions.*
import ca.uqac.lif.cep.io.*
import ca.uqac.lif.cep.tmf.*
import ca.uqac.lif.cep.util.*
dc = new CountDecimate(2)
sq = new ApplyFunction(Numbers.square)
pr = new Print()
Connector.connect(dc, sq)
Connector.connect(sq, pr)
load BeepBeep
instan�ate
processors
1
0
2
f
x2
dc sq pr
331. A simple program
@Grab("io.github.liflab:beepbeep-3:0.11")
import ca.uqac.lif.cep.Connector
import ca.uqac.lif.cep.functions.*
import ca.uqac.lif.cep.io.*
import ca.uqac.lif.cep.tmf.*
import ca.uqac.lif.cep.util.*
dc = new CountDecimate(2)
sq = new ApplyFunction(Numbers.square)
pr = new Print()
Connector.connect(dc, sq)
Connector.connect(sq, pr)
load BeepBeep
instan�ate
processors
connect
1
2
0
2
f
x2
dc sq pr
332. A simple program
squares.gvy
@Grab("io.github.liflab:beepbeep-3:0.11")
import ca.uqac.lif.cep.Connector
import ca.uqac.lif.cep.functions.*
import ca.uqac.lif.cep.io.*
import ca.uqac.lif.cep.tmf.*
import ca.uqac.lif.cep.util.*
dc = new CountDecimate(2)
sq = new ApplyFunction(Numbers.square)
pr = new Print()
Connector.connect(dc, sq)
Connector.connect(sq, pr)
load BeepBeep
instan�ate
processors
connect
1
2
0
2
f
x2
dc sq pr
333. Pipelines can be created using few lines of code
(1 per box / pipe)
A simple program
No installa�on required (@Grab)
No compila�on required: programs can be saved
to a text file and run immediately at the
command line*
*When using Groovy
334. This program does nothing ( ). It is missing
part : pushing / pulling events.
A simple program
3
ZZ
Z
335. This program does nothing ( ). It is missing
part : pushing / pulling events.
A simple program
3
ZZ
Z
Two objects can be used:
336. This program does nothing ( ). It is missing
part : pushing / pulling events.
A simple program
3
ZZ
Z
Two objects can be used:
Two objects can be used:
Pushable
!
reference to an
input pipe
used to push events
Pushable
!
337. This program does nothing ( ). It is missing
part : pushing / pulling events.
A simple program
3
ZZ
Z
Two objects can be used:
Two objects can be used:
Pushable
!
reference to an
input pipe
used to push events
Pushable
! vs.
Pullable
?
reference to an
output pipe
used to pull events
vs.
Pullable
?
339. A simple program (cont'd.)
2
f
x2
dc sq pr
...
p = dc.getPushableInput()
p
340. A simple program (cont'd.)
2
f
x2
dc sq pr
...
p = dc.getPushableInput()
p
2
1 3 4 5
p.push(1)
p.push(2)
p.push(3)
p.push(4)
p.push(5)
341. A simple program (cont'd.)
2
f
x2
dc sq pr
...
p = dc.getPushableInput()
p
2
1 3 4 5
p.push(1)
p.push(2)
p.push(3)
p.push(4)
p.push(5)
push events
3
346. 0
When a processor has
more than one input
or output pipe, the index
must be specified
1
0
1
0 0
@Grab("io.github.liflab:beepbeep-3:0.11")
import ca.uqac.lif.cep.Connector
import ca.uqac.lif.cep.functions.*
import ca.uqac.lif.cep.tmf.*
import ca.uqac.lif.cep.util.*
A = new Fork()
B = new CountDecimate(3)
C = new ApplyFunction(Numbers.addition)
Connector.connect(A, 0, B, 0)
Connector.connect(B, 0, C, 0)
Connector.connect(A, 1, C, 1)
Another example
addition.gvy
3
f
+
A
B
C
350. Syntactical shortcuts
Pipelines wri�en in Groovy can be shortened.
Connector.connect(A, B) A | B
Connector.connect(A, x, B, y) A[x] | B[y]
The | connec�ve returns the last processor of the
chain
351. Syntactical shortcuts
Pipelines wri�en in Groovy can be shortened.
Connector.connect(A, B) A | B
Connector.connect(A, x, B, y) A[x] | B[y]
The | connec�ve returns the last processor of the
chain
Connector.connect(A, B)
Connector.connect(B, C)
p = C
p = A | B | C
352. Syntactical shortcuts
Pipelines wri�en in Groovy can be shortened.
Connector.connect(A, B) A | B
Connector.connect(A, x, B, y) A[x] | B[y]
The | connec�ve returns the last processor of the
chain
Connector.connect(A, B)
Connector.connect(B, C)
p = C
p = A | B | C
Pushable objects do not need to be explicitly
obtained:
353. Syntactical shortcuts
Pipelines wri�en in Groovy can be shortened.
Connector.connect(A, B) A | B
Connector.connect(A, x, B, y) A[x] | B[y]
The | connec�ve returns the last processor of the
chain
Connector.connect(A, B)
Connector.connect(B, C)
p = C
p = A | B | C
Pushable objects do not need to be explicitly
obtained:
p = proc.getPushableInput()
p.push(x)
proc << x
356. Other processors
BeepBeep provides many other processors and
func�ons. Some examples:
SELECT * FROM foo
http://...
↑
freeze an
event
k
repeat each
event
k
insert an
event
read from an
HTTP request
read from a
DB query
serialize an
event
357. Other processors
BeepBeep provides many other processors and
func�ons. Some examples:
P
1 P
2 P
3
, ,
TANK
k
Moore machine detect peaks
pack events
tank pump splice sources
372. A program running on a Hadoop cluster processes
the 1.7 GB file in 26 minutes.*
The right tool... (cont'd)
h�ps://tomhayden3.com/2013/12/27/chess-mr-job
*
373. A program running on a Hadoop cluster processes
the 1.7 GB file in 26 minutes.*
The right tool... (cont'd)
h�ps://tomhayden3.com/2013/12/27/chess-mr-job
*
The BeepBeep pipeline on a desktop computer
takes...
374. A program running on a Hadoop cluster processes
the 1.7 GB file in 26 minutes.*
The right tool... (cont'd)
h�ps://tomhayden3.com/2013/12/27/chess-mr-job
*
The BeepBeep pipeline on a desktop computer
takes... 35 seconds.
375. A program running on a Hadoop cluster processes
the 1.7 GB file in 26 minutes.*
The right tool... (cont'd)
h�ps://tomhayden3.com/2013/12/27/chess-mr-job
*
The BeepBeep pipeline on a desktop computer
takes... 35 seconds.
376. A program running on a Hadoop cluster processes
the 1.7 GB file in 26 minutes.*
The right tool... (cont'd)
h�ps://tomhayden3.com/2013/12/27/chess-mr-job
*
The BeepBeep pipeline on a desktop computer
takes... 35 seconds.
377. A program running on a Hadoop cluster processes
the 1.7 GB file in 26 minutes.*
The right tool... (cont'd)
h�ps://tomhayden3.com/2013/12/27/chess-mr-job
*
The BeepBeep pipeline on a desktop computer
takes... 35 seconds.
378. Robots
Industrial robot moving on a shop floor, producing
an event stream of its current state
x y θ x y θ
, x y θ
, ...
θ
(x,y)
379. Robots
Industrial robot moving on a shop floor, producing
an event stream of its current state
x y θ x y θ
, x y θ
, ...
Goal: detect skidding
situa�ons
"large" difference between
orienta�on and mo�on
400. n �mes in a row
outputs if Δθ ≥ k
⊤
n
{
Σ
∧
⊤
Robots
A possible pipeline (final version):
1
f
−
f
x
1
f
−
f
y
f
atan2
f
θ
f
−
| |
f
≥
1 k
401. Conclusion
Event streams can be found in a variety of
situa�ons
Processing them in real �me does not (always)
require a complex/distributed setup
The BeepBeep stream processing engine can
perform a wide range of calcula�ons on streams
403. Pale�e
Set of processors and func�ons,
centered around a par�cular use case
There are dozens of pale�es that
extend the set of core processors for
various scenarios