Apache Beam lets you write data pipelines over unbounded, out-of-order, global-scale data that are portable across diverse backends including Apache Flink, Apache Apex, Apache Spark, and Google Cloud Dataflow. But not all use cases are pipelines of simple "map" and "combine" operations. Beam's new State API adds scalability and consistency to fine-grained stateful processing, all with Beam's usual portability. Examples of new use cases unlocked include: * Microservice-like streaming applications * Aggregations that aren't natural/efficient as an associative combiner * Fine control over retrieval and storage of intermediate values during aggregation * Output based on customized conditions, such as limiting to only "significant" changes in a learned model (resulting in potentially large cost savings in subsequent processing) This talk will introduce the new state and timer features in Beam and show how to use them to express common real-world use cases in a backend-agnostic manner.

1. Portable stateful big data processing
in Apache Beam
Kenneth Knowles
Apache Beam PMC
Software Engineer @ Google
klk@google.com / @KennKnowles
https://s.apache.org/ffsf-2017-beam-state
Flink Forward San Francisco 2017

2. Agenda
1. What is Apache Beam?
2. State
3. Timers
4. Example & Little Demo

3. What is
Apache Beam?

4. TL;DR
4
(Flink draws it more like this)

5. 20142004 2006 2008 2010 2012 20162005 2007 2009 2013 20152011
MapReduce
(paper)
Apache
Hadoop
Dataflow Model
(paper)
MillWheel
(paper)
Heron
Apache
Spark
Apache
Storm
Apache
Gearpump
(incubating)
Apache
Apex
Apache
Flink
Cloud
Dataflow
FlumeJava
(paper)
Apache Beam
DAGs, DAGs, DAGs
Apache
Samza

6. The Beam Vision
Sum Per Key
6
input.apply(
Sum.integersPerKey())
Java
input | Sum.PerKey()
Python
⋮
Apache Flink
local, on-prem,
cloud
Apache Spark
local, on-prem,
cloud
Cloud Dataflow:
fully managed
⋮
Apache Apex
local, on-prem,
cloud
Apache
Gearpump
(incubating)

7. The Beam Vision
KafkaIO
7
input | KakaIO.read()
Python
⋮
class KafkaIO extends
UnboundedSource { … }
Java
Apache Flink
local, on-prem,
cloud
Apache Spark
local, on-prem,
cloud
Cloud Dataflow:
fully managed
⋮
Apache Apex
local, on-prem,
cloud
Apache
Gearpump
(incubating)

8. The Beam Model
Pipeline
8
PTransform
PCollection
(bounded or unbounded)

9. The Beam Model
What are you computing? (read, map, reduce)
Where in event time? (event time windowing)
9
When in processing time are results produced? (triggers)
How do refinements relate? (accumulation mode)

10. What are you computing?
10
Read
Parallel connectors to
external systems
ParDo
Per element
"Map"
Grouping
Group by key,
Combine per key,
"Reduce"
Composite
Encapsulated
subgraph

11. State

12. What are you computing?
12
Read
Parallel connectors to
external systems
ParDo
Per element
"Map"
Grouping
Group by key,
Combine per key,
"Reduce"
Composite
Encapsulated
subgraph

13. State for ParDo
13
ParDo(DoFn)

14. "Example"
14
Per Key Quantiles
ParDo.of(new DoFn<...>() {
// declare some state
@ProcessElement
public void process(...) {
// update quantiles
// output if needed
}
})

15. Partitioned
15
Quantiles Quantiles Quantiles

16. Parallelism!
16
Quantiles
DoFn
Quantiles
DoFn
Quantiles
DoFn
new DoFn<Foo, Quantiles<Foo>>() {
@StateId("quantiles")
private final StateSpec<...>
quantilesState = StateSpecs.combining();
…
}

17. Windowed
17
Quantiles
DoFn
Quantiles
DoFn
Quantiles
DoFn
Window into Fixed windows of one hour
Expected result: Quantiles for each hour

18. Windowed
18
Quantiles
DoFn
Quantiles
DoFn
Quantiles
DoFn
Window into windows of 30 min sliding by 10 min
Expected result: Quantiles for 30 minutes sliding by 10 min

19. <k, w>1
<k, w>2
<k, w>3
...
"x" 3 7 15
"y" "fizz" "7" "fizzbuzz"
...
State is per key and window
Bonus: automatically garbage collected when a window expires
(vs manual clearing of keyed state)

20. Windowed
20
Quantiles
DoFn
Quantiles
DoFn
Quantiles
DoFn
Window into Fixed windows of one hour
Expected result: Quantiles for each hour

21. What about Combine?
21
Quantiles
Combiner
Window into Fixed windows of one hour
Expected result: Quantiles for each hour
Quantiles
Combiner
Quantiles
Combiner

22. Combine vs State (naive conceptualization)
22
Window hourly
Expected result:
Quantiles for each hour
Quantiles
Combiner
Window hourly
Quantiles
DoFn
Expected result:
Quantiles for each hour

23. Combine vs State (likely execution plan)
Quantiles
Combiner
Quantiles
Combiner
Quantiles
Combiner
Shuffle
Accumulators
Quantiles
DoFn
Shuffle Elements
associative
commutative
single-output
enables optimizations
(engine is in control)
non-associative*
non-commutative*
multi-output
side outputs
(user is in control)

24. Combine vs State
Quantiles
Combiner
Quantiles
DoFn
output governed by trigger
(data/computation unaware) "output only when there's an
interesting change"
(data/computation aware)

25. Example: Arbitrary indices
D,0
A
Assign indices
B
C
D
E
C,1
A,2
B,3
E,4
non-associative
non-commutative
non-deterministic
and totally fine!

26. Kinds of state
● Value - just a mutable cell for a value
● Bag - supports "blind writes"
● Combining - has a CombineFn built in; can support blind writes and lazy
accumulation
● Set - membership checking
● Map - lookups and partial writes

27. Timers

28. Timers for ParDo
28
Stateful ParDo
new DoFn<...>() {
@TimerId("timeout")
private final TimerSpec timeoutTimer =
TimerSpecs.timer(TimeDomain.PROCESSING_TIME);
@OnTimer("timout")
public void timeout(...) {
// access state, set new timers, output
}
…
}

29. Timers in processing time
"call me back in 5
seconds"
output based only
on incoming
element
output return
value of batched
RPC
buffer request
batched RPC
On Timer
On Element

30. Timers in event time
"call me back when
the watermark hits
the end of the
window"
output
speculative result
immediately
output replacement
value only if
changed
store speculative result
On Timer
On Element

31. ● Per-key arbitrary numbering
● Output only when result changes
● Tighter "side input" management for slowly changing dimension
● Streaming join-matrix / join-biclique
● Fine-grained combine aggregation and output control
● Per-key "workflows" like user sign up flow w/ expiration
● Low-latency deduplication (let the first through, squash the rest)
More example uses for state & timers

32. Performance considerations (cross-runner)
32
● Shuffle to colocate keys
● Linear processing of elements for key+window
● Window merging
● Storage of state and timers
● GC of state

33. Demo
33

34. Summary
State and Timers in Beam...
● … unlock new uses cases
● … they "just work" with event time windowing
● … are portable across runners (implementation ongoing)

35. Thank you for listening!
This talk:
● Me - @KennKnowles
● These Slides - https://s.apache.org/ffsf-2017-beam-state
Go Deeper
● Design doc - https://s.apache.org/beam-state
● Blog post - https://beam.apache.org/blog/2017/02/13/stateful-processing.html
Join the Beam community:
● User discussions - user@beam.apache.org
● Development discussions - dev@beam.apache.org
● Follow @ApacheBeam on Twitter
You can contribute to Beam + Flink
● New types of state
● Easy launch of Beam job on Flink-on-YARN
● Integration tests at scale
● Fit and finish: polish, polish, polish!
● … and lots more!
https://beam.apache.org