Chocolate, ice cream and games are perhaps 3 of the most popular universally understood words that can bring joy to anyone between 5-60 years of age! InnoGames is one of the world's leading developers and providers of online games and at InnoGames we not only have all three of those things but in addition we build up a powerful data infrastructure because it's expensive to run your business blind. And being able to evaluate key performance indicators fast to make good decisions and deliver personalized and relevant content to each and every gamer is essential to be successful and it is how a customer becomes a fan. Our data infrastructure mainly consists of a data pipeline that covers the streaming part and a data platform to perform batch processing. The latter is based on the Hadoop ecosystem using technologies such as Hive, Spark, Hue, R and more to give our data scientists a high flexibility. There were several evolutions of the data pipeline, starting with Kestrel and custom streaming applications. Later on we switched the base technologies to Apache Kafka and Apache Storm. Last year we recreated our streaming infrastructure based on Apache Flink which is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. Because having fun is the best way to learn, after a quick introduction to Flink and the Flink ecosystem this talk will focus on real-world use cases and transports the idea of those projects to live examples. This way, the audience will be part of a Flink based experiment to internalize the experience we gained with Flink.

1. LET'S PLAY FLINK
Fun with Streaming in a Gaming Company

2. HI THERE
InnoGames GmbH
Friesenstrasse 13
20097 Hamburg
https://www.innogames.com
Volker Janz
Senior Software Developer
Corporate Systems - Analytics

3. LET‘S GO ON A ROADTRIP

4. IMAGINE
You are driving
with your
family on the
backseat

5. IMAGINE
There is a lot of
traffic, you
have to
concentrate

6. IMAGINE
And now, while driving, you are closing your eyes

8. HOW DO YOU FEEL?

9. WOULD YOU EVER DO THAT?
NOPE

10. METAPHOR
The car is your company,
team or project
The passengers are your
colleagues

11. WOULD YOU EVER DO THAT?
NOPE
IN A METAPHORICAL SENSE

12. YOU HAVE TO PROCESS DATA
ON TIME AS IT HAPPENS

13. BATCH PROCESSING…
…might cause accidents
Because…

14. LIFE
DOESN’T
HAPPEN
IN BATCHES
https://mapr.com/ebooks/streaming-architecture/chapter-01-why-event-streaming.html
© Ellen Friedman, Ted Dunning

15. ICE CREAM
AND
GAMING?

16. SIMILARITIES
MAKE US HAPPY

17. SIMILARITIES
WHICH TASTES BETTER?

18. SIMILARITIES
WHICH TASTES BETTER?

19. SIMILARITIES
THE FIRST IMPRESSION COUNTS
The moment the customer enters
the shop or the player plays his/her
first session is crucial
HALO EFFECT
When one trait of a person or
thing is used to make an overall
judgment of that person or
thing

20. IN ORDER TO MAKE A
POSITIVE IMPACT
A RESPONSE NEEDS TO HAPPEN
QUICKLY

21. TIME-VALUE
OF
INFORMATION

22. REAL-TIME USER REPORTS
TRAFFIC
GAS PRICE
SPEED TRAPS

23. RESPOND TO
LIFE
AS IT
HAPPENS

24. STREAM PROCESSING
STREAMS OF DATA
GPS DATA WEB INTERACTION SENSOR DATA

25. STREAM PROCESSING
PROCESSING DATA IN MOTION

26. STREAM PROCESSING
YOUR
CODE
SOURCE SINKOPERATOR

27. STREAM PROCESSING LAKE

28. STREAM PROCESSING LAKE

29. APACHE FLINK

30. APACHE FLINK
Framework and distributed
process engine for stateful
computations on unbounded
and bounded data streams

31. STREAMS
TIME
WINDOWS

32. EVERYTHING IS A STREAM
UNBOUNDED STREAMS
BOUNDED STREAMS

33. EVERYTHING IS A STREAM
UNBOUNDED STREAMS
BOUNDED STREAMS
AKA BATCH PROCESSING

34. TIME IN STREAMING
EPISODE I EPISODE II EPISODE III EPISODE IV EPISODE V EPISODE VI EPISODE VII EPISODE VIII EPISODE IX
1999 2002 2005 1977 1980 1983 2015 2017 2019
The
Phantom
Menace
Attack of
the Clones
Revenge of
the Sith
A New
Hope
The Empire
Strikes Back
Return of
the Jedi
The Force
Awakens
The Last
Jedi
?
ORDERED BY EVENT TIME
PROCESSING TIME

35. TIME IN STREAMING
EPISODE I EPISODE II EPISODE IIIEPISODE IV EPISODE V EPISODE VI EPISODE VII EPISODE VIII EPISODE IX
1999 2002 20051977 1980 1983 2015 2017 2019
The
Phantom
Menace
Attack of
the Clones
Revenge of
the Sith
A New
Hope
The Empire
Strikes Back
Return of
the Jedi
The Force
Awakens
The Last
Jedi
?
EVENT TIME
ORDERED BY PROCESSING TIME

36. TUMBLING WINDOWS
9 1 3 2 6 8 1 3 9 8 4 5
9 1 3 2 6 8 1 3 9 8 4 5
15 18 26
SENSOR
SUM

37. SLIDING WINDOWS
9 1 3 2 6 8 1 3 9 8 4 5
9 1 3 2 6 8 1 3 9 8 4 5
9 1 3 2 6 8 1 3 9 8 4 5
15 18 26
19 21
SENSOR
SUM

38. BUILDING BLOCKS
DATA SOURCE TRANSFORMATION DATA SINK

39. API
BUILDING BLOCKS
DATA SOURCE TRANSFORMATION DATA SINK

40. BUILDING BLOCKS
SQL / TABLE API
DataStream API
ProcessFunction
APIs
(dynamic tables)
(streams, windows)
(events, state, time)
HIGH LEVEL
ANALYTICS API
STREAM AND BATCH
DATA PROCESSING
STATEFUL EVENT-
DRIVEN APPLICATIONS
CONCISENESS
EXPRESSIVENESS

41. BUILDING BLOCKS
SQL / TABLE API
APIs
(dynamic tables)
HIGH LEVEL
ANALYTICS API
UNBOUNDED STREAM UNBOUNDED STREAM
CONTINUOUS QUERY
DYNAMIC TABLE DYNAMIC TABLE

42. BUILDING BLOCKS
SQL / TABLE API
APIs
(dynamic tables)
HIGH LEVEL
ANALYTICS API
SELECT *
FROM Ticker
MATCH_RECOGNIZE (
PARTITION BY symbol
ORDER BY rowtime
MEASURES C.price AS lastPrice
ONE ROW PER MATCH
AFTER MATCH SKIP PAST LAST ROW
PATTERN (A B* C)
DEFINE
A AS A.price > 10,
B AS B.price < 15,
C AS C.price > 12
)

43. LET‘S HAVE A CLOSER LOOK

44. LET‘S HAVE A CLOSER LOOK
final StreamExecutionEnvironment env = getExecutionEnvironment();
final DataStreamSource<Integer> stream = env.fromElements(1, 2, 3, 4);
stream
.map((MapFunction<Integer, Integer>) i -> i + 2)
.filter((FilterFunction<Integer>) i -> i % 2 == 0)
.print();
env.execute();
DATA SOURCE
TRANSFORMATION
DATA SINK

45. RUNTIME
YOUR FLINK APP
FLINK RUNTIMEDEPLOY

46. RUNTIME

47. RABBIT HOLE

48. RUNTIME
SOURCE MAP PRINTFILTER STREAMING DATAFLOW
(CONDENSED VIEW)
OPERATOR CHAIN OPERATOR OPERATOR
TASK TASK TASK
SOURCE MAP
PRINT
FILTER
OPERATOR CHAIN OPERATOR
OPERATOR
SUBTASK SUBTASK
TASKSOURCE MAP FILTER
OPERATOR CHAIN OPERATOR
SUBTASK SUBTASK
STREAM
PARTITIONS
STREAMING DATAFLOW
(PARALLELIZED VIEW)

49. RUNTIME
SOURCE MAP
PRINT
FILTER
OPERATOR CHAIN OPERATOR
OPERATOR
SUBTASK SUBTASK
TASKSOURCE MAP FILTER
OPERATOR CHAIN OPERATOR
SUBTASK SUBTASK
STREAM
PARTITIONS
STREAMING DATAFLOW
(PARALLELIZED VIEW)
A Flink cluster has a JOB MANAGER and multiple
TASK MANAGERS. Each of those is a JVM.

50. RUNTIME
Each Task Manager can manage MULTIPLE THREADS
executing TASKS / SUBTASKS.
SOURCE MAP
PRINT
FILTER
OPERATOR CHAIN OPERATOR
OPERATOR
THREAD THREAD
THREADSUBTASK SUBTASK
TASKSOURCE MAP FILTER
OPERATOR CHAIN OPERATOR
THREAD THREAD
SUBTASK SUBTASK
STREAM
PARTITIONS
STREAMING DATAFLOW
(PARALLELIZED VIEW)

51. BIG DATA AND STREAMING
AT
INNOGAMES

52. COMPANY SNAPSHOT
More than
400 employees
Founded 2007
in Germany
Headquarter in
Hamburg
+160m EUR revenue
made in 2017
7 live games
>30 language versions
EBITDA margin
of 25%

53. I AM LEGEND
OUR PORTFOLIO Simulation Strategy RPG Browser Multi-device Mobile

54. TEAM

55. TEAM
BUSINESS INTELLIGENCE DATA ENGINEERING DATA SCIENCE OPERATIONS

56. EVENT TRACKING
quest
build
fight invite

57. EVENT TRACKING
1.500.000.000EVENTS PER DAY

58. DATA ARCHITECTURE
DATA PIPELINE DATA PLATFORM
milliseconds, seconds, minutes hours, days, years

59. DATA ARCHITECTURE
SQUIRREL ELEPHANT

60. DATA ARCHITECTURE
EVENTGATEWAY
EVENT BUS
STREAM
PROCESSING
DISTRIBUTED DATA STORE
DISTRIBUTED BATCH PROCESSING
BUSINESS
INTELLIGENCE
EVENT
CLIENT
EVENT
CLIENT
EVENT
CLIENT

61. DATA ARCHITECTURE
EVENT
CLIENT
EVENT
CLIENT
EVENT
CLIENT
EVENTGATEWAY
EVENT BUS
STREAM
PROCESSING
DISTRIBUTED DATA STORE
DISTRIBUTED BATCH PROCESSING
BUSINESS
INTELLIGENCE

63. DATA ARCHITECTURE
EVENT
CLIENT
EVENT
CLIENT
EVENT
CLIENT
EVENTGATEWAY
EVENT BUS
STREAM
PROCESSING
DISTRIBUTED DATA STORE
DISTRIBUTED BATCH PROCESSING
BUSINESS
INTELLIGENCE
STREAM
PROCESSING
STREAMING USE CASES

64. USE CASE EVENT METRICS

65. Metrics.java
stream
.map(streamEvent -> new Tuple2<>(streamEvent.getEventName(), 1))
.keyBy(0)
.timeWindow(Time.minutes(1))
.sum(1)
.addSink(graphiteSink).setParallelism(1).name("event_counts");

66. USE CASE EVENT METRICS

67. USE CASE LOG00 MONITOR

68. KeyedStream<StreamEvent, Integer> stream = events
.filter(event -> Arrays.asList("reg", "login").contains(event.getEventName()))
.keyBy((KeySelector<StreamEvent, Integer>) StreamEvent::getPlayerId);
Log00.java

69. Pattern<StreamEvent, StreamEvent> pattern =
Pattern.<StreamEvent>begin("reg").where(new SimpleCondition<StreamEvent>() {
@Override
public boolean filter(StreamEvent event) {
return event.getEventName().equals("reg");
}
}).followedBy("login").where(new SimpleCondition<StreamEvent>() {
@Override
public boolean filter(StreamEvent event) {
return event.getEventName().equals("login");
}
}).within(Time.seconds(60));
Log00.java

70. PatternStream<StreamEvent> patternStream = CEP.pattern(stream, pattern);
DataStream<Either<PatternResult, PatternResult>> patternResultStream =
patternStream.select(
(p, ts) -> sendTimeoutToGraphite(p, ts),
p -> sendSuccessToGraphite(p)
);
Log00.java

71. USE CASE LOG00 MONITOR

72. USE CASE NEAR TIME CRM(NTCRM)

73. USE CASE NTCRM
EVENT BUS
EVENT
CLIENT
EVENTGATEWAY
PLAYER DATANTCRM
React to events with interstitials in < 10 seconds

74. USE CASE NTCRM
Elvenar has a trading feature that sometimes
causes confusion. With NTCRM we can react to
this and show more details within interstitials
exactly when the player needs it.

75. JUST DO IT
DEMO TIME
Check it out on Github: https://github.com/prenomenon/codetalks-flinkdemo

76. GET IN TOUCH
InnoGames GmbH
Friesenstrasse 13
20097 Hamburg
https://www.innogames.com
Volker Janz
Senior Software Developer
Corporate Systems - Analytics
@prenomenon
Feedback appreciated!

77. LIFE
DOESN’T
HAPPEN
IN BATCHES

78. EAT ICE CREAM
AND
STREAM ON
Great Flink training: http://training.data-artisans.com

79. THAT’S IT FOR NOW…

80. BACKUP / DETAILS
The following slides are not part of my talk but
might give the reader more insights later

81. SQUIRREL TESTS
<dependency
<groupId>org.apache.flink</groupId>
<artifactId>flink-test-utils_2.11</artifactId>
<version>1.6.1</version>
</dependency>

82. WINDOWING
KEYED NON-KEYED
TASK 1
TASK N
SOURCE TASK 1SOURCE
KEY 1
KEY N
ALL DATA

83. CHECKPOINTING
checkpoint
barrier n
checkpoint
barrier n-1
checkpoint n+1 checkpoint n checkpoint n-1
Consistent, incremental snapshots of distributed data stream and operator state
Based on a paper from 1985, inspired by the Chandy-Lamport-Algorithm

84. STATE
SOURCE MAP
DATA SINK
SOURCE MAP
SUM(C,D)
OFFSET
OFFSET
SUM(A,B)
AB
CD

85. STATE
OPERATOR STATE KEYED STATE
Bound only to
an operator
Bound to an
operator and key
PLUGGABLE BACKEND
MULTIPLE PRIMITIVES SUPPORTED
GUARANTEED CONSISTENCY IN CASE OF A FAILURE