Java 8 introduced the Stream API as a modern, functional, and very powerful tool for processing collections of data. One of the main benefits of the Stream API is that it hides the details of iteration over the underlying data set, allowing for parallel processing within a single JVM, using a fork/join framework. I will talk about a Stream API implementation that enables parallel processing across many machines and many JVMs. With an explanation of internals of the implementation, I will give an introduction to the general design behind stream processing using DAG (directed acyclic graph) engines and how an actor-based implementation can provide in-memory performance while still leveraging industry-wide known frameworks as Java Streams API. https://www.jfokus.se/jfokus/talks.jsp#RidingtheJetStreams

1. Riding
Jet Streams
@gAmUssA @hazelcast #jfokus #hazelcastjet
http://bit.ly/streams_jfokus2017

2. @gAmUssA @hazelcast #jfokus #hazelcastjet
Solutions Architect @Hazelcast
Developer Advocate @Hazelcast
@gamussa in internetz
Please, follow me on Twitter
I’m very interesting ©
> whoami

3. @gAmUssA @hazelcast #jfokus #hazelcastjet
Agenda
Quick refresh on Java 8 Streams
Distribute and Conquer
Distributed Data
Distributed Streams
How we did all this

4. @gAmUssA @hazelcast #jfokus #hazelcastjet
Example: Word Count
Map<Integer, String> where
keys are line numbers and values are
lines.
Find how many times each word occurs

5. @gAmUssA @hazelcast #jfokus #hazelcastjet
What needs to be done?
Iterate through all the lines
Split the line into words
Update running total of counts with new
word

6. fillMapWithData("war_and_peace_eng.txt", source);
for (String line : source.values()) {
for (String word : PATTERN.split(line)) {
if (word.length() >= 5)
count.compute(
cleanWord(word).toLowerCase(),
(w, c) -> c == null ? 1 : c + 1
);
}
}
System.out.println(count.get("andrew"));
Iterate through all the lines

7. fillMapWithData("war_and_peace_eng.txt", source);
for (String line : source.values()) {
for (String word : PATTERN.split(line)) {
if (word.length() >= 5)
count.compute(
cleanWord(word).toLowerCase(),
(w, c) -> c == null ? 1 : c + 1
);
}
}
System.out.println(count.get("andrew"));
Split the line into words

8. fillMapWithData("war_and_peace_eng.txt", source);
for (String line : source.values()) {
for (String word : PATTERN.split(line)) {
if (word.length() >= 5)
count.compute(
cleanWord(word).toLowerCase(),
(w, c) -> c == null ? 1 : c + 1
);
}
}
System.out.println(count.get("andrew"));
Update running total of counts with new word

9. fillMapWithData("war_and_peace_eng.txt", source);
for (String line : source.values()) {
for (String word : PATTERN.split(line)) {
if (word.length() >= 5)
count.compute(
cleanWord(word).toLowerCase(),
(w, c) -> c == null ? 1 : c + 1
);
}
}
System.out.println(count.get("andrew"));
Print the result

10. java.util.stream

11. @gAmUssA @hazelcast #jfokus #hazelcastjet
Java 8 Streams…
An abstraction represents a sequence of
elements
Is not a data structure
Convey elements from a source through a
pipeline of operations
Operation doesn’t modify a source

12. @gAmUssA @hazelcast #jfokus #hazelcastjet
Why I should care about Stream API?
You’re Java developer

13. What does regular Java developer think
about Scala?
advanced

14. @gAmUssA @hazelcast #jfokus #hazelcastjet
Why I should care about Stream API?
You’re Java developer
Many Java developers know Java
It’s all about data processing

15. @gAmUssA @hazelcast #jfokus #hazelcastjet
java.util.stream
map(), flatMap(), filter()
reduce(), collect()
sorted(), distinct()
Intermediate operation
Terminal operation
Stateful Intermediate (Blocking) operation

19. @gAmUssA @hazelcast #jfokus #hazelcastjet
Why would one need a cluster?
One does not simply fit all Big Data in
one machine

20. @gAmUssA @hazelcast #jfokus #hazelcastjet
Problem
Data doesn’t fit just one machine

21. @gAmUssA @hazelcast #jfokus #hazelcastjet
Why would one need a cluster?
One does not simply put all Big Data
in one machine
Data is too important to have it only
one machine

24. Replication on Sharding?
http://book.mixu.net/distsys/single-page.html

25. @gAmUssA @hazelcast #jfokus #hazelcastjet
Another Requirements
Easy to use
Simple API
Embeddable
Cloud Native

26. @gAmUssA @hazelcast #jfokus #hazelcastjet
What’s Hazelcast IMDG?
In-memory Data Grid
Apache v2 Licensed
Distributed
Caches (IMap, JCache)
Java Collections (IList, ISet, IQueue)
Messaging (Topic, RingBuffer)
Computation (ExecutorService, M-R)

27. 1 900 StarsOn GitHub
100%Open Source
134 contributors

31. @gAmUssA @hazelcast #jfokus #hazelcastjet
Green
Primary
Green
Backup
Green
Shard

33. @gAmUssA @hazelcast #jfokus #hazelcastjet
What’s the problem?
Use IMap.values().stream() ?
Or IMap.entrySet().stream() ?
3
3

34. @gAmUssA @hazelcast #jfokus #hazelcastjet
Problem
Data doesn’t fit just one machine

37. @gAmUssA @hazelcast #jfokus #hazelcastjet
EASY (actually, not)!
Implement serializable version of the
interfaces
Introducing DistributedStream
3
7

38. 3
8

40. 4
0
Jet Streams

41. jet.hazelcast.org

42. @gAmUssA @hazelcast #jfokus #hazelcastjet
What’s Hazelcast Jet?
General purpose distributed data processing
framework
Based on Direct Acyclic Graph to model data
flow
Built on top of Hazelcast IMDG
Comparable to Apache Spark or Apache
Flink
4
2

44. @gAmUssA @hazelcast #jfokus #hazelcastjet
DAG
vertex vertex
vertex vertex
SOURCE
SINK

46. Benchmarks
Compared to Spark, Flink, Hadoop doing word count, running on a
cluster of 9 nodes, 40 cores each

48. @gAmUssA @hazelcast #jfokus #hazelcastjet
Future (It’s bright!)
Processing guarantees for stream
processing
Streaming features (windowing, triggering)
Higher level streaming and batching APIs
Integration with additional Hazelcast
structures (ICache, IQueue ..)

49. @gAmUssA @hazelcast #jfokus #hazelcastjet
Future (It’s bright!)
Event sourcing / CQRS
Off-heap memory support
RxJava
More connectors to additional sources
(JMS, JDBC..)

50. @gAmUssA @hazelcast #jfokus #hazelcastjet
Grab while it’s hot!
jet.hazelcast.org
hazelcast/hazelcast-jet
http://bit.ly/streams_jfokus2017
documentation
Source on Github
Presentation materials

51. @gAmUssA @hazelcast #jfokus #hazelcastjet
Conclusion
Java Stream API provides very white
range of data processing tools
War And Piece – is a Big (a lot of data)
Book!
Now we’re pretty sure that Andrew and
Pierre are the main characters

53. SlidesCarnival icons are editable
shapes.
This means that you can:
● Resize them without losing quality.
● Change fill color and opacity.
Isn’t that nice? :)
Examples:

54. Now you can use any emoji as an icon!
And of course it resizes without losing quality and you can change the color.
How? Follow Google instructions
https://twitter.com/googledocs/status/730087240156643328
✋👆👉👍👤👦👧👨👩👪💃🏃💑❤😂
😉😋😒😭👶😸🐟🍒🍔💣📌📖🔨🎃🎈
🎨🏈🏰🌏🔌🔑and many more...
😉

55. @gAmUssA @hazelcast #jfokus #hazelcastjet
Extra graphics