Data is flowing everywhere around us, from phones, credit cards, sensor-equipped buildings, vending machines, thermostats, trains, buses, planes, posts to social media, digital pictures and video and so on.... http://www.datascicon.tech

1. DIVIDE, DISTRIBUTE AND CONQUER:
STREAM V. BATCH

2. Stream v. Batch

3. Who am I?

4. Solutions Architect
Who am I?

5. Solutions Architect
Developer Advocate
Who am I?

6. Solutions Architect
Developer Advocate
@gamussa in internetz
Who am I?

7. Solutions Architect
Developer Advocate
@gamussa in internetz
Hey you, yes, you, go follow me in twitter ©
Who am I?

8. @gamussa @confluentinc @DataSciCon
BATCH PROCESSING
Data at rest

9. @gamussa @confluentinc @DataSciCon
Data and Queries
Origin and processing

10. @gamussa @confluentinc @DataSciCon

11. @gamussa @confluentinc @DataSciCon
Data…

12. @gamussa @confluentinc @DataSciCon
Data…

13. @gamussa @confluentinc @DataSciCon
✓ … inherently immutable
Data…
✓ … time-based

14. @gamussa @confluentinc @DataSciCon
CRUD -> CR

15. @gamussa @confluentinc @DataSciCon
Processing is a query

16. @gamussa @confluentinc @DataSciCon
Processing is a query
Function on full data set

17. @gamussa @confluentinc @DataSciCon
Processing is a query
Function on full data set
Projection

18. @gamussa @confluentinc @DataSciCon
Processing is a query
Function on full data set
Projection
Aggregations

19. @gamussa @confluentinc @DataSciCon
Processing is a query
Function on full data set
Projection
Aggregations
Joins

20. @gamussa @confluentinc @DataSciCon
Lambda architecture origins
http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html

23. @gamussa @confluentinc @DataSciCon
https://mapr.com/developercentral/lambda-architecture/
Lambda Architecture

24. @gamussa @confluentinc @DataSciCon

25. @gamussa @confluentinc @DataSciCon
TFW Trying to explain modern big data
landscape

26. @gamussa @confluentinc @DataSciCon

27. @gamussa @confluentinc @DataSciCon
STREAM PROCESSING
Data is motion

28. @gamussa @confluentinc @DataSciCon
Streaming Platform

29. @gamussa @confluentinc @DataSciCon
Streaming Platform

30. @gamussa @confluentinc @DataSciCon

31. @gamussa @confluentinc @DataSciCon
Interesting cases
Before You Go

32. I FOUND YOUR LACK OF FAULT TOLERANCE
DISTURBING

33. Data is too important to
store it in one computer

38. @gamussa @confluentinc @DataSciCon
How to process
«infinite» data?

39. @gamussa @confluentinc @DataSciCon
Time model

40. @gamussa @confluentinc @DataSciCon
Time model
Different use cases time semantics

41. @gamussa @confluentinc @DataSciCon
Time model
Different use cases time semantics
Majority of use cases require event-
time semantics

42. @gamussa @confluentinc @DataSciCon
Time model
Different use cases time semantics
Majority of use cases require event-
time semantics
Other use cases may require
processing-time or special variants
like ingestion-time

43. @gamussa @confluentinc @DataSciCon
Time Model

44. @gamussa @confluentinc @DataSciCon
Time Model

45. @gamussa @confluentinc @DataSciCon
Time Model

46. @gamussa @confluentinc @DataSciCon
Windowing
Input data, where
colors represent
different users events
Rectangles denote
different event-time
windows
processing-time
event-time
windowing
alice
bob
dave

47. @gamussa @confluentinc @DataSciCon
https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101

48. @gamussa @confluentinc @DataSciCon
Windowing
Windowing is an operation that groups
events
Most commonly needed: time windows,
session windows
Examples:
✗Real-time monitoring: 5-minute averages
✗Reader behavior on a website: user browsing sessions

49. @gamussa @confluentinc @DataSciCon
Out-of-order and late data
Is very common in practice, not a rare
corner case
✗Related to time model discussion

50. @gamussa @confluentinc @DataSciCon
Out-of-order and late data

51. @gamussa @confluentinc @DataSciCon
Out-of-order and late data
Users with mobile phones enter
airplane, lose Internet connectivity

52. @gamussa @confluentinc @DataSciCon
Out-of-order and late data
Users with mobile phones enter
airplane, lose Internet connectivity
Emails are being written
during the 10h flight

53. @gamussa @confluentinc @DataSciCon
Out-of-order and late data
Users with mobile phones enter
airplane, lose Internet connectivity
Emails are being written
during the 10h flight
Internet connectivity is restored,
phones will send queued emails now

54. @gamussa @confluentinc @DataSciCon
Stream Processing: results

55. @gamussa @confluentinc @DataSciCon
Stream Processing: results
• Yes, it’s possible to get computation
results in real time

56. @gamussa @confluentinc @DataSciCon
Stream Processing: results
• Yes, it’s possible to get computation
results in real time
• Windows – finite view of infinite data
• Based on temporal characteristics of the evet

57. @gamussa @confluentinc @DataSciCon
Stream Processing: results
• Yes, it’s possible to get computation
results in real time
• Windows – finite view of infinite data
• Based on temporal characteristics of the evet
• Late event processing
• You choose how long to wait

58. @gamussa @confluentinc @DataSciCon
DEMO
Let’s analyze flights

59. @gamussa @confluentinc @DataSciCon
https://www.confluent.io/blog/predicting-flight-arrivals-with-the-apache-kafka-streams-api/

60. @gamussa @confluentinc @DataSciCon
Example: Training Flight Prediction Model

61. @gamussa @confluentinc @DataSciCon
https://github.com/confluentinc/online-inferencing-blog-
application

62. @gamussa @confluentinc @DataSciCon
Thanks!
questions?
@gamussa
viktor@confluent.io