Speaker: Albert Ramírez Cerquera Video: https://youtu.be/18Tbr1LZmtM En esta charla, vemos cómo utilizar el modelo de programación Apache Beam para procesar datos por batch y en streaming. Además, se enseñará cómo se puede ejecutar Beam en ejecutores como Flink y Google Cloud Dataflow. --------------------------------------------------------------------------------------------------------------------------------------------------------------- Hay trabajos y hay carreras. Las oportunidades vienen a golpear la puerta cuando menos lo esperas. La decisión es tuya. Desde tener la oportunidad de hacer algo significativo día tras día, hasta estar rodeado de gente supremamente inteligente y motivada. ¿Estás listo? Descúbre todas nuestras oportunidades acá: https://bit.ly/2PWKky9 --------------------------------------------------------------------------------------------------------------------------------------------------------------- Síguenos en: Facebook: https://www.facebook.com/Globant/ Twitter: https://twitter.com/Globant Instagram: https://www.instagram.com/globantpics/ Linkedin: https://www.linkedin.com/company/globant Visita nuestra página web: https://bit.ly/2XLVYQD

1. Portable Batch and Streaming Processing
APACHE BEAM

2. ABOUT ME
● Data Architect at Globant
● Certified Google Data
Engineer
● Applying software engineering
practices to Big Data
development

3. AGENDA
1. Introduction
2. What is Apache Beam?
3. Why Apache Beam?
4. Crash course
5. Beam on Flink
6. Beam on DataFlow
7. Summary

4. INTRODUCTION
Computation
Result
Computation Computation Computation Computation
Result

5. INTRODUCTION
Computation Computation Computation Computation
Result
Computation Computation Computation Computation
Result
Batch Streaming

6. WHAT IS APACHE BEAM?
● Original code from Google (Cloud Dataflow SDK).
● Apache open-source project since 2016.
● Parallel/distributed data processing.
● Unified programming model for batch and streaming.
● Portable, execution engine and programming language
of your choice.

7. WHY APACHE BEAM?
Write Translate

8. SDK INSTALLATION

9. CRASH COURSE
Input Transform PCollection Transform PCollection Sink
Pipeline
Output

10. PIPELINE
A Pipeline encapsulates your entire data processing task, from start to finish. This
includes reading input data, transforming that data, and writing output data.

11. PIPELINE
Pipeline

12. PCollection
The PCollection abstraction represents a potentially distributed, multi-element data
set. You can think of a PCollection as “pipeline” data; Beam transforms use
PCollection objects as inputs and outputs. As such, if you want to work with data in
your pipeline, it must be in the form of a PCollection.

13. PCollection

14. PCollection
● The elements of a PCollection may be of any type, but
must all be of the same type.
● A PCollection is immutable.
● A PCollection can be either bounded or unbounded in
size.
● Each element in a PCollection has an associated intrinsic
timestamp.

15. PCollection
PCollection PCollection
Pipeline

16. Transforms

17. Transforms

18. Transforms

19. Transforms
● Read - parallel connectors to external systems
● ParDo - per element processing
● GroupByKey - aggregating elements per key and window
● Flatten - union of PCollections
● Window - set the windowing strategy for a PCollection

20. Transforms
Transform PCollection Transform PCollection Sink
Pipeline

21. Pipeline I/O

22. Pipeline I/O

23. Pipeline I/O
Input Transform PCollection Transform PCollection Sink
Pipeline
Output

24. Beam on Flink

25. Beam on Google Cloud DataFlow

26. BE AWARE OF
Write Translate

27. BE AWARE OF
Portable protocol
JAVA Objects

28. What is being computed?

29. Where in event time?

30. Summary
● Apache Beam is not an execution engine, is a programming model.
● Apache Flink and Google Cloud Dataflow are the execution engines
that supports all the features of Beam.
● The JAVA SDK is the the most up to date.
● PCollections are the main Data Structure in Beam.
● Transformations receive PCollections as inputs and produces the
same as outputs.
● Check the compatibility table before jump into the development.

31. Q&A

32. Thank you for listening!