Interoperating a Zoo of Data Processing Platforms Using with Rheem Sebastian Kruse and Yasser Idris (
We are witnessing a proliferation of big data, which has lead to a zoo of data processing systems. Each system providing a different set of features. For example, Spark provides scalability to analytic tasks, but Java 8 Streams provides low-latency. Furthermore, complex applications, such as ETL and ML, are now requiring a mixture of platforms to perform tasks efficiently. In such complex data analytics pipelines, the use of multiple data processing system is not only for performance reasons, but also because of data diversity. Datasets often natively reside on different data formats and storage engines. Unfortunately, developers are left alone in the challenging tasks of: (1) choosing the right platform for their applications; and (2) performing tedious and costly data migration and integration tasks to obtain the results. In this talk, we will present Rheem, an open source scalable cross-platform system that frees developers from these burdens. Rheem provides an abstraction layer on top of Spark (and other processing platforms) with the aim of enabling cross-platform optimization and interoperability. It automatically selects the best data processing platforms for a given task and also handles the cross-platform execution. In particular, we will discuss how Rheem allows Spark to work in tandem with other platforms in order to achieve higher performance. We will also show how easy a developer can write complex applications on top of Rheem to seamlessly use multiple different data processing platforms according to their tasks at hand. Using Rheem developers do not have to worry about the integration or data migration between Spark and other platforms.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Interoperating a Zoo of Data Processing Platforms Using with Rheem Sebastian Kruse and Yasser Idris (
Similaire à Interoperating a Zoo of Data Processing Platforms Using with Rheem Sebastian Kruse and Yasser Idris ( (20)
Plus de Databricks
Plus de Databricks (20)
Dernier
Dernier (20)
Interoperating a Zoo of Data Processing Platforms Using with Rheem Sebastian Kruse and Yasser Idris (
- 1. INTEROPERATINGA ZOO OF DATA PROCESSING PLATFORMS YASSER IDRIS, SEBASTIAN KRUSE
- 3. Apps On Spark Spark Spark SQL MLlib Spark Streaming GraphX Spark Batch LFS S3HDFS Batch App DB App Stream App ML APP Graph App 3
- 4. Beyond a Single Platform Complex Apps Spark Spark SQL MLlib Spark Streaming GraphX Spark Batch LFS S3HDFS BI App 4
- 5. Beyond a Single Platform Big vs. Small Data Spark Spark SQL MLlib Spark Streaming GraphX Spark Batch LFS S3HDFS ML App 5
- 6. Beyond a Single Platform Big vs. Small Data Spark Spark SQL MLlib Spark Streaming GraphX Spark Batch LFS S3HDFS ML App Java Streams 6
- 7. RHEEM Complex Apps BI App RHEEM Spark Spark SQL MLlib Spark Streaming GraphX Spark Batch Java Streams LFS S3HDFS 7
- 8. RHEEM Big vs. Small Data ML App RHEEM Spark Spark SQL MLlib Spark Streaming GraphX Spark Batch LFS S3HDFS Java Streams 8
- 9. RHEEM Big vs. Small Data ML App RHEEM Spark Spark SQL MLlib Spark Streaming GraphX Spark Batch LFS S3HDFS Java Streams 9
- 10. ML App RHEEM Spark Spark SQL MLlib Spark Streaming GraphX Spark Batch LFS S3HDFS RHEEM Big vs. Small Data Java Streams 10 CROSS-PLATFORM USE CASES
- 11. Platform Independence Spark Spark SQL MLlib Spark Streaming GraphX Spark Batch Postgres LFS S3HDFS DB App RHEEM GraphChi SELECT SUM(O_TOTALPRICE) FROM ORDERS; Java Streams 11
- 12. Platform Independence Spark Spark SQL MLlib Spark Streaming GraphX Spark Batch Postgres LFS S3HDFS DB App RHEEM GraphChi Java Streams SELECT SUM(O_TOTALPRICE) FROM ORDERS; 12
- 13. Platform Independence Spark Spark SQL MLlib Spark Streaming GraphX Spark Batch Postgres LFS S3HDFS DB App RHEEM GraphChi Java Streams SELECT SUM(O_TOTALPRICE) FROM ORDERS; 13
- 14. Platform Independence Spark Spark SQL MLlib Spark Streaming GraphX Spark Batch Postgres LFS S3HDFS DB App RHEEM GraphChi Java Streams SELECT SUM(O_TOTALPRICE) FROM ORDERS; 14
- 24. RHEEM INTERNALS 24 Let’s have a look under the hood…
- 25. RHEEM in a Nutshell Rheem App Apache Spark Java StreamsGraphChi Postgres HDFS LFS S3 Java Driver Rheem API Spark DriverPg Driver Monitor Cross-Platform Optimizer Cost Learner Cross-Platform Executor 25
- 26. Fine-grained Platform Selection Table source Filter Map Group by Collect Table source Filter Map Group by Collect Get real and predicted weights from the year 2017 Calculate MSE Group by Airline Output results 26 Logical plan RHEEM plan Execution plan
- 27. Automatic Data Movement Table source Filter Map Group by Collect Table source Filter Map Group by Collect Get real and predicted weights from the year 2017 Calculate MSE Group by Airline Output results 27 Stage 2 Spark Stage 1 Postgres Logical plan RHEEM plan Execution plan
- 28. Automatic Data Movement Table source Filter Map Group by Collect Table source Filter Map Group by Collect Get real and predicted weights from the year 2017 Calculate MSE Group by Airline Output results SQL 2 RDD 28 Stage 2 Spark Stage 1 Postgres Logical plan RHEEM plan Execution plan
- 29. Rheem Extensibility Spark Spark SQL MLlib Spark Streaming GraphX Spark Batch Postgres LFS S3HDFS Java Streams GraphChiHadoop Map Filter Reduce Count Approx. … … … 29 Map Filter Reduce Map … Filter Page Rank … Map … Count Approx. Page Rank
- 30. Scala REST PIGLATIN PythonJava Multiple APIs Spark Spark SQL MLlib Spark Streaming GraphX Spark Batch Postgres LFS S3HDFS Cross-Platform Apps RHEEM Java Streams GraphChi 30
- 31. Rheem Studio 31
- 33. Scala REST PIGLATIN PythonJava The dRHEEM goes on… Spark Spark SQL MLlib Spark Streaming GraphX Spark Batch Postgres LFS S3HDFS Cross-Platform Apps RHEEM Java Streams GraphChi 33 Add more platforms Integrate with resource managers Enhance data exchange paths Continuously improve optimizer Re-use intermediate datasets across jobs
- 34. We want you! Website: http://da.qcri.org/rheem/ GitHub: https://github.com/rheem-ecosystem Apache Incubator Very Soon! Interested? Then, become a dRHEEMer! Gitter: https://gitter.im/rheem-ecosystem/Lobby