Vectorized R Execution in Apache Spark
•
0 j'aime•471 vues
Apache Spark already has a vectorization optimization in many operations, for instance, internal columnar format, Parquet/ORC vectorized read, Pandas UDFs, etc. Vectorization improves performance greatly in general. In this talk, the performance aspect of SparkR will be discussed and vectorization in SparkR will be introduced with technical details. SparkR vectorization allows users to use the existing codes as are but boost the performance around several thousand present faster when they execute R native functions or convert Spark DataFrame to/from R DataFrame.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Vectorized R Execution in Apache Spark
Similaire à Vectorized R Execution in Apache Spark (20)
Plus de Databricks
Plus de Databricks (20)
Dernier
Dernier (20)
Vectorized R Execution in Apache Spark
- 1. WIFI SSID:Spark+AISummit | Password: UnifiedDataAnalytics
- 2. Hyukjin Kwon, Databricks Vectorized R Execution in Apache Spark #UnifiedDataAnalytics #SparkAISummit
- 3. #UnifiedDataAnalytics #SparkAISummit Hyukjin Kwon 3 • Apache Spark PMC and Committer • Koalas committer • PySpark, SparkSQL, SparkR, build, etc. • Active in Spark dev @HyukjinKwon
- 4. #UnifiedDataAnalytics #SparkAISummit Agenda • SparkR and R interaction • Native Implementation • Apache Arrow • Vectorized Implementation • Future Work 4
- 5. #UnifiedDataAnalytics #SparkAISummit 5 SparkR and R interaction
- 6. #UnifiedDataAnalytics #SparkAISummit Why? Scala API R API 6 Cool!
- 7. #UnifiedDataAnalytics #SparkAISummit Why? Scala API R API 7 12.5x slower … ?
- 8. #UnifiedDataAnalytics #SparkAISummit Why? Scala API R API 8 40x slower … ???
- 9. #UnifiedDataAnalytics #SparkAISummit createDataFrame Create Spark DataFrame from R DataFrame and lists. 9
- 10. #UnifiedDataAnalytics #SparkAISummit collect Collect R DataFrame from Spark DataFrame at Driver. 10
- 11. #UnifiedDataAnalytics #SparkAISummit dapply Apply R native function to each partition 11
- 12. #UnifiedDataAnalytics #SparkAISummit gapply Apply R native function on each group. 12
- 13. #UnifiedDataAnalytics #SparkAISummit 13 Native Implementation
- 14. #UnifiedDataAnalytics #SparkAISummit SparkR Architecture 14 Spark Driver JVM JVM DataSources JVMR RBackend R R R R
- 15. #UnifiedDataAnalytics #SparkAISummit Driver implementation 15 1. RBackend opens a server port and waits for connections 4. RBackendHandler handles and process requests. It sends back row by row 2. R establishes the socket connections 3. Each SparkR call sends serialized data over the socket and waits for response R JVM Backend
- 16. #UnifiedDataAnalytics #SparkAISummit createDataFrame and collect 16 DataFrame R Data Frame R Data Frame Rows to Array(Array(…)) list(list(…)) parallelize(…) row, row, ... row, row, ... parallelize(…) Bytes to rows Bytes to lists data.frame(…)
- 17. #UnifiedDataAnalytics #SparkAISummit Worker Implementation 17 R R JVM 1. RRunner sends data and serialized R function through a socket. 2. R receives the serialized function and data. 3. R deserializes the function and the data row by row 4. R executes the function, and send the results back to RRunner.
- 18. #UnifiedDataAnalytics #SparkAISummit dapply and gapply 18 RRunner PhysicalOperator row by row row by row Invoke R function serialize row by row deserialize row by row row, row, ... row, row, ...
- 20. #UnifiedDataAnalytics #SparkAISummit Apache Arrow A cross-language development platform for in-memory data Columnar In-Memory SparkR supports Arrow 0.12.1+(?) 20 20
- 21. #UnifiedDataAnalytics #SparkAISummit Vectorization 21 21 https://www.slideshare.net/Hadoop_Summit/the-columnar-roadmap-apache-parquet-and-apache-arrow-102997214 SIMD Pipelining https://medium.com/wasmer/webassembly-and-simd-13badb9bf1a8
- 22. #UnifiedDataAnalytics #SparkAISummit Interchangeable, no copy 22 Each system has its own internal memory format 70-80% computation wasted on (de)serialization Similar functionality implemented in multiple projects All systems utilize the same memory format No overhead for cross-system communication Projects can share functionality (eg, Parquet-to-Arrow reader)
- 23. #UnifiedDataAnalytics #SparkAISummit Serialization and deserialization 23 23 https://wesmckinney.com/blog/arrow-streaming-columnar/ See also Arrow format https://sapbr.com/2016/08/15/dictionary-encoding/
- 24. #UnifiedDataAnalytics #SparkAISummit "Portable" Data Frames 24 Share data and algorithm at ~zero cost https://www.slideshare.net/wesm/apache-arrow-at-dataengconf-barcelona-2018
- 25. #UnifiedDataAnalytics #SparkAISummit 25 Vectorized Implementation
- 26. #UnifiedDataAnalytics #SparkAISummit createDataFrame and collect Use Arrow to Serialize/Deserialize data Streaming format for Interprocess messaging / communication (IPC) ArrowWriter and ArrowColumnVector Communicate JVM and R worker via Socket createDataFrame in SQLContext.R readArrowStreamFromFile in SQLUtils.scala collect in DataFrame.R collectAsArrowToR in Dataset.scala 26
- 27. #UnifiedDataAnalytics #SparkAISummit createDataFrame and collect 27 DataFrame R Data Frame R Data Frame Rows to Array(Array(…)) list(list(…)) parallelize(…) row, row, ... row, row, ... parallelize(…) Bytes to rows Bytes to lists data.frame(…)
- 28. #UnifiedDataAnalytics #SparkAISummit createDataFrame and collect 28 DataFrame R Data Frame Arrow batches R Data Frame Arrow batches to Spark DataFrameArrow batches Arrow batches Arrow batches Spark DataFrame to Arrow bathes
- 29. #UnifiedDataAnalytics #SparkAISummit Benchmark 29 collect No Arrow: 20.85112 secs Arrow: 1.224419 secs 17x faster No Arrow: 240.50508 secs Arrow: 5.707062 secs 42x faster createDataFrame
- 30. #UnifiedDataAnalytics #SparkAISummit dapply and gapply Use Arrow to Serialize/Deserialize data Streaming format for Interprocess messaging / communication (IPC) ArrowWriter and ArrowColumnVector Communicate JVM and R worker via Socket ArrowRRunner Physical Operators for each R native function executions MapPartitionsInRWithArrowExec FlatMapGroupsInRWithArrowExec 30
- 31. #UnifiedDataAnalytics #SparkAISummit dapply and gapply 31 RRunner PhysicalOperator row by row row by row Invoke R function serialize row by row deserialize row by row row, row, ... row, row, ...
- 32. #UnifiedDataAnalytics #SparkAISummit dapply and gapply 32 ArrowRRunner PhysicalOperator group of rows group of rows Invoke R function serialize Arrow batches deserialize Arrow batches Arrow batches Arrow batches
- 33. #UnifiedDataAnalytics #SparkAISummit Benchmark 33 gapply No Arrow: 699.0714 secs Arrow: 16.2713 secs 43x faster No Arrow: 202.36236 secs Arrow: 6.222105 secs 33x faster dapply
- 34. #UnifiedDataAnalytics #SparkAISummit Benchmark 34 Can’t believe? Can’t wait to try it out by yourself? Try it out here on a live Jupyter notebook github.com/HyukjinKwon/spark-notebooks
- 36. #UnifiedDataAnalytics #SparkAISummit Apache Arrow ARROW-4512 Actually, the vectorized implementation does not fully work in a streaming manner yet. 36
- 39. #UnifiedDataAnalytics #SparkAISummit dapplyCollect and gapplyCollect 39 ArrowRRunner PhysicalOperator group of rows Invoke R function serialize Arrow batches deserialize Arrow batches Arrow batches Arrow batches ??? Schema is unknown before eager execution
- 40. #UnifiedDataAnalytics #SparkAISummit dapplyCollect and gapplyCollect 40ArrowRRunner PhysicalOperator Invoke R function serialize Arrow batches deserialize Arrow batches DataFrame R Data Frame R Data Frame Arrow batches Arrow batches Arrow batchesBy-passed Arrow batches By-pass with DataFrame<binary>
- 41. #UnifiedDataAnalytics #SparkAISummit That’s it! Special thanks to Felix Cheung Bryan Cutler Liang-Chi Hsieh Hossein Falaki Takuya Ueshin Yanbo Liang and, last but not least Apache Arrow community :D 41
- 42. DON’T FORGET TO RATE AND REVIEW THE SESSIONS SEARCH SPARK + AI SUMMIT