Apache Spark Operations
•Télécharger en tant que PPTX, PDF•
7 j'aime•1,912 vues
Presented by Kostas Sakellis at Spark Lightning Night 2015
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
En vedette (19)
Similaire à Apache Spark Operations
Similaire à Apache Spark Operations (20)
Plus de Cloudera, Inc.
Plus de Cloudera, Inc. (20)
Dernier
Dernier (20)
Apache Spark Operations
- 1. 1© Cloudera, Inc. All rights reserved. Spark Operations Kostas Sakellis
- 2. 2© Cloudera, Inc. All rights reserved. Me • Software Engineer at Cloudera • Contributor to Apache Spark • Before that, contributed to Cloudera Manager
- 3. 3© Cloudera, Inc. All rights reserved. Building a proof of concept! Courtesy of: http://www.nefloridadesign.com/mbimages/6.jpg
- 4. 4© Cloudera, Inc. All rights reserved. Example sc.textFile(“hdfs://data/u.item”, 4) .map(Movie(_)) .filter(_.month.equals(“Nov”)) .collect()
- 5. 5© Cloudera, Inc. All rights reserved. Example sc.textFile(“hdfs://data/u.item”, 4) .map(Movie(_)) .filter(_.month.equals(“Nov”)) .collect()
- 6. 6© Cloudera, Inc. All rights reserved. Example sc.textFile(“hdfs://data/u.item”, 4) .map(Movie(_)) .filter(_.month.equals(“Nov”)) .collect()
- 7. 7© Cloudera, Inc. All rights reserved. Partitions sc.textFile(“hdfs://data/u.item”, 4) .map(Movie(_)) .filter(_.month.equals(“Nov”)) .collect() HDFS Partition 1 Partition 2 Partition 3 Partition 4
- 8. 8© Cloudera, Inc. All rights reserved. RDDs sc.textFile(“hdfs://data/u.item”, 4) .map(Movie(_)) .filter(_.month.equals(“Nov”)) .collect() …RDD HDFS Partition 1 Partition 2 Partition 3 Partition 4
- 9. 9© Cloudera, Inc. All rights reserved. RDDs sc.textFile(“hdfs://data/u.item”, 4) .map(Movie(_)) .filter(_.month.equals(“Nov”)) .collect() …RDD …RDD HDFS Partition 1 Partition 2 Partition 3 Partition 4 Partition 1 Partition 2 Partition 3 Partition 4
- 10. 10© Cloudera, Inc. All rights reserved. RDDs sc.textFile(“hdfs://data/u.item”, 4) .map(Movie(_)) .filter(_.month.equals(“Nov”)) .collect() …RDD …RDD HDFS Partition 1 Partition 2 Partition 3 Partition 4 Partition 1 Partition 2 Partition 3 Partition 4 …RDD Partition 1 Partition 2 Partition 3 Partition 4
- 11. 11© Cloudera, Inc. All rights reserved. …RDD …RDD RDDs HDFS Partition 1 Partition 2 Partition 3 Partition 4 sc.textFile(“hdfs://data/u.item”, 4) .map(Movie(_)) .filter(_.month.equals(“Nov”)) .collect() Partition 1 Partition 2 Partition 3 Partition 4 …RDD Partition 1 Partition 2 Partition 3 Partition 4 Collect
- 12. 12© Cloudera, Inc. All rights reserved. …RDD …RDD RDD Lineage HDFS Partition 1 Partition 2 Partition 3 Partition 4 sc.textFile(“hdfs://data/u.item”, 4) .map(Movie(_)) .filter(_.month.equals(“Nov”)) .collect() Partition 1 Partition 2 Partition 3 Partition 4 …RDD Partition 1 Partition 2 Partition 3 Partition 4 Collect Lineage
- 13. 13© Cloudera, Inc. All rights reserved. Task …RDD …RDD HDFS Partition 1 Partition 2 Partition 3 Partition 4 Partition 1 Partition 2 Partition 3 Partition 4 …RDD Partition 1 Partition 2 Partition 3 Partition 4 Collect • A pipelined set of transformation on a single thread
- 14. 14© Cloudera, Inc. All rights reserved. Spark Architecture
- 15. 15© Cloudera, Inc. All rights reserved. Spark System Architecture
- 16. 16© Cloudera, Inc. All rights reserved. Deployments • Spark supports pluggable Cluster Managers • local, Standalone, YARN and Mesos • In early 2014, CDH 4.x with Spark 0.9 only supported Standalone • CDH 5.x includes Spark on YARN support
- 17. 17© Cloudera, Inc. All rights reserved. Standalone Master Worker Client Worker Process App Master Process
- 18. 18© Cloudera, Inc. All rights reserved. Standalone • On cluster ./sbin/start-master.sh ./sbin/start-slave.sh <master-spark-URL> • Submit job spark-submit --master <master-spark-URL> …
- 19. 19© Cloudera, Inc. All rights reserved. Container YARN Architecture Resource Manager Node Manager Client Node Manager Container Process App Master Container Process
- 20. 20© Cloudera, Inc. All rights reserved. Container Spark on YARN Architecture Resource Manager Node Manager Client Node Manager Container Process App Master Container Process
- 21. 21© Cloudera, Inc. All rights reserved. Container Spark on YARN Architecture Resource Manager Node Manager Client Node Manager Container Process App Master Container Process
- 22. 22© Cloudera, Inc. All rights reserved. Spark on YARN • Submit job spark-submit --master yarn-client … • Cluster mode spark-submit --master yarn-cluster … • Spark shell only works in client mode!
- 23. 23© Cloudera, Inc. All rights reserved. Customers often have shared infrastructure Courtesy of: https://radioglobalistic.files.wordpress.com/2011/02/lagos-traffic.jpg
- 24. 24© Cloudera, Inc. All rights reserved. Multi-tenancy • Cluster utilization is top metric • Target: 70-80% utilization • Mixed workloads from mixed customers • We recommend YARN • Built in resource manager
- 25. 25© Cloudera, Inc. All rights reserved. Underutilized Clusters Courtesy of: http://media.nbclosangeles.com/images/1200*675/60-freeway-repair-dec16-2-empty.JPG
- 26. 26© Cloudera, Inc. All rights reserved. Dynamic Allocation • Spark applications scale the number of executors based on load • Removes need for: --num-executors • Idle executors get killed • First supported in CDH 5.4 • Ideal for: • Long ETL jobs with large shuffles • shell applications: hive and spark shell
- 27. 27© Cloudera, Inc. All rights reserved. Dynamic Allocation Limitations • Still required to specify cores • --num-cores • Memory • --executor-memory • Includes JVM overhead • Need to do the math yourself • Our customers still get it wrong!
- 28. 28© Cloudera, Inc. All rights reserved. The Future of Dynamic Allocation • Only “task size” needed: --task-size • Eliminates • --num-cores • --num-executors • --executor-memory • Leads to better cluster utilization
- 29. 29© Cloudera, Inc. All rights reserved. Security, now it’s getting serious. Courtesy of: https://www.iti.illinois.edu/sites/default/files/Cybersecurity_image.jpg
- 30. 30© Cloudera, Inc. All rights reserved. Authentication • Kerberos – the necessary evil • Ubiquitous amongst other services • YARN, HDFS, Hive, HBase, etc. • Spark utilizes delegation tokens
- 31. 31© Cloudera, Inc. All rights reserved. Encryption • Control plane • File distribution • Block Manager • User UI / REST API • Data-at-rest (shuffle files) SPARK-6028 (Replace with netty) Replace with netty Spark 1.4 SPARK-2750 (SSL) SPARK-5682
- 32. 32© Cloudera, Inc. All rights reserved. Authorization • Enterprises have sensitive data • Beyond HDFS file permissions • Partial access to data • Column level granularity • Apache Sentry • HDFS-Sentry synchronization plugin • Record Service • Column level security for Spark!
- 33. 33© Cloudera, Inc. All rights reserved. Thank you We’re Hiring!
Notes de l'éditeur
- Lets talk about what we have seen as issues from our customers as issues as they try to get Spark into production.
- In scope - Focus on operational issues - Not on building the code itself Experience from our customer support tickets
- Spark makes building a proof of concept with a subset of data relatively easy. But then things go wrong Plug for my talk at Hadoop Summit
- Lets start with an example program in Spark.
- Lets start with an example program in Spark.
- The sum() call launches a job
- A chunk of data somewhere Could be on Hadoop File System (HDFS) Could be cached in Spark Defines the degree of parallelism
- Describes a way of generating input and output partitions Immutable – very important! RDDs can depend on other RDDs Most have single parent Joins have multiple parents Lineage over replication for fault tolerance https://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdf
- Describes a way of generating input and output partitions Immutable – very important! RDDs can depend on other RDDs Most have single parent Joins have multiple parents Lineage over replication for fault tolerance https://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdf
- Describes a way of generating input and output partitions Immutable – very important! RDDs can depend on other RDDs Most have single parent Joins have multiple parents Lineage over replication for fault tolerance https://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdf
- Describes a way of generating input and output partitions Immutable – very important! RDDs can depend on other RDDs Most have single parent Joins have multiple parents Lineage over replication for fault tolerance https://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdf
- Describes a way of generating input and output partitions Immutable – very important! RDDs can depend on other RDDs Most have single parent Joins have multiple parents Lineage over replication for fault tolerance https://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdf
- Lets review the general Spark architecture
- A driver Where the DAG scheduler lives Drives the show Single point of failure Executors Communicates with driver Runs the tasks created by the driver Think of this as a ThreadPoolExecutor in java Pluggable cluster managers YARN, Mesos, standalone
- In scope - Focus on operational issues - Not on building the code itself Experience from our customer support tickets
- Lets review the general Spark architecture
- In scope - Focus on operational issues - Not on building the code itself Experience from our customer support tickets
- Lets review the general Spark architecture
- Lets review the general Spark architecture
- Lets review the general Spark architecture
- In scope - Focus on operational issues - Not on building the code itself Experience from our customer support tickets
- Spark makes building a proof of concept with a subset of data relatively easy.
- Spark makes building a proof of concept with a subset of data relatively easy.
- Control plane File distribution Block Manager User UI / REST API Data-at-rest (shuffle files)