Big data analytics with Spark & Cassandra
•
6 j'aime•2,723 vues
This document provides an agenda and overview of Big Data Analytics using Spark and Cassandra. It discusses Cassandra as a distributed database and Spark as a data processing framework. It covers connecting Spark and Cassandra, reading and writing Cassandra tables as Spark RDDs, and using Spark SQL, Spark Streaming, and Spark MLLib with Cassandra data. Key capabilities of each technology are highlighted such as Cassandra's tunable consistency and Spark's fault tolerance through RDD lineage. Examples demonstrate basic operations like filtering, aggregating, and joining Cassandra data with Spark.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
En vedette (13)
Similaire à Big data analytics with Spark & Cassandra
Similaire à Big data analytics with Spark & Cassandra (20)
Dernier
Dernier (20)
Big data analytics with Spark & Cassandra
- 1. Big Data Analytics mit Spark & Cassandra_ JUG Stuttgart 01/2016 Matthias Niehoff
- 2. •Cassandra •Spark •Spark & Cassandra •Spark Applications •Spark Streaming •Spark SQL •Spark MLLib Agenda_ 2
- 3. Cassandra 3
- 4. •Distributed database •Highly Available •Linear Scalable •Multi Datacenter Support •No Single Point Of Failure •CQL Query Language • Similiar to SQL • No Joins and aggregates •Eventual Consistency „Tunable Consistency“ Cassandra_ 4
- 5. Distributed Data Storage_ 5 Node 1 Node 2 Node 3 Node 4 1-25 26-5051-75 76-0
- 6. CQL - Querying Language With Limitations_ 6 SELECT * FROM performer WHERE name = 'ACDC' —> ok SELECT * FROM performer WHERE name = 'ACDC' and country = 'Australia' —> not ok SELECT country, COUNT(*) as quantity FROM artists GROUP BY country ORDER BY quantity DESC —> not supported performer name (PK) genre country
- 7. Spark 7
- 8. •Open Source & Apache project since 2010 •Data processing Framework • Batch processing • Stream processing What Is Apache Spark_ 8
- 9. •Fast • up to 100 times faster than Hadoop • a lot of in-memory processing • linear scalable using more nodes •Easy • Scala, Java and Python API • Clean Code (e.g. with lambdas in Java 8) • expanded API: map, reduce, filter, groupBy, sort, union, join, reduceByKey, groupByKey, sample, take, first, count •Fault-Tolerant • easily reproducible Why Use Spark_ 9
- 10. •RDD‘s – Resilient Distributed Dataset • Read–Only description of a collection of objects • Distributed among the cluster (on memory or disk) • Determined through transformations • Allows automatically rebuild on failure •Operations • Transformations (map,filter,reduce...) —> new RDD • Actions (count, collect, save) •Only Actions start processing! Easily Reproducable?_ 10
- 11. •Partitions • Describes the Partitions (i.e. one per Cassandra Partition) •Dependencies • dependencies on parent RDD’s •Compute • The function to compute the RDD’s partitions •(Optional) Partitioner • How is the data partitioned? (Hash, Range..) •(Optional) Preferred Location • Where to get the data (i.e. List of Cassandra Node IP’s) Properties Of A RDD_ 11
- 13. Reproduce RDD’s Using A Tree_ 13 Datenquelle rdd1 rdd3 val1 rdd5 rdd2 rdd4 val2 rdd6 val3 map(..)filter(..) union(..) count() count() count() sample(..) cache()
- 14. •Transformations • map, flatMap • sample, filter, distinct • union, intersection, cartesian •Actions • reduce • count • collect,first, take • saveAsTextFile • foreach Spark Transformations & Actions_ 14
- 15. Run Spark In A Cluster_ 15
- 16. •Memory • A lot of data in memory • More memory —> Less disk IO —> Faster processing • Minimum 8 GB / Node •Network • Communication between Driver, Cluster Manager & Worker • Important for reduce operations • 10 Gigabit LAN or better •CPU • Less communication between threads • Good to parallelize • Minimum 8 – 16 Cores / Node What About Hardware?_ 16
- 17. •Master Web UI (8080) How To Monitor? (1/3)_ 17
- 18. •Worker Web UI (8081) How To Monitor? (2/3)_ 18
- 19. •Application Web UI (4040) How To Monitor? (3/3)_ 19
- 21. •Parallelization! • keys are use for partitioning • pairs with different keys are distributed across the cluster •Efficient processing of • aggregate by key • group by key • sort by key • joins, union based on keys Why Use Pair RDD’s_ 21
- 22. RDD Dependencies_ 22 „Narrow“ (pipeline-able) map, filter union join on co partitioned data
- 23. RDD Dependencies_ 23 „Wide“ (shuffle) groupBy on non partitioned data join on non co partitioned data
- 24. Spark Demo 24
- 26. Use Spark And Cassandra In A Cluster_ 26 Spark Client Spark Driver C* C* C*C* Spark WN Spark WN Spark WN Spark WN Spark Master
- 27. Two Datacenter - Two Purposes_ 27 C* C* C*C* C* C* C*C* Spark WN Spark WN Spark WN Spark WN Spark Master DC1 - Online DC2 - Analytics
- 28. •Spark Cassandra Connector by Datastax • https://github.com/datastax/spark-cassandra-connector •Cassandra tables as Spark RDD (read & write) •Mapping of C* tables and rows onto Java/Scala objects •Server-Side filtering („where“) •Compatible with • Spark ≥ 0.9 • Cassandra ≥ 2.0 •Clone & Compile with SBT or download at Maven Central Connecting Spark With Cassandra_ 28
- 29. •Start the shell bin/spark-shell --jars ~/path/to/jar/spark-cassandra-connector- assembly-1.3.0.jar --conf spark.cassandra.connection.host=localhost •Import Cassandra Classes scala> import com.datastax.spark.connector._ Use The Connector In The Shell_ 29
- 30. •Read complete table val movies = sc.cassandraTable("movie","movies") // returns CassandraRDD[CassandraRow] •Read selected columns val movies = sc.cassandraTable("movie","movies").select("title","year") •Filter rows val movies = sc.cassandraTable("movie","movies").where("title = 'Die Hard'") •Access Columns in Result Set movies.collect.foreach(r => println(r.get[String]("title"))) Read A Cassandra Table_ 30
- 32. Read As Case Class case class Movie(title: String, year: Int) sc.cassandraTable[Movie]("movie","movies").select("title","year") sc.cassandraTable("movie","movies").select("title","year").as(Movie) Read A Cassandra Table_ 32
- 33. •Every RDD can be saved • Using Tuples val tuples = sc.parallelize(Seq(("Hobbit",2012),("96 Hours",2008))) tuples.saveToCassandra("movie","movies", SomeColumns("title","year") • Using Case Classes case class Movie (title:String, year: int) val objects = sc.parallelize(Seq(Movie("Hobbit",2012),Movie("96 Hours",2008))) objects.saveToCassandra("movie","movies") Write Table_ 33
- 34. // Load and format as Pair RDD val pairRDD = sc.cassandraTable("movie","director") .map(r => (r.getString("country"),r)) // Directors / Country, sorted pairRDD.mapValues(v => 1).reduceByKey(_+_) .sortBy(-_._2).collect.foreach(println) // or, unsorted pairRDD.countByKey().foreach(println) // All Countries pairRDD.keys() Pair RDDs With Cassandra_ 34 director name text K country text
- 35. •Joins can be expensive as they may require shuffling val directors = sc.cassandraTable(..) .map(r => (r.getString("name"),r)) val movies = sc.cassandraTable() .map(r => (r.getString("director"),r)) movies.join(directors) // RDD[(String, (CassandraRow, CassandraRow))] Pair RDDs With Cassandra - Join 35 director name text K country text movie title text K director text
- 36. •Automatically on read •Not automatically on write • No Shuffling Spark Operations -> Writes are local • Shuffeling Spark Operartions • Fan Out writes to Cassandra • repartitionByCassandraReplica(“keyspace“, “table“) before write •Joins with data locality Using Data Locality With Cassandra_ 36 sc.cassandraTable[CassandraRow](KEYSPACE, A) .repartitionByCassandraReplica(KEYSPACE, B) .joinWithCassandraTable[CassandraRow](KEYSPACE, B) .on(SomeColumns("id"))
- 37. •cassandraCount() • Utilizes Cassandra query • vs load the table into memory and do a count •spanBy(), spanByKey() • group data by Cassandra partition key • does not need shuffling • should be preferred over groupBy/groupByKey CREATE TABLE events (year int, month int, ts timestamp, data varchar, PRIMARY KEY (year,month,ts)); sc.cassandraTable("test", "events") .spanBy(row => (row.getInt("year"), row.getInt("month"))) sc.cassandraTable("test", "events") .keyBy(row => (row.getInt("year"), row.getInt("month"))) .spanByKey Further Transformations & Actions_ 37
- 38. Spark & Cassandra Demo 38
- 40. •Normal Scala Application •SBT as build tool •source in src/main/scala-2.10 •assembly.sbt in root and project directory •build.sbt in root directory •sbt assembly to build Scala Application_ 40 libraryDependencies += "com.datastax.spark" % "spark-cassandra-connector" % "1.3.0" libraryDependencies += "org.apache.spark" % "spark-core" % "1.3.1" % "provided" libraryDependencies += "org.apache.spark" % "spark-mllib_2.10" % "1.3.1" % "provided" libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.3.1" % "provided"
- 41. •Normal Java Application •Java 8! •MVN as build tool •source in src/main/java •in pom.xml • dependencies (spark-core, spark-streaming, spark-mllib, spark-cassandra-connector) • assembly-plugin or shade-plugin •mvn clean install to build Java Application_ 41
- 42. •Special classes for Java SparkConf conf = new SparkConf().setMaster("local[2]").setAppName("Java") .set("spark.cassandra.connection.host", "127.0.0.1"); JavaSparkContext sc = new JavaSparkContext(conf); JavaStreamingContext ssc = new JavaStreamingContext(conf, Durations.seconds(1L)); JavaRDD<Integer> rdd = sc.parallelize(Arrays.asList(1, 2, 3, 4, 5, 6)); rdd.filter(e -> e % 2 == 0).foreach(System.out::println); Java Specials_ 42
- 43. •Special classes for Java import static com.datastax.spark.connector.japi.CassandraJavaUtil.*; CassandraTableScanJavaRDD<CassandraRow> table = javaFunctions(sc.sparkContext()) .cassandraTable("keyspace", „table"); CassandraTableScanJavaRDD<Entity> table = javaFunctions(sc.sparkContext()) .cassandraTable("keyspace", "table",mapRowTo(Entity.class)) javaFunctions(someRDD).writerBuilder("keyspace", "table", mapToRow(Entity.class)).saveToCassandra(); Java Specials - Cassandra_ 43
- 44. Spark SQL 44
- 45. •SQL Queries with Spark (SQL & HiveQL) • On structured data • On DataFrame • Every result of Spark SQL is a DataFrame • All operations of the GenericRDD‘s available •Supports (even on non primary key columns) • Joins • Union • Group By • Having • Order By Spark SQL_ 45
- 46. val sqlContext = new SQLContext(sc) val persons = sqlContext.jsonFile(path) // Show the schema persons.printSchema() persons.registerTempTable("persons") val adults = sqlContext.sql("SELECT name FROM persons WHERE age > 18") adults.collect.foreach(println) Spark SQL - JSON Example_ 46 {"name":"Michael"} {"name":"Jan", "age":30} {"name":"Tim", "age":17}
- 47. val csc = new CassandraSQLContext(sc) csc.setKeyspace("musicdb") val result = csc.sql("SELECT country, COUNT(*) as anzahl" + "FROM artists GROUP BY country" + "ORDER BY anzahl DESC"); result.collect.foreach(println); Spark SQL - Cassandra Example_ 47
- 50. •Real Time Processing using micro batches •Supported sources: TCP, S3, Kafka, Twitter,.. •Data as Discretized Stream (DStream) •Same programming model as for batches •All Operations of the GenericRDD & SQL & MLLib •Stateful Operations & Sliding Windows Stream Processing With Spark Streaming_ 50
- 52. •Maintain State for each key in a DStream: updateStateByKey Spark Streaming - Stateful Operations_ 52 def updateAlbumCount(newValues: Seq[Int],runningCount: Option[Int]) : Option[Int] = { val newCount = runningCount.getOrElse(0) + newValues.size Some(newCount) } val countStream = stream.updateStateByKey[Int] (updateAlbumCount _) Stream is a DStream of Pair RDD's
- 53. •One Receiver -> One Node • Start more receivers and union them val numStreams = 5 val kafkaStreams = (1 to numStreams).map { i => KafkaUtils.createStream(...) } val unifiedStream = streamingContext.union(kafkaStreams) unifiedStream.print() •Received data will be split up into blocks • 1 block => 1 task • blocks = batchSize / blockInterval •Repartition data to distribute over cluster Spark Streaming - Parallelism_ 53
- 55. Spark MLLib 55
- 56. •Fully integrated in Spark • Scalable • Scala, Java & Python APIs • Use with Spark Streaming & Spark SQL •Packages various algorithms for machine learning •Includes • Clustering • Classification • Prediction • Collaborative Filtering •Still under development • performance, algorithms Spark MLLib_ 56
- 57. MLLib Example - Clustering_ 57 age set of data points meaningful clusters income
- 59. MLLib Example - Classification_ 59
- 60. MLLib Example - Classification_ 60
- 63. MLLib Example - Collaborative Filtering_ 63
- 66. Use Cases 66
- 67. •In particular for huge amounts of external data •Support for CSV, TSV, XML, JSON und other Use Cases for Spark and Cassandra_ 67 Data Loading case class User (id: java.util.UUID, name: String) val users = sc.textFile("users.csv") .repartition(2*sc.defaultParallelism) .map(line => line.split(",") match { case Array(id,name) => User(java.util.UUID.fromString(id), name)}) users.saveToCassandra("keyspace","users")
- 68. Validate consistency in a Cassandra database •syntactic • Uniqueness (only relevant for columns not in the PK) • Referential integrity • Integrity of the duplicates •semantic • Business- or Application constraints • e.g.: At least one genre per movies, a maximum of 10 tags per blog post Use Cases for Spark and Cassandra_ 68 Validation & Normalization
- 69. •Modelling, Mining, Transforming, .... •Use Cases • Recommendation • Fraud Detection • Link Analysis (Social Networks, Web) • Advertising • Data Stream Analytics ( Spark Streaming) • Machine Learning ( Spark ML) Use Cases for Spark and Cassandra_ 69 Analyses (Joins, Transformations,..)
- 70. •Changes on existing tables • New table required when changing primary key • Otherwise changes could be performed in-place •Creating new tables • data derived from existing tables • Support new queries •Use the CassandraConnectors in Spark Use Cases for Spark and Cassandra_ 70 Schema Migration
- 71. Thank you for your attention! 71
- 72. Questions? Matthias Niehoff, IT-Consultant 90 codecentric AG Zeppelinstraße 2 76185 Karlsruhe, Germany mobil: +49 (0) 172.1702676 matthias.niehoff@codecentric.de www.codecentric.de blog.codecentric.de matthiasniehoff