Tachyon and Apache Spark
•
27 j'aime•14,593 vues
Meet Tachyon: an exciting new project trying to unify in-memory computing and provide a universal data-exchange layer for Hadoop ecosystem projects.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
En vedette (13)
Similaire à Tachyon and Apache Spark
Similaire à Tachyon and Apache Spark (20)
Plus de rhatr
Plus de rhatr (7)
Dernier
Dernier (20)
Tachyon and Apache Spark
- 1. Tachyon and Apache Spark: heralds of in-memory computing era. Roman Shaposhnik Director of Open Source @Pivotal (Twitter: @rhatr)
- 2. Who’s this guy? • Director of Open Source @Pivotal • Apache Software Foundation guy (Member, VP of Apache Incubator, committer on Hadoop, Giraph, Sqoop, etc) • Used to be root@Cloudera • Used to be PHB@Yahoo! (original Hadoop team)
- 4. 20 minute to figure out Hadoop vs. Spark
- 5. 20 minute to figure out Hadoop++ == Spark
- 6. 20 minute to figure out Hadoop + Spark
- 7. But wait! There’s more! Tachyon
- 8. Long, long time ago… HDFS ASF Projects FLOSS Projects Pivotal Products MapReduce
- 9. In a blink of an eye MLib Shark GraphX Streaming HDFS Crunch Mahout Pig Sqoop Flume Coordination and workflow management Zookeeper Command Center ASF Projects FLOSS Projects Pivotal Products GemFire XD Oozie MapReduce Hive Tez Giraph Hadoop UI Hue SolrCloud Phoenix HBase Spark Impala HAWQ SpringXD MADlib Hamster PivotalR YARN Tachyon
- 10. A Spark view? HDFS MLib Shark YARN GraphX Streaming Tachyon Sqoop Flume Hadoop UI Hue Coordination and workflow management Zookeeper Command Center ASF Projects FLOSS Projects Pivotal Products GemFire XD Oozie SolrCloud Phoenix HBase Spark SpringXD
- 11. BDAS
- 12. Long, long time ago…
- 13. This is 2014
- 14. What changed?
- 15. Your datacenter … server 1 server N
- 16. Hadoop’s view MapReduce server 1 server N HDFS
- 17. HDFS: decoupled storage … MR HDFS MR
- 19. Anatomy of MapReduce HDFS mappers reducers HDFS a b c d a c a 3 b 1 c 2 a 1 b 1 c 1 a 1 c 1 a 1 a 1 1 1 b 1 c 1 1
- 20. What’s wrong with MR? Source: UC Berkeley Spark project (just the image)
- 21. This looks familiar… $ grep –R | awk | sort …
- 22. Spark innovations • Resilient Distribtued Datasets (RDDs) • Distributed on a cluster • Manipulated via parallel operators (map, etc.) • Automatically rebuilt on failure • A parallel ecosystem • A solution to iterative and multi-stage apps
- 23. RDDs warnings = textFile(…).filter(_.contains(“warning”)) .map(_.split(‘ ‘)(1)) HadoopRDD path = hdfs:// FilteredRDD contains… MappedRDD split…
- 24. Parallel operators • map, reduce • sample, filter • groupBy, reduceByKey • join, leftOuterJoin, rightOuterJoin • union, cross
- 25. What is really happening? MLib Shark GraphX Streaming HDFS Crunch Mahout Pig Sqoop Flume Coordination and workflow management Zookeeper Command Center ASF Projects FLOSS Projects Pivotal Products GemFire XD Oozie MapReduce Hive Tez Giraph Hadoop UI Hue SolrCloud Phoenix HBase Spark Impala HAWQ SpringXD MADlib Hamster PivotalR YARN Tachyon
- 26. May be its not so bad server 1 server N
- 27. But HDFS/YARN are safe? HDFS, Ceph, S3, NAS, etc. New HDFS New YARN
- 28. Tachyon • In-memory data-exchange layer • A set of evolving APIs: • filesystem • caching • RDDs • Materialized views
- 29. Tachyon
- 30. Spark is best for cloud
- 31. It will be called Hadoop MLib Shark GraphX Streaming HDFS Crunch Mahout Pig Sqoop Flume Coordination and workflow management Zookeeper Command Center ASF Projects FLOSS Projects Pivotal Products GemFire with Tachyon Oozie MapReduce Hive Tez Giraph Hadoop UI Hue SolrCloud Phoenix HBase Spark Impala HAWQ SpringXD MADlib Hamster PivotalR YARN
- 32. Spark/Tachyon recap • Is it “Big Data” (Yes) • Is it “Hadoop” (No) • It’s one of those “in memory” things, right (Yes) • JVM, Java, Scala (All) • Is it Real or just another shiny technology with a long, but ultimately small tail (Yes and ?)
- 33. A NEW PLATFORM FOR A NEW ERA
- 34. Questions ?