Valentyn Kropov, Big Data Solutions Architect has recently attended "Hadoop World / Strata" – biggest and coolest Big Data conference in a World, and he can't wait to share fresh trends and topics straight from New-York. Come and learn how Hadoop cluster will help NASA to explore Mars, how Netflix build 10PB platform, what are the latest trends in Spark, to learn about newest, just announced storage engine from Cloudera called Kudu and many many more interesting stuff.

1. Trends and Topics
Valentyn Kropov
Solutions Architect, SAG, SoftServe

2. Agenda
1. Conference Overview
2. Bright Future of Hadoop Map-Reduce
3. Apache Spark Data Frames
4. Cloudera Kudu
5. Most Popular Reference Architecture
6. Use Cases

3. #1
Conference Overview

7. #2
Bright Future of
Hadoop MapReduce

9. Spark is a Future
Cloudera Anounces One Platform Initiative (Sep, 9 2015)

10. Spark is a Present
It appeared in 72% of
presentations and use-cases
At Hadoop World Conference

11. Spark is Easier to Code
Map Reduce / Java Spark / Scala

12. Spark is Faster
Up to 100x
faster!

13. Spark is Interactive

14. Spark is Real-Time

15. And they have Power 
• 400 contributors
• From 100+ companies
• Databricks (1 y.o, 30->100
people, $47 million)
• Cloudera (370 patches, 43k
lines of code)

16. Cloudera One Platform:
Read More
http://goo.gl/jSK0h6

17. #3
Spark Data Frames

18. Most of Data is Still
Structured!
• No Sorting?
• No Joins?
• No Aggregations?
• No Filtering?
• No cross-DB connections?

19. Data Frame is…
• API
• like a Table (RDBMS)
• or Data Frame (Python/R)
• Abstraction layer over RDD

20. Construct Data Frame
# Constructs a DataFrame from Hive
users = context.table("users")
# from JSON files in S3
logs = context.load("s3n://data.json", "
json")

21. Filtering
# Create a new DataFrame that
contains “young users” only
young = users.filter(users.age < 21)

22. Group By
# Count the number of young
users by gender
young.groupBy("gender").count()

23. Joins!
# Join users with another
DataFrame called logs
users.join(logs, logs.userId ==
users.userId, "left_outer")

24. Spark Languages
Spark Survey 2015

25. Why Not Python + RDD?

26. Data Frames and Python
• Compiled into JVM bytecode
• Data Never Leaves the JVM
• Python passes commands only
• Commands are pushed down

27. Data Frames Performance

28. Data Frames: Read More
http://www.slideshare.net/J
onHaddad/enter-the-snake-
pit-for-fast-and-easy-spark

29. #4
Cloudera’s Kudu

30. What’s Kudu?
• Columnar Storage for Hadoop
• Not just a file-format
• Supports low-latency random access (ms)
• Good alternative for Impala + Parquet
• Integrates with Spark, Hadoop, Impala
• It’s in Beta now

31. Faster than Parquet

32. Kudu: Architecture

33. Kudu: use-cases
• Write: newly-arrived data immediately
available to users
• Time-Series applications which needs to
support both random and scattered
reads

34. Kudu: Read More
http://getkudu.io/

35. #5
Most Popular
Reference Architecture

36. Reference Architecture
Yarn (90%)
Mesos (10%)

37. Kafka
• Highly-scalable
• Fault-tolerant (commit-log)
• Partition-based load-balancing

38. Spark Streaming
• Processes data in micro batches (Dstream,
windows slides)
• Supports data locality with Cassandra
• Real-time data science (Data Frames, Mlib)
• BI Support (Spark SQL)

39. Cassandra
• No SPOF
• Masterless (easy operations and scaling)
• Replicates data across data-centers
• Most mature and fast growing
• Evolves into New SQL (transactions)
• SQL-like-CQL

40. Spark
• Is Awesome for Analytics (both
real-time and batch)

41. Reference Architecture:
Read More
http://www.datastax.com/d
ev/blog/streaming-big-data-
with-spark-spark-streaming-
kafka-cassandra-and-akka

42. #6
Netflix Big Data
Platform

43. Netflix: Size
•20PB DW on S3
•Read ~10% of data daily
•Write ~10% of read data
daily
•500 billion events daily

44. Netflix: Analyze
•300 Data Scientists
•Python, R, Scala, etc

45. Netflix: Compute and
Storage
• Separate Compute and
storage (S3)
• To have heterogeneous
clusters
• And no-downtime upgrades

46. Netflix: Architecture

47. Netflix: Read More
http://strataconf.com/big-
data-conference-ny-
2015/public/schedule/detail
/43373

48. #7
Big Data
Mission to Mars

49. Mission Orion

50. Mission Orion: Size
• 350k measurands
• 2TB / hour
• 1200 telemetry sensors
• 3 x 1GB networks busy
• Data retention is 25 years 

51. Data Reader/Simulator Ingest
Packet
Measurands
(GPBs)
Kafka
Message
Bus
Packet
Measurands
(GPBs)
Deduplica
tion
(Spark)
HBase
Writer
(Spark)
mach5-
sample Obj
Splitter +
Decom
(GDS)
C++ client Reads
Packets and
Decommutates
Tlm
Data
Packet Measurands
GPB File
(represents a Packet(s)
and contains
decommutated
measurands)
Header Metadata
apid:seqctr:time: value1
…
..
apid:seqctr:time: valueN
mach5-
sample
(Spark)
Packet
Measurands
(GPBs)
Lockheed Martin Proprietary Information
Storage
Analytics
HDFS
HFiles
(HBase-RDD)
Mach-5 Data Ingest for Orion
HBase
Web/UITomcat
Glassfish
Etc.
Trace
FOSS
widgets
Aggregati
on
(Spark)
Alerting
(Spark)
Limit
Checking
(Spark)

52. Orion: Read More
http://strataconf.com/big-
data-conference-ny-
2015/public/schedule/detail
/43181

53. Thanks!
valentine.kropov@gmail.com