This presentation introduces the Big Data topic to Software Quality Assurance Engineers. It can also be useful for Software Developers and other software professionals.

1. Big Data for Quality Engineers
Ahmed Misbah

2. Agenda
• Introduction to Big Data
– Problem with traditional Large Scale Systems
– Requirements for the new approach
– Hadoop’s Approach
– Batch Processing and Steam Processing
• Big Data Technologies
– Batch Processing Technologies
– Stream Processing Technologies
• Testing Big Data Solutions

3. Rules
• Phones silent
• No laptops
• Questions/Discussions at anytime welcome
• 10 minute break every 1 hour

4. INTRODUCTION TO BIG DATA

5. PROBLEMS WITH TRADITIONAL
LARGE SCALE SYSTEMS

6. Traditional Large Scale Computing
• Traditionally, computation has been
processor-bound:
– Small amounts of data
– Lots of complex processing
• Early solution: Bigger computers!!
– Faster processor(s)
– More memory

8. Distributed Systems (1/3)
• More computers instead of bigger computers
• Distributed systems evolved
• Use multiple machines for a single job

9. Distributed Systems (2/3)
“In pioneer days they used oxen for heavy
pulling, and when one ox couldn’t budge a log,
we didn’t try to grow a larger ox. We shouldn’t
be trying for bigger computers, but for more
systems of computers”
Grace Hopper

10. Distributed Systems (3/3)

11. Problems with Distributed Systems
(1/2)
• Programming for traditional distributed
systems in complex:
– Keeping data and processes in sync
– Finite bandwidth
– Partial failures

12. Problems with Distributed Systems
(2/2)
“Failure is the defining difference between
distributed and local programming, so you
have to design distributed systems with the
expectation of failure”
Ken Arnold, CORBA Designer

13. The Data Bottleneck (1/4)
• Moore’s Law has held firm for over 40 years:
– Processing power doubles every two years
– Processing speed is no longer the problem
• Getting the data to the processor becomes the
bottleneck

14. The Data Bottleneck (2/4)
• Example:
– Typical disk data transfer rate: 75MB/sec
– Time taken to transfer 100GB of data to the
processor ≈ 22 minutes
– Actual time will be worse since most servers have
less than 100GB of RAM

15. The Data Bottleneck (3/4)
• Typically, data is stored in a central location
• Data is copied to the processors at runtime
• Acceptable for limited amounts of data

16. The Data Bottleneck (4/4)
• Modern system have much more data
– Terabytes/day
– Petabytes/year
• A new approach is required

17. REQUIREMENTS FOR THE NEW
APPROACH

18. Requirements for the new approach
(1/2)
• Partial failure support:
– Failure of a component should result in a graceful
degradation of the application performance
– It should not lead to a complete failure of the entire
system
• Data recoverability:
– If a component of the system fails, its workload should
be assumed by still-functioning units in the system
• Component recovery:
– If a component fails then recovers, it should be able to
rejoin the system without requiring full system restart

19. Requirements for the new approach
(2/2)
• Consistency:
– Component failures during execution of a job
should not affect the outcome of the job
• Scalability:
– Adding load to the system should result in graceful
degradation in performance and not the failure of
the entire system
– Increasing resources should support proportional
increase in load capacity

20. HADOOP’S APPROACH

21. A new approach to distributed
computing!
• Distribute data when the data is being stored
• Run computation where the data is stored

22. Core Concept (1/3)
• Distribute the data as it is initially stored in the
system
• Individual nodes can work on the data local to
those nodes
• No data transfer over the network is required
for initial processing

23. Core Concept (2/3)
• Applications are written in high-level code
• Developers need not to worry about network
programming or low-level infrastructure
• Nodes talk to each other as little as possible

24. Core Concept (3/3)
• Data is spread among machines in advance
• Computation happens where the data is
stored
• Data is replicated multiple times on the
system for increased availability and reliability

25. Fault Tolerance
• If a node fails, the master will detect the failure
and re-assign the work to a different node on the
system
• Restarting a task does not require the
communication with nodes working on other
portions of the data
• If a failed node restarts it is automatically added
back to the system and assigned a new task
• If a node appears to be running slowly, the
master can redundantly execute another instance
of the same task

27. BATCH PROCESSING VS STREAM
PROCESSING

28. Batch Processing
• Also known as History-based processing
• Processing is executed against large data
already stored in some storage medium (e.g.
HDFS or S3)

29. Stream Processing
• Processing executed against batches of data
coming continuously from a stream

30. BIG DATA TECHNOLOGIES

31. Batch Processing Technologies (1/2)
• Hadoop

32. Batch Processing Technologies (2/2)
• Spark

33. Stream Processing Technologies (1/2)
• Spark Streaming

34. Stream Processing Technologies (2/2)
• Apache Storm
• Apache Flink

35. Supporting Technologies
• Apache Kafka
• Akka

36. TESTING BIG DATA TECHNOLOGIES

37. Hadoop MapReduce (1/3)
• LocalJobRunner:
– Does not require any Hadoop daemons to be
running
– Uses the local file system instead of HDFS
• MRUnit:
– Built on top of JUnit
– Works with Mockito Framework to provide
required mock objects

38. Hadoop MapReduce (2/3)
• Apache Hue:
– Is an open source Web interface for analyzing data
with Apache Hadoop

39. Hadoop MapReduce (3/3)
• MapReduce Job Tracker Web Interface

40. Apache Spark (1/3)
• Run locally using Eclipse of IntelliJ
• Run using Spark Standalone
• Spark Testing Base:
– For implementing unit tests for Spark code
• Spark Validator:
– A library you can include in your Spark job to validate
the counters and perform operations on success

41. Apache Spark (2/3)
• Spark UI and History Server:

43. Apache Spark (3/3)
• Apache Zeppelin (using Sparklet on
Windows)

45. Performance Testing Tools
• Gatling
• Yahoo Cloud Serving Benchmark (YCSB)
• Jumbune
• Netflix Inviso
• TestDFSIO
• TeraSort
• NNBench
• MRbench
• BigBench

46. More tools
• https://github.com/Intel-bigdata/HiBench
• https://github.com/yahoo/streaming-
benchmarks
• https://github.com/tdas/spark-streaming-
benchmark
• https://github.com/BBVA/spark-benchmarks
• https://github.com/databricks/spark-perf

47. Monitoring Tools
• https://ambari.apache.org/
• https://github.com/groupon/sparklint
• https://github.com/linkedin/dr-elephant
• https://github.com/ibm-research-
ireland/sparkoscope
• https://supergloo.com/spark-
monitoring/spark-performance-monitoring-
tools/

48. Important Considerations
• Number of clusters/nodes
• Hardware Specifications (HDD or SSD)
• Application/Environment Configurations (no.
of cores, no. of partitions, no. of threads,
disk/memory persistence, etc.)
• Data format (Text, Sequence, Avro, etc.)
• Data size
• Compression (Snappy, Gzip, etc.)
• Number of Reducers (MapReduce)

49. Spark Best Practices
• https://medium.com/teads-engineering/spark-
performance-tuning-from-the-trenches-
7cbde521cf60
• https://dzone.com/articles/apache-spark-
performance-tuning-degree-of-parallel
• https://databricks.com/glossary/spark-tuning
• https://blog.cloudera.com/how-to-tune-your-
apache-spark-jobs-part-1/
• https://www.bi4all.pt/en/news/en-blog/apache-
spark-best-practices/

50. Sampling
• Sampling is defined as: “the act, process, or
technique of selecting a representative part of
a population for the purpose of determining
parameters or characteristics of the whole
population” - Merriam- Webster dictionary

51. Useful Resources (1/3)
• Benchmarking Hadoop and HBase on Violin
• Benchmarking Cassandra on Violin
• http://blog.cloudera.com/blog/2014/11/bigbe
nch-toward-an-industry-standard-benchmark-
for-big-data-analytics/

52. Useful Resources (2/3)
• http://blog.cloudera.com/blog/2015/08/ycsb-the-
open-standard-for-nosql-benchmarking-joins-cloudera-
labs/
• https://discuss.zendesk.com/hc/en-
us/articles/200864057-Running-DFSIO-MapReduce-
benchmark-test
• http://www.michael-
noll.com/blog/2011/04/09/benchmarking-and-stress-
testing-an-hadoop-cluster-with-terasort-testdfsio-
nnbench-mrbench/

53. Useful Resources (3/3)
• http://bdaafall2015.readthedocs.io/en/latest/
nnbench.html
• http://bdaafall2015.readthedocs.io/en/latest/
mrbench.html

54. Thank You!