Vgu bis2010 Mapreduce and Batch processing
•Télécharger en tant que PPTX, PDF•
0 j'aime•947 vues
Recommandé
Recommandé
Contenu connexe
Similaire à Vgu bis2010 Mapreduce and Batch processing
Similaire à Vgu bis2010 Mapreduce and Batch processing (20)
Dernier
Dernier (20)
Vgu bis2010 Mapreduce and Batch processing
- 1. MapReduce and Batch Processing VGU BIS2010, Group 13 Son Pham: phamtranthaison@gmail.com | Phong Le: bigbangvn@gmail.com | Lam Pham: lam.pts.vn@gmail.com | Chuong Nguyen: chuongit@gmail.com | Chapter 4
- 2. Content Part 1: Son Pham Batch Layer < Part 2: Phong Le > MapReduce Part 3: Lam Pham MapReduce < Part 4: Chuong Nguyen > Demo
- 4. Batch Layer • Precomputation • High latency • Linearly Scalable
- 6. Batch Layer – Linear Scalability “Scalability is the ability of a system to maintain performance under increased load by adding more resources”
- 7. Linear vs. Non-Linear Scalability Linear Scalability Non- Linear Scalability “A linearly scalable system can maintain performance under increased load by adding resources in proportion to the increased load”
- 8. MapReduce A distributed computing paradigm originally pioneered by Google Inspired by the “Map” and “Reduce” functions commonly used in functional programming (LISP) Operating on data stored in a distributed filesystem (HDFS…) A population free implementation is Apache Hadoop.
- 9. MapReduce
- 10. MapReduce - “Word count” Example
- 11. MapReduce Scalability Automatically parallelize the computation across the cluster of machines Fault-Tolerance Reassign failed tasks
Notes de l'éditeur
- In the Lambda Architecture, you precompute your queries so that at read time you can get your results quickly.
- In the batch layer, you run functions of your entire dataset to precompute your queries. Because it is running functions of the entiredataset, the batch layer is slow to update. However, the fact that it is looking at all the data at once means the batch layer can precompute query
- Balancing precomputation and on-the-fly computation:You can't always precompute. Instead, what you can do is precompute an intermediate result and then do somecomputation on the fly to complete queries based on those intermediate results
- Load in a Big Data context is a combination of the total amount of data you have, how much new data you receive every day, how many requests per second your application serves, and so on.
- "Map" step: The master node takes the input, divides it into smaller sub-problems, and distributes them to worker nodes. A worker node may do this again in turn, leading to a multi-level tree structure. The worker node processes the smaller problem, and passes the answer back to its master node."Reduce" step: The master node then collects the answers to all the sub-problems and combines them in some way to form the output – the answer to the problem it was originally trying to solve.The MapReduce library in the user program first shards the input files into M pieces of typically 16 megabytes to 64 megabytes (MB) per piece. It then starts up many copies of the program on a cluster of machines. One of the copies of the program is special: the master. The rest are workers that are assigned work by the master. There are M map tasks and R reduce tasks to assign. The master picks idle workers and assigns each one a map task or a reduce task. A worker who is assigned a map task reads the contents of the corresponding input shard. It parses key/value pairs out of the input data and passes each pair to the user-defined Map function. The intermediate key/value pairs produced by the Map function are buffered in memory. Periodically, the buffered pairs are written to local disk, partitioned into R regions by the partitioning function. The locations of these buffered pairs on the local disk are passed back to the master, who is responsible for forwarding these locations to the reduce workers.When a reduce worker is notified by the master about these locations, it uses remote procedure calls to read the buffered data from the local disks of the map workers. When a reduce worker has read all intermediate data, it sorts it by the intermediate keys so that all occurrences of the same key are grouped together. If the amount of intermediate data is too large to fit in memory, an external sort is used.The reduce worker iterates over the sorted intermediate data and for each unique intermediate key encountered, it passes the key and the corresponding set of intermediate values to the user's Reduce function. The output of the Reduce function is appended to a final output file for this reduce partition. When all map tasks and reduce tasks have been completed, the master wakes up the user program. At this point, the MapReduce call in the user program returns back to the user code. After successful completion, the output of the MapReduce execution is available in the R output files. [1]
- To detect failure, the master pings every worker periodically. If no response is received from a worker in a certain amount of time, the master marks the worker as failed. Any map tasks completed by the worker are reset back to their initial idle state, and therefore become eligible for scheduling on other workers. Similarly, any map task or reduce task in progress on a failed worker is also reset to idle and becomes eligible for rescheduling. Completed map tasks are re-executed when failure occurs because their output is stored on the local disk(s) of the failed machine and is therefore inaccessible. Completed reduce tasks do not need to be re-executed since their output is stored in a global fille system.