MapReduce provides an easy way to process large datasets in a distributed manner. It uses mappers to process input data and generate intermediate key-value pairs, and reducers to combine those intermediate pairs into the final output. Key aspects include job tracking, splitting data into tasks, and storing intermediate output locally rather than on HDFS for efficiency, since it is discarded after reducing.

1. MAP REDUCE: BASICS
1

2. MR: Processing
<key,
value>
<key, value>
MR Box/Job
MR: For distributed data processing (here discussing with reference to Hadoop).
Provides easy scalability.
Essential data processing units- Mapper and Reducer.
However other additional units can be introduced based on application.

3. MR: Key-Value Pairs
Following are the requirements for key-value pairs for MR Jobs:
Both the key and value classes need to be serializable. Which
implies they need to implement Writable interface.
Key class need to implement WritableComparable interface, to
enable the sorting.

4. MR: Detailed Processing
<key, value>
Map
<key, value>
Combine Reduce
<key, value> <key, value>

5. MR: Terminology
• Job: Execution of mapper/reducer program over a data set.
• Job Tracker: Schedules jobs and tracks the assign jobs to Task tracker.
• Task: Each job is divided into smaller tasks, thus it is execution of map/reduce on slice
of data.
• Task Tracker: Tracks the task and reports status to JobTracker.
• Task Attempt: Attempt to execute a task on slave node.
• Payload:

6. MR: Types of Nodes
• Named Node: Manages the Hadoop Distributed File System. JobTracker sits here.
• Data Node: Holds all the data required for processing. TaskTrackers sit here.
• Master Node: Executes JobTracker and accepts all the job requests from the client.
• Slave Node: Runs the mapper and reducer jobs/program.

7. MR: Process
• Map task executes map function on each split of the data. (Split
size is important)
• Output of the map job is written to the local disk not on HDFS
(to avoid replication). Since this is just an intermediately output
and is rejected once the entire process is done. Hence storing
on HDFS will cause replication.
• Output of map is input to reduce job (for simple Map-Reduce
model). Output of reduce (i.e. the final output) is stored on
HDFS (first on local node and then replicated on disk)

8. MR: References
• https://hadoop.apache.org/docs/current/hadoop-mapreduce-
client/hadoop-mapreduce-client-core/MapReduceTutorial.html
• https://www.guru99.com/introduction-to-mapreduce.html