This Edureka Hadoop Architecture Tutorial (Hadoop Architecture blog: https://goo.gl/hna58X) will help you understand the architecture of Apache Hadoop in detail. Below are the topics covered in this Hadoop Architecture Tutorial: 1) Hadoop Components 2) DFS – Distributed File System 3) HDFS Services 4) Blocks in Hadoop 5) Block Replication 6) Rack Awareness 7) HDFS Architecture 8) HDFS Read/Write Mechanisms 9) Hadoop HDFS Commands Check our complete Hadoop playlist here: https://goo.gl/ExJdZs

1. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING

2. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Agenda for today’s Session
1. Hadoop Components
2. DFS – Distributed File System
3. HDFS Services
4. Blocks in Hadoop
5. Block Replication
6. Rack Awareness
7. HDFS Architecture
8. HDFS Read/Write Mechanism

3. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Big Data Storage & Computation ?
Storing Big Data was a Problem
Even if a part of Big Data is
Stored- Processing it, took Years

4. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Hadoop Solves Big Data Problems
Storing Big Data was no more a Problem
And Processing did not take Years

5. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Hadoop has a Distributed File System.
But Why?

6. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
DFS – Distributed File System

7. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
DFS – Distributed File System

8. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Hadoop Components
2 main Hadoop Components
Storage Processing

9. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Hadoop 2.x Daemons

10. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
NameNode and DataNode
NameNode
• Master daemon
• Maintains and Manages DataNodes
• Records metadata e.g. location of blocks stored, the
size of the files, permissions, hierarchy, etc.
• Receives heartbeat and block report from all the
DataNodes
DataNode
• Slave daemons
• Stores actual data
• Serves read and write requests from the clients

11. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
A Quick Sneak peek of a
Hadoop Cluster Architecture

12. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Hadoop Cluster Architecture – Master Slave Topology
Master Machine
Slave Machines
distributed
over the cluster

13. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Let us talk about,
how data is stored in HDFS?

14. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
HDFS Blocks
• Each file is stored on HDFS as blocks
• The default size of each block is 128 MB in Apache Hadoop 2.x (64 MB in Apache Hadoop 1.x)
• Let us say, I have a file example.txt of size 248 MB. Below is the representation of how it will be
stored on HDFS

15. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
HDFS Blocks
• Each file is stored on HDFS as blocks
• The default size of each block is 128 MB in Apache Hadoop 2.x (64 MB in Apache Hadoop 1.x)
• Let us say, I have a file example.txt of size 248 MB. Below is the representation of how it will be
stored on HDFS
How many blocks will be created if a
file of size 514 MB is copied to HDFS ?

16. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
HDFS Blocks
• Each file is stored on HDFS as blocks
• The default size of each block is 128 MB in Apache Hadoop 2.x (64 MB in Apache Hadoop 1.x)
• Let us say, I have a file example.txt of size 248 MB. Below is the representation of how it will be
stored on HDFS
How many blocks will be created if a
file of size 514 MB is copied to HDFS ?

17. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Is it safe to have just 1 copy of each block?
What do you think?

18. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Hadoop Architecture – Block Replication

19. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
How does Hadoop decide
where to store the replica’s of the blocks created?

20. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Hadoop Architecture: Rack Awareness

21. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Hadoop Architecture: Rack Awareness

22. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Hadoop Architecture: Rack Awareness

23. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Hadoop Architecture: Rack Awareness

24. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Hadoop Architecture: Rack Awareness

25. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Hadoop Architecture: Rack Awareness

26. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Hadoop Architecture: Rack Awareness

27. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Architecture of HDFS

28. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
HDFS Architecture

29. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
HDFS Read/Write Mechanism

30. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
HDFS Write Mechanism – Pipeline Setup

31. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
HDFS Write Mechanism – Writing a Block

32. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
HDFS Write Mechanism - Acknowledgment

33. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
HDFS Multi-Block Write Mechanism
For Block A: 1A -> 2A -> 3A -> 4A
For Block B: 1B -> 2B -> 3B -> 4B -> 5B -> 6B

34. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
HDFS Read Mechanism

35. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Let us see few Hadoop/HDFS Commands

36. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Hadoop/HDFS Commands
Check the version of Hadoop
Command: hadoop version
Check HDFS Health
Command: hdfs fsck /

37. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Hadoop/HDFS Commands
List all the HDFS
files/directories
Command: hdfs dfs –ls /
Copy data from local system to
HDFS
Command: hdfs dfs –put
/home/edureka/test /user

38. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Hadoop/HDFS Commands
List all the HDFS commands and the syntax to use them
Command: hdfs dfs –help

39. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Learning Resources
 Hadoop Tutorial: www.edureka.co/blog/hadoop-tutorial
 HDFS Tutorial: www.edureka.co/blog/hdfs-tutorial
 Hadoop Architecture: www.edureka.co/blog/apache-hadoop-hdfs-architecture

40. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Thank You …
Questions/Queries/Feedback