To know more, Register for Online Hadoop Training at WizIQ. Click here : http://www.wiziq.com/course/21308-hadoop-big-data-training A complete guide to Hadoop Installation that will help you when ever you face problems while installing Hadoop !!

1. A GUIDE TO HADOOP
INSTALLATION
BY WizIQ

2. Hadoop Installation
• Putty connectivity
• Java Installation (openjdk-7-jdk)
• Group/User Creation &SSH Certification
• Hadoop Install (Hadoop 2.2.0)
• Hadoop configuration
• Hadoop Services

3. Putty Connectivity
User ununtu@<name>
 Make sure you have SSHAuth
browse to .ppk file(location)

5. Java Installation
• ubuntu@ip-10-45-133-21:~$ sudo apt-get
install openjdk-7-jdk
• Error : Err http://us-east-
1.ec2.archive.ubuntu.com/ubuntu/ precise-
updates/main liblvm2app2.2 amd64
2.02.66-4ubuntu7.1
403 Forbidden] ……
• Solution: ubuntu@ip-10-45-133-21:~$
sudo apt-get update
ubuntu@ip-10-45-133-21:~$ sudo apt-get
install openjdk-7-jdk
ubuntu@ip-10-45-133-21:~$ cd /usr/lib/jvm
• Error:
ubuntu@ip-10-45-133-21:/usr/lib/jvm$ ln -s
java-7-openjdk-amd64 jdk
ln: failed to create symbolic link `jdk':
Permission denied
ubuntu@ip-10-45-133-21:/usr/lib/jvm$
• Solution:
ubuntu@ip-10-45-133-21:/usr/lib/jvm$sudo su
root@ip-10-45-133-21:/usr/lib/jvm# ln -s java-
7-openjdk-amd64 jdk

6. root@ip-10-45-133-21:/usr/lib/jvm# sudo apt-get install openssh-server

7. Group/User Creation &SSH Certification
• ubuntu@ip-10-45-133-21$ sudo addgroup
hadoop
• ubuntu@ip-10-45-133-21$ sudo adduser
--ingroup hadoop hduser
• ubuntu@ip-10-45-133-21$ sudo adduser
hduser sudo
**After user is created, re-login into ubuntu
using hduser
ubuntu@ip-10-45-133-21:~$ su –l hduser
Password: Setup SSH Certificate
hduser@ip-10-45-133-21$ ssh-keygen -t rsa -P ''
...
Your identification has been saved in
/home/hduser/.ssh/id_rsa.
Your public key has been saved in
/home/hduser/.ssh/id_rsa.pub.
...
hduser@ip-10-45-133-21$ cat ~/.ssh/id_rsa.pub
>> ~/.ssh/authorized_keys
hduser@ip-10-45-133-21$ ssh localhost
**

8. **Note: I have ignored the password here for %id_rsa:

9. Hadoop Install(Hadoop 2.2.0)
• hduser@ip-10-45-133-21:~# su -l hduser
• hduser@ip-10-45-133-21$ cd ~
• hduser@ip-10-45-133-21$ wget
http://www.trieuvan.com/apache/hadoop/c
ommon/hadoop- hduser@ip-10-45-133-
212.2.0/hadoop-2.2.0.tar.gz
• hduser@ip-10-45-133-21$ sudo tar vxzf
hadoop-2.2.0.tar.gz -C /usr/local
• hduser@ip-10-45-133-21$ cd /usr/local
• hduser@ip-10-45-133-21$ sudo mv hadoop-
2.2.0 hadoop
• hduser@ip-10-45-133-21$ sudo chown -R
hduser:hadoop hadoop

10. Hadoop Configuration
• Setup Hadoop Environment Variables:
• hduser@ip-10-45-133-21$cd ~
• hduser@ip-10-45-133-21$vi .bashrc
Copy & Paste following to the end of the file:
#Hadoop variables
export JAVA_HOME=/usr/lib/jvm/jdk/
export HADOOP_INSTALL=/usr/local/hadoop
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
###end of paste
hduser@ip-10-45-133-21$ cd /usr/local/hadoop/etc/hadoop
hduser@ip-10-45-133-21$ vi hadoop-env.sh
#modify JAVA_HOME
export JAVA_HOME=/usr/lib/jvm/jdk/

11. ****Re-login into Ubuntu using hdser and
check hadoop version******
hduser@ip-10-45-133-21$ hadoop version
Hadoop 2.2.0
Subversion
https://svn.apache.org/repos/asf/hadoop/
common -r 1529768
Compiled by hortonmu on 2013-10-
07T06:28Z
Compiled with protoc 2.5.0
From source with checksum
79e53ce7994d1628b240f09af91e1af4
This command was run using
/usr/local/hadoop-
2.2.0/share/hadoop/common/hadoop-
common-2.2.0.jar
At this point, Hadoop is installed.

12. Configure Hadoop :
hduser@ip-10-45-133-21$ cd /usr/local/hadoop/etc/hadoop
hduser@ip-10-45-133-21$ vi core-site.xml

13. #Paste following between <configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
hduser@ip-10-45-133-21$ vi yarn-site.xml
#Paste following between <configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
hduser@ip-10-45-133-21$ mv mapred-site.xml.template mapred-site.xml
hduser@ip-10-45-133-21$ vi mapred-site.xml

14. • #Paste following between <configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
hduser@ip-10-45-133-21$ cd ~
hduser@ip-10-45-133-21$ mkdir -p mydata/hdfs/namenode
hduser@ip-10-45-133-21$ mkdir -p mydata/hdfs/datanode
hduser@ip-10-45-133-21$ cd /usr/local/hadoop/etc/hadoop
hduser@ip-10-45-133-21$ vi hdfs-site.xml
• #Paste following between <configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/home/hduser/mydata/hdfs/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/home/hduser/mydata/hdfs/datanode</value>
</property>

15. Format Namenode :
hduser@ip-10-45-133-21$ hdfs namenode -format

16. Hadoop Services
hduser@ip-10-45-133-21$ start-dfs.sh
....
hduser@ip-10-45-133-21$ start-yarn.sh
....
Error:

17. Tried checking again:
1) SSH Certification creation
2) ssh localhost
3) vi hadoop-env.sh
4) vi hdfs-site.xml
5) Namenode and datanode directories
6) reformatted namenode
7) Started the services
That’s it after that it started working as below screens. Not sure what fixed the issue
as observed in next slide
• Error1: WARN util.NativeCodeLoader: Unable to load native-hadoop library for
your platform... using builtin-java classes where applicable ….
• Error2: Localhost: Permission Denied (Publickey)…

19. • hduser@ip-10-45-133-21:~$ jps
• All went on successfully and could see
below services running.

20. • Reconnected and works fine

21. THANK YOU!
Enroll now for Hadoop and Big Data Training @ WizIQ.com
Visit: http://www.wiziq.com/course/21308-hadoop-big-data-training
For more information, feel free to contact us at courses@wiziq.com