Run Hadoop on CentOS 7

Install Hadoop

• Yarn
• Hadoop version 2.7
• Java 1.7
• centos 7

$ cat /etc/redhat-release
CentOS Linux release 7.1.1503 (Core)

Stop firewalld

$ sudo systemctl stop firewalld

Installl Java

$ yum install java-1.7.0-openjdk

create hadoop user

$ useradd hadoop

switch to hadoop user

$ su - hadoop

generate SSH key

$ mkdir ~/.ssh
$ ssh-keygen -t rsa -b 4096 -C "[e-mail]"
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

chmod ssh kye

$ chmod 0600 ~/.ssh/authorized_keys

verify ssh login works and check no password is required

$ ssh localhost
$ exit

download hadoop 2.7.0

$ cd
$ wget http://apache.claz.org/hadoop/common/hadoop-2.7.0/hadoop-2.7.0.tar.gz
$ tar zxvf hadoop-2.7.0.tar.gz

Configure hadoop

Following command are execued by hadoop user

Set envionmental variable

# .bashrc
export JAVA_HOME=/usr/lib/jvm/jre
export HADOOP_HOME=/home/hadoop/hadoop-2.7.0
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native:$JAVA_LIBRARY_PATH

Initiate environmental variable

$ source $HOME/.bashrc

Put the property info between the 'configuration' tags for each file tgs for each file.

$HADOOP_HOME/etc/hadoop/core-site.xml

<property>
  <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
</property>

$HADOOP_HOME/etc/hadoop/hdfs-site.xml

<property>
 <name>dfs.replication</name>
 <value>1</value>
</property>
 
<property>
  <name>dfs.name.dir</name>
    <value>file:///home/hadoop/hadoopdata/hdfs/namenode</value>
</property>
 
<property>
  <name>dfs.data.dir</name>
    <value>file:///home/hadoop/hadoopdata/hdfs/datanode</value>
</property>

copy template

$ cp $HADOOP_HOME/etc/hadoop/mapred-site.xml.template $HADOOP_HOME/etc/hadoop/mapred-site.xml

$HADOOP_HOME/etc/hadoop/mapred-site.xml

<property>
  <name>mapreduce.framework.name</name>
   <value>yarn</value>
</property>

Edit $HADOOP_HOME/etc/hadoop/yarn-site.xml

<property>
  <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>

$HADOOP_HOME/etc/hadoop/hadoop-env.sh and add the following line

export JAVA_HOME=/usr/lib/jvm/jre

Start Hadoop

$ hdfs namenode -format

$ start-dfs.sh

$ start-yarn.sh

refs