LoginSignup
3
3

More than 5 years have passed since last update.

@moukuto(Yuya Yasue)

Hadoopクラスタを構築してみた(CentOS7)

Last updated at Posted at 2018-11-25

内容

以下構成のHadoopクラスタを構築した
・マスタノード1台、スレーブノード3台
・OS:CentOS Linux release 7.5.1804
・hadoop:3.1.1

Hadoop用ファイルシステム作成(スレーブノード)

・ディスク確認

[root@localhost ~]#  lsblk -a
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda               8:0    0   50G  0 disk
 sda1            8:1    0  500M  0 part /boot
 sda2            8:2    0 49.5G  0 part
  centos-root 253:0    0 45.6G  0 lvm  /
  centos-swap 253:1    0  3.9G  0 lvm  [SWAP]
sdb               8:16   0   50G  0 disk
sr0              11:0    1 1024M  0 rom

・パーティション作成

[root@localhost ~]# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x32350de9.

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-104857599, default 2048): 2048
Last sector, +sectors or +size{K,M,G} (2048-104857599, default 104857599): 104857599
Partition 1 of type Linux and of size 50 GiB is set

Command (m for help): p

Disk /dev/sdb: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x32350de9

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048   104855551    52426752   83  Linux
Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

・デバイスをフォーマット

[root@localhost ~]#  mkfs.xfs /dev/sdb1
meta-data=/dev/sdb1              isize=512    agcount=4, agsize=3276672 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=13106688, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=6399, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

・マウント

mkdir /Hadoop
mount /dev/sdb1 /Hadoop
[root@localhost ~]# df -T
Filesystem              Type     1K-blocks    Used Available Use% Mounted on
/dev/mapper/centos-root xfs       47781076 1443976  46337100   4% /
devtmpfs                devtmpfs   1928504       0   1928504   0% /dev
tmpfs                   tmpfs      1940480       0   1940480   0% /dev/shm
tmpfs                   tmpfs      1940480    9032   1931448   1% /run
tmpfs                   tmpfs      1940480       0   1940480   0% /sys/fs/cgroup
/dev/sda1               xfs         508588  288936    219652  57% /boot
tmpfs                   tmpfs       388096       0    388096   0% /run/user/1000
/dev/sdb1               xfs       52401156   32944  52368212   1% /Hadoop

・OS再起動時もmountされるように修正
「/etc/fstab」ファイルに
/dev/sdb1 /Hadoop xfs defaults 0 0
を追記する

※参考
https://qiita.com/aosho235/items/ad9a4764e77ba43c9d76#%E3%83%91%E3%83%BC%E3%83%86%E3%82%A3%E3%82%B7%E3%83%A7%E3%83%B3%E3%82%92%E5%88%87%E3%82%8B
https://kazmax.zpp.jp/linux_beginner/fdisk.html

Firewall,SELinux無効化

systemctl stop firewalld
systemctl disable firewalld
vi /etc/selinux/config ←「disabled」に変更

OS設定

・hostname設定
nmcli general hostname test.localdomain
・hosts設定
「/etc/hosts」ファイルに以下を追記

192.168.11.237 hadoopmaster.local
192.168.11.238 hadoopslave1.local
192.168.11.239 hadoopslave2.local
192.168.11.240 hadoopslave3.local

・Hadoopクラスタ利用ユーザー作成

useradd -m hadoop
echo hadoop | passwd hadoop --stdin
chown -R hadoop:hadoop /Hadoop ←hadoopユーザーにて書き込みできるよう権限変更

・マスタノードでの鍵作成

[hadoop@hadoopmaster ~]$ whoami
hadoop
[hadoop@hadoopmaster ~]$ ssh-keygen -t rsa -P '' -f /home/hadoop/.ssh/id_rsa
[hadoop@hadoopmaster ~]$ ls -l /home/hadoop/.ssh
total 8
-rw-------. 1 hadoop hadoop 1679 Nov 25 07:16 id_rsa
-rw-r--r--. 1 hadoop hadoop  407 Nov 25 07:16 id_rsa.pub

・スレーブノードへの公開鍵コピー
hadoopユーザーがマスタノードからスレーブノードにSSHログインできるようにするため、マスタノードの公開鍵を全スレーブノードにコピー

[hadoop@hadoopmaster ~]$ ssh-copy-id hadoop@hadoopslave1.local
/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/hadoop/.ssh/id_rsa.pub"
The authenticity of host 'hadoopslave1.local (192.168.11.238)' can't be established.
ECDSA key fingerprint is SHA256:/GfKYMZHk4GrIkT7q6cvY/DD4fxWHrQZVEoLay3U6UY.
ECDSA key fingerprint is MD5:50:56:37:b9:3b:a0:b7:12:bf:aa:e2:e3:14:4f:b9:e2.
Are you sure you want to continue connecting (yes/no)? yes
/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
hadoop@hadoopslave1.local's password: ←hadoop

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'hadoop@hadoopslave1.local'"
and check to make sure that only the key(s) you wanted were added.

Javaインストール

[root@hadoopmaster ~]# yum install -y java
[root@hadoopmaster ~]# yum install -y java-1.7.0-openjdk-devel
[root@hadoopmaster ~]# java -version
openjdk version "1.8.0_191"
OpenJDK Runtime Environment (build 1.8.0_191-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)

・環境変数設定
vi /home/hadoop/.bash_profileに以下を追記

export LANG=en_US.utf8
export JAVA_HOME=/usr/lib/jvm/jre
export HADOOP_HOME=/home/hadoop/hadoop-3.1.1
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native:$JAVA_LIBRARY_PATH

Hadoop設定

・インストール

[hadoop@hadoopmaster ~]$ pwd
/home/hadoop
[hadoop@hadoopmaster ~]$ wget http://ftp.tsukuba.wide.ad.jp/software/apache/hadoop/common/hadoop-3.1.1/hadoop-3.1.1.tar.gz
[hadoop@hadoopmaster ~]$ tar xzvf hadoop-3.1.1.tar.gz -C ./

・設定ファイル作成
①$HADOOP_HOME/etc/hadoop/core-site.xml
マスタノードの指定、データのI/Oのバッファサイズの指定など
「fs.default.name」の値に、「hdfs://hadoopmaster.local:9000」を指定

[hadoop@hadoopmaster ~]$ vi $HADOOP_HOME/etc/hadoop/core-site.xml
<configuration>
<property>
 <name>fs.default.name</name>
  <value>hdfs://hadoopmaster.local:9000</value>
</property>
</configuration>

②$HADOOP_HOME/etc/hadoop/yarn-site.xml
Hadoopクラスタ全体の資源管理を行うリソースマネージャの指定や、メモリ割り当て容量など
リソースマネージャにhadoopmaster.localを指定する

[hadoop@hadoopmaster ~]$ vi $HADOOP_HOME/etc/hadoop/yarn-site.xml
<configuration>
<property>
 <name>yarn.resourcemanager.hostname</name>
  <value>hadoopmaster.local</value>
</property>
<property>
 <name>yarn.nodemanager.aux-services</name>
 <value>mapreduce_shuffle</value>
</property>
</configuration>

③$HADOOP_HOME/etc/hadoop/hdfs-site.xml
レプリカ数の指定、HDFSのディレクトリの指定など
「dfs.replication」の値でレプリカ数を指定
「dfs.name.dir」の値でHDFSを構成するディレクトリのパス(マスタノード)
「dfs.data.dir」の値でHDFSを構成するディレクトリのパス(データノード)

[hadoop@hadoopmaster ~]$ vi $HADOOP_HOME/etc/hadoop/hdfs-site.xml
<configuration>
<property>
 <name>dfs.replication</name>
  <value>3</value>
</property>
<property>
 <name>dfs.name.dir</name>
  <value>file:///home/hadoop/hdfs/namenode</value>
</property>
</configuration>

[hadoop@hadoopslave1 ~]$ vi $HADOOP_HOME/etc/hadoop/hdfs-site.xml
<configuration>
<property>
 <name>dfs.replication</name>
  <value>3</value>
</property>
<property>
 <name>dfs.data.dir</name>
  <value>file:///Hadoop/hdfs/datanode</value>
</property>
</configuration>

④$HADOOP_HOME/etc/hadoop/mapred-site.xml
分散処理の仕組みであり「MapReduce」の各種パラメータの設定など
CPUやメモリなどのHW資源のスケジューリングや分散処理基板向けのアプリケーション開発のためのフレームワークであるYARNを、分散処理のフレームワークとして指定する

[hadoop@hadoopmaster ~]$ vi $HADOOP_HOME/etc/hadoop/mapred-site.xml
<configuration>
<property>
 <name>mapreduce.framework.name</name>
 <value>yarn</value>
</property>
</configuration>

・slavesファイルの作成
スレーブノードのhost名を記載したファイルを作成する

[hadoop@hadoopmaster ~]$ vi $HADOOP_HOME/etc/hadoop/slaves
hadoopslave1.local
hadoopslave2.local
hadoopslave3.local

Hadoop起動

・HDFSのフォーマット(マスタノード)

[hadoop@hadoopmaster ~]$ which hdfs
~/hadoop-3.1.1/bin/hdfs
[hadoop@hadoopmaster ~]$ hdfs namenode -format

・namenodeの起動(マスタノード)

[hadoop@hadoopmaster ~]$ hadoop-daemon.sh start namenode
WARNING: Use of this script to start HDFS daemons is deprecated.
WARNING: Attempting to execute replacement "hdfs --daemon start" instead.

hdfs --daemon startコマンドが推奨されているらしい
・resoucemanagerの起動(マスタノード)

[hadoop@hadoopmaster ~]$ yarn-daemon.sh start resourcemanager
WARNING: Use of this script to start YARN daemons is deprecated.
WARNING: Attempting to execute replacement "yarn --daemon start" instead.

yarn --daemon startコマンドが推奨されているらしい

・HDFSメタデータの確認(マスタノード)

[hadoop@hadoopmaster ~]$ ls -l /home/hadoop/hdfs/namenode/current/
total 16
-rw-rw-r--. 1 hadoop hadoop 391 Nov 25 08:59 fsimage_0000000000000000000
-rw-rw-r--. 1 hadoop hadoop  62 Nov 25 08:59 fsimage_0000000000000000000.md5
-rw-rw-r--. 1 hadoop hadoop   2 Nov 25 08:59 seen_txid
-rw-rw-r--. 1 hadoop hadoop 217 Nov 25 08:59 VERSION

・データノードの起動(スレーブノード)

[hadoop@hadoopslave1 ~]$ hadoop-daemon.sh start datanode
WARNING: Use of this script to start HDFS daemons is deprecated.
WARNING: Attempting to execute replacement "hdfs --daemon start" instead.
[hadoop@hadoopslave1 ~]$ ls -l /Hadoop/hdfs/datanode/
total 4
drwxrwxr-x 3 hadoop hadoop 70 Nov 25 09:22 current
-rw-rw-r-- 1 hadoop hadoop 23 Nov 25 09:22 in_use.lock

hdfs --daemon startコマンドが推奨されているらしい
namenodeと起動コマンド一緒？

・nodemanagerの起動(スレーブノード)

[hadoop@hadoopslave1 ~]$ yarn-daemon.sh start nodemanager
WARNING: Use of this script to start YARN daemons is deprecated.
WARNING: Attempting to execute replacement "yarn --daemon start" instead.

yarn --daemon startコマンドが推奨されているらしい
resourcemanagerと同じコマンド？

・HDFS確認(マスタノード)

[hadoop@hadoopmaster ~]$ hdfs dfsadmin -report
Configured Capacity: 160978448384 (149.92 GB)
Present Capacity: 160876912640 (149.83 GB)
DFS Remaining: 160876900352 (149.83 GB)
DFS Used: 12288 (12 KB)
DFS Used%: 0.00%
Replicated Blocks:
        Under replicated blocks: 0
        Blocks with corrupt replicas: 0
        Missing blocks: 0
        Missing blocks (with replication factor 1): 0
        Pending deletion blocks: 0
Erasure Coded Block Groups:
        Low redundancy block groups: 0
        Block groups with corrupt internal blocks: 0
        Missing block groups: 0
        Pending deletion blocks: 0

-------------------------------------------------
Live datanodes (3): ←現在稼働中のノード数

Name: 192.168.11.238:9866 (hadoopslave1.local)
Hostname: hadoopslave1.local
Decommission Status : Normal
Configured Capacity: 53658783744 (49.97 GB)
DFS Used: 4096 (4 KB)
Non DFS Used: 33845248 (32.28 MB)
DFS Remaining: 53624934400 (49.94 GB)
DFS Used%: 0.00%
DFS Remaining%: 99.94%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Sun Nov 25 09:31:36 EST 2018
Last Block Report: Sun Nov 25 09:22:42 EST 2018
Num of Blocks: 0
以下略

・スレーブノードの確認(マスタノード)

[hadoop@hadoopmaster ~]$ yarn node -list
2018-11-25 09:35:21,739 INFO client.RMProxy: Connecting to ResourceManager at hadoopmaster.local/192.168.11.237:8032
Total Nodes:3
         Node-Id             Node-State Node-Http-Address       Number-of-Running-Containers
hadoopslave1.local:41836                RUNNING hadoopslave1.local:8042                            0
hadoopslave3.local:36843                RUNNING hadoopslave3.local:8042                            0
hadoopslave2.local:39948                RUNNING hadoopslave2.local:8042                            0

・Webコンソール
http://192.168.11.237:8088/cluster

テスト

・データ作成

[hadoop@hadoopslave1 ~]$ mkdir localdir01
[hadoop@hadoopslave1 ~]$ cat find.sh
#!/bin/sh
n=1
for i in `find /usr/share/doc -type f`;
do
 cp -a $i /home/hadoop/localdir01/`basename ${i}_${n}`; n=`expr ${n} + 1`;
done

・コピー先ディレクトリ作成

[hadoop@hadoopslave1 ~]$ hdfs dfs -ls /
[hadoop@hadoopslave1 ~]$ hdfs dfs -mkdir -p /user/hadoop/datadir01
[hadoop@hadoopslave1 ~]$ hdfs dfs -ls /user/hadoop
Found 1 items
drwxr-xr-x   - hadoop supergroup          0 2018-11-25 09:57 /user/hadoop/datadir01

・データのコピー
hdfs dfs -put /home/hadoop/localdir01/* /user/hadoop/datadir01/
・コピーしたファイルの確認
hdfs dfs -ls /user/hadoop/datadir01

3
3
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
3
3