Help us understand the problem. What is going on with this article?

OSXで擬似分散モードのApacheHadoop2.7.0を動かす

More than 3 years have passed since last update.

遊んでたクラスタが諸事情により使えなくなったから自分のマシン上に擬似分散モードのを動かしたメモ
基本的には HadoopAdventCalender2013 9日目 Hadoopのドキュメントを読む を参考

ダウンロード

http://www.apache.org/dyn/closer.cgi/hadoop/common/ から、hadoop-2.7.0.tar.gzをダウンロードする

適当なところに展開する。(適当に/usr/local2/ってディレクトリ掘った)

$ tar zxvf ~/Download/hadoop-2.7.0.tar.gz -C /usr/local2/

設定ファイルバックアップ取って必要なやつコピーする

$ mv /usr/local2/hadoop-2.7.0/etc/hadoop /usr/local2/hadoop-2.7.0/etc/hadoop-backup
$ cp /usr/local2/hadoop-2.7.0/etc/hadoop-backup/core-site.xml /usr/local2/hadoop-2.7.0/etc/hadoop
$ cp /usr/local2/hadoop-2.7.0/etc/hadoop-backup/hdfs-site.xml /usr/local2/hadoop-2.7.0/etc/hadoop
$ cp /usr/local2/hadoop-2.7.0/etc/hadoop-backup/yarn-site.xml /usr/local2/hadoop-2.7.0/etc/hadoop
$ cp /usr/local2/hadoop-2.7.0/etc/hadoop-backup/capacity-scheduler.xml /usr/local2/hadoop-2.7.0/etc/hadoop
$ cp /usr/local2/hadoop-2.7.0/etc/hadoop-backup/hadoop-env.sh /usr/local2/hadoop-2.7.0/etc/hadoop

/usr/local2/hadoop/etc 以下にコピーした設定ファイルに必要な分追記

/usr/local2/hadoop-2.7.0/etc/hadoop/core-site.xml
<property>
    <name>fs.defaultFS</name>
    <value>localhost:9000</value>
</property>
/usr/local2/hadoop-2.7.0/etc/hadoop/hdfs-site.xml
<property>
    <name>dfs.replication</name>
    <value>1</value>
</property>
/usr/local2/hadoop-2.7.0/etc/hadoop/yarn-site.xml
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>

JAVA_HOMEは/System/Library/Frameworks/JavaVM.framework/Versions/Current/Commands/java_homeの出力そのまま

/usr/local2/hadoop-2.7.0/etc/hadoop/hadoop-env.sh
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home
export HADOOP_CONF_DIR=/usr/local2/hadoop-2.7.0/etc/hadoop

初回起動前にnamenodeフォーマットする

$ /usr/local2/hadoop-2.7.0/bin/hdfs namenode -format

起動する

## DataNodeの起動
/usr/local2/hadoop-2.7.0/sbin/hadoop-daemon.sh --config /usr/local2/hadoop-2.7.0/etc/hadoop start datanode
## NameNodeの起動
/usr/local2/hadoop-2.7.0/sbin/hadoop-daemon.sh --config /usr/local2/hadoop-2.7.0/etc/hadoop start namenode
## ResourceManagerの起動
/usr/local2/hadoop-2.7.0/sbin/yarn-daemon.sh --config /usr/local2/hadoop-2.7.0/etc/hadoop start resourcemanager
## NodeManagerの起動
/usr/local2/hadoop-2.7.0/sbin/yarn-daemon.sh --config /usr/local2/hadoop-2.7.0/etc/hadoop start nodemanager
## JobHistoryServerの起動
/usr/local2/hadoop-2.7.0/sbin/mr-jobhistory-daemon.sh --config /usr/local2/hadoop-2.7.0/etc/hadoop start historyserver

jpsで確認

$ jps
42608 ApplicationMaster
33139 NodeManager
46486 Jps
311 
33273 ResourceManager
33499 NameNode
33388 ApplicationHistoryServer
33613 DataNode

Webインターフェース見る

動作確認

## 必要なディレクトリ作る
$ /usr/local2/hadoop-2.7.0/bin/hdfs dfs -mkdir /user
$ /usr/local2/hadoop-2.7.0/bin/hdfs dfs -mkdir /user/kodai_abe
$ /usr/local2/hadoop-2.7.0/bin/hdfs dfs -put /usr/local2/hadoop-2.7.0/etc/hadoop/hadoop-env.sh   

## wordcount走らせる
$ /usr/local2/hadoop-2.7.0/bin/yarn jar /usr/local2/hadoop-2.7.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.0.jar wordcount hadoop-env.sh wordcount_out  

## 出力結果を確認する
$ /usr/local2/hadoop-2.7.0/bin/hdfs dfs -ls wordcount_out 
2015-06-27 18:03:45,985 WARN  [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
-rw-r--r--   1 kodai_abe supergroup          0 2015-06-27 18:02 wordcount_out/_SUCCESS
-rw-r--r--   1 kodai_abe supergroup       3774 2015-06-27 18:02 wordcount_out/part-r-00000

追記

この環境でSpark動かした

この環境でFlink動かした

Why do not you register as a user and use Qiita more conveniently?
  1. We will deliver articles that match you
    By following users and tags, you can catch up information on technical fields that you are interested in as a whole
  2. you can read useful information later efficiently
    By "stocking" the articles you like, you can search right away
Comments
Sign up for free and join this conversation.
If you already have a Qiita account
Why do not you register as a user and use Qiita more conveniently?
You need to log in to use this function. Qiita can be used more conveniently after logging in.
You seem to be reading articles frequently this month. Qiita can be used more conveniently after logging in.
  1. We will deliver articles that match you
    By following users and tags, you can catch up information on technical fields that you are interested in as a whole
  2. you can read useful information later efficiently
    By "stocking" the articles you like, you can search right away