Mac On CDH Hadoop

前提条件

• JDK1.7インストール済み
• brewインストール済み

tap追加

$ brew tap hammer/cdh

HDFS/MapReduceインストール

$ brew install cdh-hadoop
$ brew install cdh-mr1

localhostへの鍵認証

systemsetup -f -setremotelogin on
ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys
for host_id in localhost 0.0.0.0; do
  ssh-keyscan $host_id >> ~/.ssh/known_hosts
done

DFS起動

$ `brew --cellar`/cdh-hadoop/4.2.1/bin/hdfs namenode -format
$ `brew --cellar`/cdh-hadoop/4.2.1/libexec/sbin/start-dfs.sh
$ jps

MapReduce起動

$ export HADOOP_MAPRED_HOME=`brew --cellar`/cdh-mr1/4.2.1/libexec
$ `brew --cellar`/cdh-mr1/4.2.1/bin/start-mapred.sh
$ jps

HDFSテスト

`brew --cellar`/cdh-hadoop/4.2.1/bin/hadoop fs -mkdir input
`brew --cellar`/cdh-hadoop/4.2.1/bin/hadoop fs -put `brew --cellar`/cdh-mr1/4.2.1/libexec/conf/*.xml input

MapReduceテスト

$ `brew --cellar`/cdh-mr1/4.2.1/bin/hadoop jar `brew --cellar`/cdh-mr1/4.2.1/libexec/hadoop-examples-2.0.0-mr1-cdh4.2.1.jar grep input output 'dfs[a-z.]+'