LoginSignup
11

More than 5 years have passed since last update.

Amazon EMRのCLIスニペット集

Last updated at Posted at 2015-03-16

このポストは個人のメモであり、私の所属する組織を代表するものではありません。

概要

EMRのスニペット、いつも同じようなものを書いてしまうのでここにまとめていくことにする。

SparkとPrestoをインストールされたClusterを起動する

install_spark_and_presto.sh
aws emr create-cluster \
--name Spark_and_Presto \
--ami-version 3.4.0 \
--applications Name=Hive Name=Hue \
--instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=c3.xlarge InstanceGroupType=CORE,InstanceCount=2,InstanceType=c3.xlarge \
--ec2-attributes SubnetId=YOUR_SUBNET,KeyName=YOUR_KEYPAIR \
--log-uri s3://PATH/TO/LOG/ \
--enable-debugging \
--bootstrap-actions \
Name=Install_Spark,Path=s3://support.elasticmapreduce/spark/install-spark \
Name=Install_Presto,Path=s3://beta.elasticmapreduce/bootstrap-actions/presto/install-presto.rb

既存のクラスタにHiveのジョブステップを追加する

add_hive_step.sh
aws emr add-steps --cluster-id YOUR_CLUSTER_ID --steps Type=HIVE,Name='Hive program',ActionOnFailure=CONTINUE,Args=[-f,s3://PATH/TO/YOUR/QUERY.q]

S3上のCloudFrontのログを外部テーブルとしてマウントするCREATE TABLE

create_external_table_for_cflog_on_s3.q
CREATE EXTERNAL TABLE cflog (
  date string,
  time string,
  x_edge_location string,
  sc_bytes string,
  c_ip string,
  cs_method string,
  cs_host string,
  cs_uri_stem string,
  sc_status string,
  cs_referer string,
  cs_ua string,
  cs_uri_query string,
  cs_cookie string,
  x_edge_result_type string,
  x_edge_request_id string,
  x_host_header string,
  cs_protocol string,
  cs_bytes string,
  time_taken string
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
LOCATION 's3://PATH/TO/LOGS/';

AmplabのBigdata BenchmarkのUservisitsテーブルを作成

create_external_table_for_uservisits
CREATE EXTERNAL TABLE uservisits (
  sourceIP STRING,
  destURL STRING,
  visitDate STRING,
  adRevenue STRING,
  userAgent STRING,
  countryCode STRING,
  languageCode STRING,
  searchWord STRING,
  duration STRING
) 
ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n'
LOCATION 's3://big-data-benchmark/pavlo/text/tiny/uservisits/';

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
11