EMRでRDSのMySQLを外部メタストアに指定するとTerminateされる #aws #emr #AWS

環境

EMR: emr-4.3.0
RDS: MySQL 5.6.27

現象

Configurationsに以下を指定してEMRを起動すると、Terminated with errorsOn the master instance (i-xxxxxxxx), application provisioning failed が発生してTerminateされる。

Configurations:

[
  {
    "classification": "hive-site",
    "properties": {
        "hive.exec.dynamic.partition.mode": "nonstrict",
        "javax.jdo.option.ConnectionURL": "jdbc:mysql://xxx.local:3306/hive",
        "javax.jdo.option.ConnectionDriverName": "com.mysql.jdbc.Driver",
        "javax.jdo.option.ConnectionUserName": "xxx",
        "javax.jdo.option.ConnectionPassword": "xxx"
    }
  }
]

マスターのプロビジョニング標準エラー出力に以下のエラーが出力される。

s3://aws-logs-xxxxxxxxxxxx-ap-northeast-1/elasticmapreduce/j-xxxxxxxxxxxxx/node/i-xxxxxxxx/provision-node/apps-phase/stderr.gz:

Error: /usr/lib/hive/bin/schematool -dbType mysql -initSchema -verbose returned 1 instead of one of [0]
Error: /Stage[main]/Hadoop_hive::Init_metastore_schema/Exec[init hive-metastore schema]/returns: change from notrun to 0 failed: /usr/lib/hive/bin/schematool -dbType mysql -initSchema -verbose returned 1 instead of one of [0]
Error: /Stage[main]/Hadoop_hive::Init_metastore_schema/Exec[init hive-metastore schema]: Failed to call refresh: /usr/lib/hive/bin/schematool -dbType mysql -initSchema -verbose returned 1 instead of one of [0]
Error: /Stage[main]/Hadoop_hive::Init_metastore_schema/Exec[init hive-metastore schema]: /usr/lib/hive/bin/schematool -dbType mysql -initSchema -verbose returned 1 instead of one of [0]
Warning: /Stage[main]/Hadoop_hive::Metastore_server/Service[hive-metastore]: Skipping because of failed dependencies
Warning: /Stage[main]/Hadoop_hive::Server/Package[hive-server2]: Skipping because of failed dependencies
Warning: /Stage[main]/Hadoop_hive::Server/Service[hive-server2]: Skipping because of failed dependencies
2016-04-16 02:16:54,022 ERROR main: Encountered a problem while provisioning
com.amazonaws.emr.node.provisioner.puppet.api.PuppetException: Unable to complete transaction and some changes were applied.
    at com.amazonaws.emr.node.provisioner.puppet.api.ApplyCommand.handleExitcode(ApplyCommand.java:74)
    at com.amazonaws.emr.node.provisioner.puppet.api.ApplyCommand.call(ApplyCommand.java:56)
    at com.amazonaws.emr.node.provisioner.bigtop.BigtopPuppeteer.applyPuppet(BigtopPuppeteer.java:49)
    at com.amazonaws.emr.node.provisioner.bigtop.BigtopDeployer.deploy(BigtopDeployer.java:21)
    at com.amazonaws.emr.node.provisioner.NodeProvisioner.provision(NodeProvisioner.java:28)
    at com.amazonaws.emr.node.provisioner.phase.PhaseWorkflow.work(PhaseWorkflow.java:51)
    at com.amazonaws.emr.node.provisioner.phase.ProvisionAppsPhase.perform(ProvisionAppsPhase.java:30)
    at com.amazonaws.emr.node.provisioner.Program.main(Program.java:17)

原因

javax.jdo.option.ConnectionDriverName は不要とのこと。

AWS Developer Forums: Unable to start EMR cluster with ...

ドキュメントの例には javax.jdo.option.ConnectionDriverName を設定するようになっているので注意。

Create a Hive Metastore Outside the Cluster - Amazon Elastic MapReduce

対策

Configurationsから javax.jdo.option.ConnectionDriverName を削除してEMRを再構築。

[
  {
    "classification": "hive-site",
    "properties": {
        "hive.exec.dynamic.partition.mode": "nonstrict",
        "javax.jdo.option.ConnectionURL": "jdbc:mysql://xxx.local:3306/hive",
        "javax.jdo.option.ConnectionUserName": "xxx",
        "javax.jdo.option.ConnectionPassword": "xxx"
    }
  }
]