Posted at

EMRでRDSのMySQLを外部メタストアに指定するとTerminateされる #aws #emr

More than 3 years have passed since last update.


環境


  • EMR: emr-4.3.0

  • RDS: MySQL 5.6.27


現象

Configurationsに以下を指定してEMRを起動すると、Terminated with errorsOn the master instance (i-xxxxxxxx), application provisioning failed が発生してTerminateされる。

Configurations:

[

{
"classification": "hive-site",
"properties": {
"hive.exec.dynamic.partition.mode": "nonstrict",
"javax.jdo.option.ConnectionURL": "jdbc:mysql://xxx.local:3306/hive",
"javax.jdo.option.ConnectionDriverName": "com.mysql.jdbc.Driver",
"javax.jdo.option.ConnectionUserName": "xxx",
"javax.jdo.option.ConnectionPassword": "xxx"
}
}
]

マスターのプロビジョニング標準エラー出力に以下のエラーが出力される。

s3://aws-logs-xxxxxxxxxxxx-ap-northeast-1/elasticmapreduce/j-xxxxxxxxxxxxx/node/i-xxxxxxxx/provision-node/apps-phase/stderr.gz:

Error: /usr/lib/hive/bin/schematool -dbType mysql -initSchema -verbose returned 1 instead of one of [0]

Error: /Stage[main]/Hadoop_hive::Init_metastore_schema/Exec[init hive-metastore schema]/returns: change from notrun to 0 failed: /usr/lib/hive/bin/schematool -dbType mysql -initSchema -verbose returned 1 instead of one of [0]
Error: /Stage[main]/Hadoop_hive::Init_metastore_schema/Exec[init hive-metastore schema]: Failed to call refresh: /usr/lib/hive/bin/schematool -dbType mysql -initSchema -verbose returned 1 instead of one of [0]
Error: /Stage[main]/Hadoop_hive::Init_metastore_schema/Exec[init hive-metastore schema]: /usr/lib/hive/bin/schematool -dbType mysql -initSchema -verbose returned 1 instead of one of [0]
Warning: /Stage[main]/Hadoop_hive::Metastore_server/Service[hive-metastore]: Skipping because of failed dependencies
Warning: /Stage[main]/Hadoop_hive::Server/Package[hive-server2]: Skipping because of failed dependencies
Warning: /Stage[main]/Hadoop_hive::Server/Service[hive-server2]: Skipping because of failed dependencies
2016-04-16 02:16:54,022 ERROR main: Encountered a problem while provisioning
com.amazonaws.emr.node.provisioner.puppet.api.PuppetException: Unable to complete transaction and some changes were applied.
at com.amazonaws.emr.node.provisioner.puppet.api.ApplyCommand.handleExitcode(ApplyCommand.java:74)
at com.amazonaws.emr.node.provisioner.puppet.api.ApplyCommand.call(ApplyCommand.java:56)
at com.amazonaws.emr.node.provisioner.bigtop.BigtopPuppeteer.applyPuppet(BigtopPuppeteer.java:49)
at com.amazonaws.emr.node.provisioner.bigtop.BigtopDeployer.deploy(BigtopDeployer.java:21)
at com.amazonaws.emr.node.provisioner.NodeProvisioner.provision(NodeProvisioner.java:28)
at com.amazonaws.emr.node.provisioner.phase.PhaseWorkflow.work(PhaseWorkflow.java:51)
at com.amazonaws.emr.node.provisioner.phase.ProvisionAppsPhase.perform(ProvisionAppsPhase.java:30)
at com.amazonaws.emr.node.provisioner.Program.main(Program.java:17)


原因

javax.jdo.option.ConnectionDriverName は不要とのこと。

ドキュメントの例には javax.jdo.option.ConnectionDriverName を設定するようになっているので注意。


対策

Configurationsから javax.jdo.option.ConnectionDriverName を削除してEMRを再構築。

[

{
"classification": "hive-site",
"properties": {
"hive.exec.dynamic.partition.mode": "nonstrict",
"javax.jdo.option.ConnectionURL": "jdbc:mysql://xxx.local:3306/hive",
"javax.jdo.option.ConnectionUserName": "xxx",
"javax.jdo.option.ConnectionPassword": "xxx"
}
}
]