環境
- EMR: emr-4.3.0
- RDS: MySQL 5.6.27
現象
Configurationsに以下を指定してEMRを起動すると、Terminated with errorsOn the master instance (i-xxxxxxxx), application provisioning failed
が発生してTerminateされる。
Configurations:
[
{
"classification": "hive-site",
"properties": {
"hive.exec.dynamic.partition.mode": "nonstrict",
"javax.jdo.option.ConnectionURL": "jdbc:mysql://xxx.local:3306/hive",
"javax.jdo.option.ConnectionDriverName": "com.mysql.jdbc.Driver",
"javax.jdo.option.ConnectionUserName": "xxx",
"javax.jdo.option.ConnectionPassword": "xxx"
}
}
]
マスターのプロビジョニング標準エラー出力に以下のエラーが出力される。
s3://aws-logs-xxxxxxxxxxxx-ap-northeast-1/elasticmapreduce/j-xxxxxxxxxxxxx/node/i-xxxxxxxx/provision-node/apps-phase/stderr.gz:
Error: /usr/lib/hive/bin/schematool -dbType mysql -initSchema -verbose returned 1 instead of one of [0]
Error: /Stage[main]/Hadoop_hive::Init_metastore_schema/Exec[init hive-metastore schema]/returns: change from notrun to 0 failed: /usr/lib/hive/bin/schematool -dbType mysql -initSchema -verbose returned 1 instead of one of [0]
Error: /Stage[main]/Hadoop_hive::Init_metastore_schema/Exec[init hive-metastore schema]: Failed to call refresh: /usr/lib/hive/bin/schematool -dbType mysql -initSchema -verbose returned 1 instead of one of [0]
Error: /Stage[main]/Hadoop_hive::Init_metastore_schema/Exec[init hive-metastore schema]: /usr/lib/hive/bin/schematool -dbType mysql -initSchema -verbose returned 1 instead of one of [0]
Warning: /Stage[main]/Hadoop_hive::Metastore_server/Service[hive-metastore]: Skipping because of failed dependencies
Warning: /Stage[main]/Hadoop_hive::Server/Package[hive-server2]: Skipping because of failed dependencies
Warning: /Stage[main]/Hadoop_hive::Server/Service[hive-server2]: Skipping because of failed dependencies
2016-04-16 02:16:54,022 ERROR main: Encountered a problem while provisioning
com.amazonaws.emr.node.provisioner.puppet.api.PuppetException: Unable to complete transaction and some changes were applied.
at com.amazonaws.emr.node.provisioner.puppet.api.ApplyCommand.handleExitcode(ApplyCommand.java:74)
at com.amazonaws.emr.node.provisioner.puppet.api.ApplyCommand.call(ApplyCommand.java:56)
at com.amazonaws.emr.node.provisioner.bigtop.BigtopPuppeteer.applyPuppet(BigtopPuppeteer.java:49)
at com.amazonaws.emr.node.provisioner.bigtop.BigtopDeployer.deploy(BigtopDeployer.java:21)
at com.amazonaws.emr.node.provisioner.NodeProvisioner.provision(NodeProvisioner.java:28)
at com.amazonaws.emr.node.provisioner.phase.PhaseWorkflow.work(PhaseWorkflow.java:51)
at com.amazonaws.emr.node.provisioner.phase.ProvisionAppsPhase.perform(ProvisionAppsPhase.java:30)
at com.amazonaws.emr.node.provisioner.Program.main(Program.java:17)
原因
javax.jdo.option.ConnectionDriverName
は不要とのこと。
ドキュメントの例には javax.jdo.option.ConnectionDriverName
を設定するようになっているので注意。
対策
Configurationsから javax.jdo.option.ConnectionDriverName
を削除してEMRを再構築。
[
{
"classification": "hive-site",
"properties": {
"hive.exec.dynamic.partition.mode": "nonstrict",
"javax.jdo.option.ConnectionURL": "jdbc:mysql://xxx.local:3306/hive",
"javax.jdo.option.ConnectionUserName": "xxx",
"javax.jdo.option.ConnectionPassword": "xxx"
}
}
]