LoginSignup
3
3

More than 5 years have passed since last update.

@wapa5pow

embulk(0.8.12)でMySQLからBigQueryにデータ流そうとしたらハマったのでログを残す

Last updated at Posted at 2016-08-12

2016/8/15 20:00 更新

  • httpclient 2.8.2がでて修正されました。

2016/8/15 17:00 更新

  • httpclientのバージョンを下げても以下は対処できます。
embulk gem install httpclient -v 2.8.0
embulk gem uninstall httpclient -v 2.8.1

前提

MacOS上では大丈夫

EC2のAmazon Linux上で以下の組み合わせでエラー(EC2上でUbuntu)も試したけど同じ。

embulk: 0.8.12
embulk-input-mysql: 0.7.2
embulk-output-bigquery: 0.3.6

エラー内容

D, [2016-08-12T14:02:56.308000 #13984] DEBUG -- : Sending HTTP get https://www.googleapis.com/bigquery/v2/projects/xxx/datasets/xxx_mobile?
D, [2016-08-12T14:02:58.317000 #13984] DEBUG -- : Caught error sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
D, [2016-08-12T14:02:58.327000 #13984] DEBUG -- : Error - #<Hurley::SSLError: OpenSSL::SSL::SSLError>

org.embulk.exec.PartialExecutionException: org.jruby.exceptions.RaiseException: (TransmissionError) sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at org.embulk.exec.BulkLoader$LoaderState.buildPartialExecuteException(org/embulk/exec/BulkLoader.java:363)
    at org.embulk.exec.BulkLoader.doRun(org/embulk/exec/BulkLoader.java:572)
    at org.embulk.exec.BulkLoader.access$000(org/embulk/exec/BulkLoader.java:33)
    at org.embulk.exec.BulkLoader$1.run(org/embulk/exec/BulkLoader.java:374)
    at org.embulk.exec.BulkLoader$1.run(org/embulk/exec/BulkLoader.java:370)
    at org.embulk.spi.Exec.doWith(org/embulk/spi/Exec.java:25)
    at org.embulk.exec.BulkLoader.run(org/embulk/exec/BulkLoader.java:370)
    at org.embulk.EmbulkEmbed.run(org/embulk/EmbulkEmbed.java:180)
    at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)
    at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:453)
    at org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:314)
    at RUBY.run(/home/xxx/.embulk/bin/embulk!/embulk/runner.rb:84)
    at RUBY.run(/home/xxx/.embulk/bin/embulk!/embulk/command/embulk_run.rb:302)
    at RUBY.<top>(/home/xxx/.embulk/bin/embulk!/embulk/command/embulk_main.rb:2)
    at org.jruby.Ruby.runInterpreter(org/jruby/Ruby.java:849)
    at org.jruby.Ruby.loadFile(org/jruby/Ruby.java:2986)
    at org.jruby.RubyKernel.requireCommon(org/jruby/RubyKernel.java:951)
    at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:944)
    at RUBY.(root)(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1)
    at home.xxx.$_dot_embulk.bin.embulk.embulk.command.embulk_bundle.<top>(file:/home/xxx/.embulk/bin/embulk!/embulk/command/embulk_bundle.rb:51)
    at java.lang.invoke.MethodHandle.invokeWithArguments(java/lang/invoke/MethodHandle.java:627)
    at org.jruby.Ruby.runScript(org/jruby/Ruby.java:833)
    at org.jruby.Ruby.runScript(org/jruby/Ruby.java:825)
    at org.jruby.Ruby.runNormally(org/jruby/Ruby.java:760)
    at org.jruby.Ruby.runFromMain(org/jruby/Ruby.java:579)
    at org.jruby.Main.doRunFromMain(org/jruby/Main.java:425)
    at org.jruby.Main.internalRun(org/jruby/Main.java:313)
    at org.jruby.Main.run(org/jruby/Main.java:242)
    at org.jruby.Main.main(org/jruby/Main.java:204)
    at org.embulk.cli.Main.main(org/embulk/cli/Main.java:23)
    Suppressed: java.lang.NullPointerException
        at org.embulk.exec.BulkLoader.doCleanup(BulkLoader.java:479)
        at org.embulk.exec.BulkLoader$3.run(BulkLoader.java:410)
        at org.embulk.exec.BulkLoader$3.run(BulkLoader.java:406)
        at org.embulk.spi.Exec.doWith(Exec.java:25)
        at org.embulk.exec.BulkLoader.cleanup(BulkLoader.java:406)
        at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:184)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:453)
        at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:314)
        at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:45)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:313)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:316)
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:77)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:109)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:95)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:273)
        at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:79)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)
        at org.jruby.ir.instructions.CallBase.interpret(CallBase.java:423)
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:348)
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:77)
        at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:86)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:197)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:313)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:316)
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:77)
        at org.jruby.ir.interpreter.Interpreter.INTERPRET_ROOT(Interpreter.java:112)
        at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:99)
        at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:35)
        at org.jruby.ir.IRTranslator.execute(IRTranslator.java:42)
        at org.jruby.Ruby.runInterpreter(Ruby.java:849)
        at org.jruby.Ruby.loadFile(Ruby.java:2986)
        at org.jruby.runtime.load.LibrarySearcher$ResourceLibrary.load(LibrarySearcher.java:235)
        at org.jruby.runtime.load.LibrarySearcher$FoundLibrary.load(LibrarySearcher.java:34)
        at org.jruby.runtime.load.LoadService.tryLoadingLibraryOrScript(LoadService.java:900)
        at org.jruby.runtime.load.LoadService.smartLoadInternal(LoadService.java:540)
        at org.jruby.runtime.load.LoadService.requireCommon(LoadService.java:427)
        at org.jruby.runtime.load.LoadService.require(LoadService.java:393)
        at org.jruby.RubyKernel.requireCommon(RubyKernel.java:951)
        at org.jruby.RubyKernel.require19(RubyKernel.java:944)
        at org.jruby.RubyKernel$INVOKER$s$1$0$require19.call(RubyKernel$INVOKER$s$1$0$require19.gen)
        at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodOneOrNBlock.call(JavaMethod.java:383)
        at org.jruby.internal.runtime.methods.AliasMethod.call(AliasMethod.java:61)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:313)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
        at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:316)
        at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:77)
        at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:86)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:197)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:313)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
        at home.xxx.$_dot_embulk.bin.embulk.embulk.command.embulk_bundle.invokeOther66:require(file:/home/xxx/.embulk/bin/embulk!/embulk/command/embulk_bundle.rb)
        at home.xxx.$_dot_embulk.bin.embulk.embulk.command.embulk_bundle.RUBY$script(file:/home/xxx/.embulk/bin/embulk!/embulk/command/embulk_bundle.rb:51)
        at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
        at org.jruby.ir.Compiler$1.load(Compiler.java:111)
        at org.jruby.Ruby.runScript(Ruby.java:833)
        at org.jruby.Ruby.runScript(Ruby.java:825)
        at org.jruby.Ruby.runNormally(Ruby.java:760)
        at org.jruby.Ruby.runFromMain(Ruby.java:579)
        at org.jruby.Main.doRunFromMain(Main.java:425)
        at org.jruby.Main.internalRun(Main.java:313)
        at org.jruby.Main.run(Main.java:242)
        at org.jruby.Main.main(Main.java:204)
        at org.embulk.cli.Main.main(Main.java:23)
Caused by: org.jruby.exceptions.RaiseException: (TransmissionError) sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at RUBY.error(/home/xxx/.embulk/jruby/2.3.0/gems/google-api-client-0.9.11/lib/google/apis/core/http_command.rb:256)
    at RUBY.execute_once(/home/xxx/.embulk/jruby/2.3.0/gems/google-api-client-0.9.11/lib/google/apis/core/http_command.rb:290)
    at RUBY.block in execute(/home/xxx/.embulk/jruby/2.3.0/gems/google-api-client-0.9.11/lib/google/apis/core/http_command.rb:107)
    at RUBY.block in retriable(/home/xxx/.embulk/jruby/2.3.0/gems/retriable-2.1.0/lib/retriable.rb:54)
    at org.jruby.RubyFixnum.times(org/jruby/RubyFixnum.java:297)
    at RUBY.retriable(/home/xxx/.embulk/jruby/2.3.0/gems/retriable-2.1.0/lib/retriable.rb:48)
    at RUBY.block in execute(/home/xxx/.embulk/jruby/2.3.0/gems/google-api-client-0.9.11/lib/google/apis/core/http_command.rb:104)
    at RUBY.block in retriable(/home/xxx/.embulk/jruby/2.3.0/gems/retriable-2.1.0/lib/retriable.rb:54)
    at org.jruby.RubyFixnum.times(org/jruby/RubyFixnum.java:297)
    at RUBY.retriable(/home/xxx/.embulk/jruby/2.3.0/gems/retriable-2.1.0/lib/retriable.rb:48)
    at RUBY.execute(/home/xxx/.embulk/jruby/2.3.0/gems/google-api-client-0.9.11/lib/google/apis/core/http_command.rb:96)
    at RUBY.execute_or_queue_command(/home/xxx/.embulk/jruby/2.3.0/gems/google-api-client-0.9.11/lib/google/apis/core/base_service.rb:346)
    at RUBY.get_dataset(/home/xxx/.embulk/jruby/2.3.0/gems/google-api-client-0.9.11/generated/google/apis/bigquery_v2/service.rb:133)
    at RUBY.get_dataset(/home/xxx/.embulk/jruby/2.3.0/gems/embulk-output-bigquery-0.3.6/lib/embulk/output/bigquery/bigquery_client.rb:352)
    at RUBY.transaction(/home/xxx/.embulk/jruby/2.3.0/gems/embulk-output-bigquery-0.3.6/lib/embulk/output/bigquery.rb:272)
    at RUBY.transaction(/home/xxx/.embulk/bin/embulk!/embulk/output_plugin.rb:64)
    at RUBY.run(/home/xxx/.embulk/bin/embulk!/embulk/runner.rb:84)
    at RUBY.run(/home/xxx/.embulk/bin/embulk!/embulk/command/embulk_run.rb:302)
    at RUBY.<top>(/home/xxx/.embulk/bin/embulk!/embulk/command/embulk_main.rb:2)
    at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:944)
    at RUBY.(root)(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1)
    at home.xxx.$_dot_embulk.bin.embulk.embulk.command.embulk_bundle.<top>(file:/home/xxx/.embulk/bin/embulk!/embulk/command/embulk_bundle.rb:51)

Twitterで助けてもらった

@hiroysato さんありがとうございますmm

結局どうしたのか

cd ~/server/config/google_service_accounts
wget https://www.geotrust.com/resources/root_certificates/certificates/Equifax_Secure_Certificate_Authority.pem
export SSL_CERT_FILE=~/server/config/google_service_accounts/Equifax_Secure_Certificate_Authority.pem

上記をやったうえで、embulk runを実行

このSSL_CERT_FILEはグローバルで設定するといろいろやっかいだと思うので、気をつける。

3
3
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
3
3