0
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 5 years have passed since last update.

spark 2.0.0をzepellin 0.7.0で使うには

Last updated at Posted at 2016-07-28
  • 更新して、buildしなおせばいけるっぽい。

git stash
git pull origin master
mvn clean package -Dscala-2.11 -Pspark-2.0 -Dspark.version=2.0.0 -Ppyspark -Psparkr -Pyarn -Phadoop-2.6 -DskipTests

↓のエラーがでてるっぽいが、

  • spark sql以外sparkの動作には問題がない
  • spark 1.6.2で動いてた独自 module jarはそのままでは動かないかも
  • scala 2.11でbuildしないといけない?
  • でも、sbtでdefaultでbuildしてtarget/scala-2.11以下に吐いているのでうまくいくはずだが。。。
  • 独自module jarの問題を解決して、そのあと、spark sql経由でhiveが動くかどうかを確認
ERROR [2016-07-28 23:47:31,743] ({pool-2-thread-2} Utils.java[invokeMethod]:40) - org.apache.spark.sql.SparkSession.wrapped()
java.lang.NoSuchMethodException: org.apache.spark.sql.SparkSession.wrapped()
        at java.lang.Class.getMethod(Class.java:1670)
        at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:38)
        at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:33)
        at org.apache.zeppelin.spark.SparkInterpreter.getSQLContext_2(SparkInterpreter.java:239)
        at org.apache.zeppelin.spark.SparkInterpreter.getSQLContext(SparkInterpreter.java:227)
        at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:727)
        at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
        at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:93)
        at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:383)
        at org.apache.zeppelin.scheduler.Job.run(Job.java:176)
        at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

1.6.2では動いたのに、2.0.0では独自jarが読めない
かつ、#executorsの設定が反映されない。

zeppelinではなくて sparkのspark-submitのoptionが効いてない?

0
0
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
0
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?