LoginSignup
0
0

More than 3 years have passed since last update.

@tworks55(T M)

Kafka REST Proxy on Mac

Last updated at Posted at 2019-12-28

はじめに

Kafka REST ProxyをMacで使用するにあたり、少しはまったのでメモを残しておきます。

動作確認環境

OS:macOS Mojave Version 10.14.6 (18G84)

1. Kafka REST Proxyのビルドでエラー

> git clone https://github.com/confluentinc/kafka-rest.git
> cd kafka-rest
> mvn install

で以下のようなエラーが発生。

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for kafka-rest-parent 5.5.0-SNAPSHOT:
[INFO] 
[INFO] kafka-rest-parent .................................. SUCCESS [  2.062 s]
[INFO] kafka-rest-common .................................. FAILURE [  0.063 s]
[INFO] kafka-rest-scala-consumer .......................... SKIPPED
[INFO] kafka-rest ......................................... SKIPPED
[INFO] kafka-rest-console-scripts ......................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  2.575 s
[INFO] Finished at: 2019-12-28T17:44:08+09:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project kafka-rest-common: Could not resolve dependencies for project io.confluent:kafka-rest-common:jar:5.5.0-SNAPSHOT: Failure to find io.confluent:rest-utils:jar:5.5.0-SNAPSHOT in http://packages.confluent.io/maven/ was cached in the local repository, resolution will not be reattempted until the update interval of confluent has elapsed or updates are forced -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :kafka-rest-common```

どうやら、この記事によると、SNAPSHOTSはローカルには無いのでFAQを見るかConfluent Platformを使えば良いとのこと。

2. Kafka関連の色々をBuild

ということで、以下のProjectをビルドしローカルのMavenリポジトリに必要なライブラリをインストールします。

Project Clone Build with
Kafka https://github.com/confluentinc/kafka/ ./gradlew installAll
Confluent Common library https://github.com/confluentinc/common mvn clean install
Confluent Rest-Utils https://github.com/confluentinc/rest-utils mvn clean install
Avro Converter https://github.com/confluentinc/schema-registry mvn clean install

まずはkafka本体。
初回は１時間過ぎてもずっと終わらず、一旦停止。よくわからないが再度実行すると今度は成功。

> git clone https://github.com/confluentinc/kafka.git
> cd kafka
> ./gradlew installAll

そして、次にConfluent Common libraryをビルド。エラーが発生。。。

> git clone https://github.com/confluentinc/common.git
> cd common
> mvn clean install
(中略)
>
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for common 5.5.0-SNAPSHOT:
[INFO] 
[INFO] assembly-plugin-boilerplate ........................ SUCCESS [  1.468 s]
[INFO] Build Tools ........................................ SUCCESS [  1.596 s]
[INFO] common ............................................. SUCCESS [  2.422 s]
[INFO] utils .............................................. SUCCESS [  2.426 s]
[INFO] metrics ............................................ SUCCESS [  3.572 s]
[INFO] config ............................................. SUCCESS [  2.342 s]
[INFO] common-logging ..................................... SUCCESS [  3.647 s]
[INFO] confluent-log4j-extensions ......................... SUCCESS [  2.562 s]
[INFO] confluent-log4j2-extensions ........................ SUCCESS [  2.888 s]
[INFO] package ............................................ FAILURE [  0.905 s]
[INFO] Common ............................................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  24.527 s
[INFO] Finished at: 2019-12-28T17:19:13+09:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:3.0.0:single (package-assembly) on project common-package: Execution package-assembly of goal org.apache.maven.plugins:maven-assembly-plugin:3.0.0:single failed: group id '1614999791' is too big ( > 2097151 ). Use STAR or POSIX extensions to overcome this limit -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :common-package

Mac特有のエラーのようなのでこちらを参考にtarLongFileModeのオプションを追記し、再度ビルドすると成功。

pom.xml
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-assembly-plugin</artifactId>
                    <version>${maven-assembly.version}</version>

                    <!-- added this configuration tag -->
                    <configuration>
                     <descriptors>
                      <descriptor>src/assembly/distribution.xml</descriptor>
                      </descriptors>
                      <tarLongFileMode>posix</tarLongFileMode>
                      <outputDirectory>${project.build.directory}</outputDirectory>
                      <appendAssemblyId>false</appendAssemblyId>
                     </configuration>
                </plugin>

次はConfluent Rest-Utils。こちらは普通にビルド成功。

> git clone https://github.com/confluentinc/rest-utils.git
> cd rest-utils
> mvn clean install

そしてAvro Converter。時間は掛かったがこちらも普通にビルド成功。

> git clone https://github.com/confluentinc/schema-registry.git
> cd schema-registry
> mvn clean install

3. Kafka REST Proxyを改めてBuild

やっと成功。

>mvn install
(中略)

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for kafka-rest-parent 5.5.0-SNAPSHOT:
[INFO] 
[INFO] kafka-rest-parent .................................. SUCCESS [  6.480 s]
[INFO] kafka-rest-common .................................. SUCCESS [  5.371 s]
[INFO] kafka-rest-scala-consumer .......................... SUCCESS [ 18.041 s]
[INFO] kafka-rest ......................................... SUCCESS [07:35 min]
[INFO] kafka-rest-console-scripts ......................... SUCCESS [  3.344 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  08:09 min
[INFO] Finished at: 2019-12-28T18:41:40+09:00
[INFO] ------------------------------------------------------------------------

4. CLASSPATHを追加で設定

Kafka REST Proxyを起動するも、

Error: Could not find or load main class io.confluent.kafkarest.KafkaRestMain

ダメ押しでbin/kafka-rest-run-classを修正

for dir in $base_dir/kafka-rest/target/kafka-rest-*-development; do
  CLASSPATH=$CLASSPATH:$dir/share/java/kafka-rest/*
done

以下に変更。

for dir in $base_dir/kafka-rest/target/kafka-rest-*-SNAPSHOT; do
  CLASSPATH=$CLASSPATH:$dir/share/java/kafka-rest/*
done

動くようになったと思ったらlog4jのエラー。

log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /kafka-rest.log (Permission denied)
    at java.io.FileOutputStream.open0(Native Method)
    at java.io.FileOutputStream.open(FileOutputStream.java:270)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
    at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
    at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:207)
    at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
    at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
    at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
    at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
    at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
    at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
    at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
    at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
    at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
    at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
    at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
    at org.slf4j.impl.Log4jLoggerFactory.<init>(Log4jLoggerFactory.java:66)
    at org.slf4j.impl.StaticLoggerBinder.<init>(StaticLoggerBinder.java:72)
    at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:45)
    at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
    at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
    at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412)
    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)
    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383)
    at io.confluent.kafkarest.KafkaRestMain.<clinit>(KafkaRestMain.java:28)

とりあえず面倒なので、

  if [ -e "$base_dir/config/log4j.properties" ]; then # Dev environment
    KAFKAREST_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/config/log4j.properties"

を以下に変更。

  if [ -e "$base_dir/config/log4j.properties" ]; then # Dev environment
    KAFKAREST_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/config/log4j.properties -Dkafka-rest.log.dir=$base_dir"

ようやく使えるようになりました。

参考URL

https://github.com/confluentinc/kafka-rest/wiki/FAQ
https://stackoverflow.com/questions/56388665/could-not-find-artifact-io-confluentkafka-rest-parentpom5-4-0-snapshot-and-p
https://tutorialmore.com/questions-826377.htm

0
0
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
0
0