はじめに
HortonworksのHDFを試そうと思いましたが、PCではリソース不足で試せなかったのでそれぞれのパーツで試してみます。まずはNiFiとKafkaをWindows環境で使ってみようと思いま
す。
追記：後日Apache MiNiFi on Windowsも実施しています。
1. 前提
以下の環境で動作した内容を記述しています。
Windows 10 Home (64bit) *Memory 8G
JDK 1.8
2. NiFiのインストールと起動
http://nifi.apache.org/download.htmlよりnifi-1.4.0-bin.zipなど比較的新しいzipファイルをダウンロードし、動作させたいフォルダに展開します。
DOS窓を立ち上げ、bin\run-nifi.batを起動します。
C:\dev\NIFI\nifi-1.4.0\bin>dir
ドライブ C のボリューム ラベルは Windows です
ボリューム シリアル番号は A295-434A です
C:\dev\NIFI\nifi-1.4.0\bin のディレクトリ
2018/01/08 15:00 <DIR> .
2018/01/08 15:00 <DIR> ..
2017/09/28 13:48 1,872 dump-nifi.bat
2017/09/28 13:48 1,120 nifi-env.bat
2017/09/28 13:48 1,110 nifi-env.sh
2017/09/28 13:48 11,888 nifi.sh
2017/09/28 13:48 1,871 run-nifi.bat
2017/09/28 13:48 1,832 status-nifi.bat
6 個のファイル 19,693 バイト
2 個のディレクトリ 34,516,406,272 バイトの空き領域
C:\dev\NIFI\nifi-1.4.0\bin>run-nifi.bat
2018-01-08 15:02:22,666 INFO [main] org.apache.nifi.bootstrap.Command Starting Apache NiFi...
2018-01-08 15:02:22,668 INFO [main] org.apache.nifi.bootstrap.Command Working Directory: C:\dev\NIFI\NIFI-1~1.0
2018-01-08 15:02:22,670 INFO [main] org.apache.nifi.bootstrap.Command Command: C:\dev\Java\jdk1.8.0_102\bin\java.exe -classpath C:\dev\NIFI\NIFI-1~1.0\.\conf;C:\dev\NIFI\NIFI-1~1.0\.\lib\javax.servlet-api-3.1.0.jar;C:\dev\NIFI\NIFI-1~1.0\.\lib\jcl-over-slf4j-1.7.25.jar;C:\dev\NIFI\NIFI-1~1.0\.\lib\jetty-schemas-3.1.jar;C:\dev\NIFI\NIFI-1~1.0\.\lib\jul-to-slf4j-1.7.25.jar;C:\dev\NIFI\NIFI-1~1.0\.\lib\log4j-over-slf4j-1.7.25.jar;C:\dev\NIFI\NIFI-1~1.0\.\lib\logback-classic-1.2.3.jar;C:\dev\NIFI\NIFI-1~1.0\.\lib\logback-core-1.2.3.jar;C:\dev\NIFI\NIFI-1~1.0\.\lib\nifi-api-1.4.0.jar;C:\dev\NIFI\NIFI-1~1.0\.\lib\nifi-framework-api-1.4.0.jar;C:\dev\NIFI\NIFI-1~1.0\.\lib\nifi-nar-utils-1.4.0.jar;C:\dev\NIFI\NIFI-1~1.0\.\lib\nifi-properties-1.4.0.jar;C:\dev\NIFI\NIFI-1~1.0\.\lib\nifi-runtime-1.4.0.jar;C:\dev\NIFI\NIFI-1~1.0\.\lib\slf4j-api-1.7.25.jar -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx512m -Xms512m -Djava.security.egd=file:/dev/urandom -Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -XX:+UseG1GC -Djava.protocol.handler.pkgs=sun.net.www.protocol -Dnifi.properties.file.path=C:\dev\NIFI\NIFI-1~1.0\.\conf\nifi.properties -Dnifi.bootstrap.listen.port=55171 -Dapp=NiFi -Dorg.apache.nifi.bootstrap.config.log.dir=C:\dev\NIFI\NIFI-1~1.0\bin\..\\logs org.apache.nifi.NiFi
2018-01-08 15:02:23,069 WARN [main] org.apache.nifi.bootstrap.Command Failed to set permissions so that only the owner can read pid file C:\dev\NIFI\NIFI-1~1.0\bin\..\run\nifi.pid; this may allows others to have access to the key needed to communicate with NiFi. Permissions should be changed so that only the owner can read this file
2018-01-08 15:02:23,074 WARN [main] org.apache.nifi.bootstrap.Command Failed to set permissions so that only the owner can read status file C:\dev\NIFI\NIFI-1~1.0\bin\..\run\nifi.status; this may allows others to have access to the key needed to communicate with NiFi. Permissions should be changed so that only the owner can read this file
2018-01-08 15:02:23,085 INFO [main] org.apache.nifi.bootstrap.Command Launched Apache NiFi with Process ID 8140
※停止する場合は、CTRL+Cを使用します。
WEBブラウザでhttp://127.0.0.1:8080/nifi/ or http://localhost:8080/nifi or http://[PCのIPアドレス]:8080/nifi にアクセスし、以下のように表示されれば成功です。
3. Kafkaのインストールと起動
https://kafka.apache.org/downloadsよりkafka_2.11-1.0.0.tgzなど比較的新しいtgzファイルをダウンロードし、動作させたいフォルダに展開します。(Scalaのバージョンにも依存するようですが、2.11がおすすめとのことです。)
3.1 ZooKeeperの起動
DOS窓を立ち上げ、zookeeper-server-start.batを起動します。
c:\dev\kafka\kafka_2.11-1.0.0\bin\windows のディレクトリ
2018/01/08 14:28 <DIR> .
2018/01/08 14:28 <DIR> ..
2017/10/28 00:56 1,241 connect-distributed.bat
2017/10/28 00:56 1,239 connect-standalone.bat
2017/10/28 00:56 873 kafka-acls.bat
2017/10/28 00:56 885 kafka-broker-api-versions.bat
2017/10/28 00:56 876 kafka-configs.bat
2017/10/28 00:56 925 kafka-console-consumer.bat
2017/10/28 00:56 925 kafka-console-producer.bat
2017/10/28 00:56 883 kafka-consumer-groups.bat
2017/10/28 00:56 884 kafka-consumer-offset-checker.bat
2017/10/28 00:56 938 kafka-consumer-perf-test.bat
2017/10/28 00:56 874 kafka-mirror-maker.bat
2017/10/28 00:56 900 kafka-preferred-replica-election.bat
2017/10/28 00:56 940 kafka-producer-perf-test.bat
2017/10/28 00:56 888 kafka-reassign-partitions.bat
2017/10/28 00:56 880 kafka-replay-log-producer.bat
2017/10/28 00:56 886 kafka-replica-verification.bat
2017/10/28 00:56 5,276 kafka-run-class.bat
2017/10/28 00:56 1,377 kafka-server-start.bat
2017/10/28 00:56 997 kafka-server-stop.bat
2017/10/28 00:56 882 kafka-simple-consumer-shell.bat
2017/10/28 00:56 875 kafka-topics.bat
2017/10/28 00:56 1,192 zookeeper-server-start.bat
2017/10/28 00:56 905 zookeeper-server-stop.bat
2017/10/28 00:56 977 zookeeper-shell.bat
24 個のファイル 27,518 バイト
2 個のディレクトリ 35,582,828,544 バイトの空き領域
c:\dev\kafka\kafka_2.11-1.0.0\bin\windows>zookeeper-server-start.bat ..\..\config\zookeeper.properties
[2018-01-08 14:36:18,762] INFO Reading configuration from: ..\..\config\zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2018-01-08 14:36:18,769] INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.DatadirCleanupManager)
[2018-01-08 14:36:18,769] INFO autopurge.purgeInterval set to 0 (org.apache.zookeeper.server.DatadirCleanupManager)
[2018-01-08 14:36:18,771] INFO Purge task is not scheduled. (org.apache.zookeeper.server.DatadirCleanupManager)
[2018-01-08 14:36:18,772] WARN Either no config or no quorum defined in config, running in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)
[2018-01-08 14:36:18,789] INFO Reading configuration from: ..\..\config\zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2018-01-08 14:36:18,790] INFO Starting server (org.apache.zookeeper.server.ZooKeeperServerMain)
[2018-01-08 14:36:18,802] INFO Server environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-08 14:36:18,802] INFO Server environment:host.name=DESKTOP-F0AFN64 (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-08 14:36:18,803] INFO Server environment:java.version=1.8.0_102 (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-08 14:36:18,806] INFO Server environment:java.vendor=Oracle Corporation (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-08 14:36:18,806] INFO Server environment:java.home=C:\dev\Java\jdk1.8.0_102\jre (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-08 14:36:18,809] INFO Server environment:java.class.path=c:\dev\kafka\kafka_2.11-1.0.0\libs\aopalliance-repackaged-2.5.0-b32.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\argparse4j-0.7.0.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\commons-lang3-3.5.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\connect-api-1.0.0.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\connect-file-1.0.0.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\connect-json-1.0.0.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\connect-runtime-1.0.0.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\connect-transforms-1.0.0.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\guava-20.0.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\hk2-api-2.5.0-b32.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\hk2-locator-2.5.0-b32.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\hk2-utils-2.5.0-b32.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\jackson-annotations-2.9.1.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\jackson-core-2.9.1.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\jackson-databind-2.9.1.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\jackson-jaxrs-base-2.9.1.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\jackson-jaxrs-json-provider-2.9.1.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\jackson-module-jaxb-annotations-2.9.1.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\javassist-3.20.0-GA.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\javassist-3.21.0-GA.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\javax.annotation-api-1.2.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\javax.inject-1.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\javax.inject-2.5.0-b32.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\javax.servlet-api-3.1.0.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\javax.ws.rs-api-2.0.1.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\jersey-client-2.25.1.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\jersey-common-2.25.1.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\jersey-container-servlet-2.25.1.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\jersey-container-servlet-core-2.25.1.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\jersey-guava-2.25.1.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\jersey-media-jaxb-2.25.1.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\jersey-server-2.25.1.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\jetty-continuation-9.2.22.v20170606.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\jetty-http-9.2.22.v20170606.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\jetty-io-9.2.22.v20170606.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\jetty-security-9.2.22.v20170606.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\jetty-server-9.2.22.v20170606.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\jetty-servlet-9.2.22.v20170606.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\jetty-servlets-9.2.22.v20170606.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\jetty-util-9.2.22.v20170606.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\jopt-simple-5.0.4.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\kafka-clients-1.0.0.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\kafka-log4j-appender-1.0.0.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\kafka-streams-1.0.0.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\kafka-streams-examples-1.0.0.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\kafka-tools-1.0.0.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\kafka_2.11-1.0.0-javadoc.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\kafka_2.11-1.0.0-javadoc.jar.asc;c:\dev\kafka\kafka_2.11-1.0.0\libs\kafka_2.11-1.0.0-scaladoc.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\kafka_2.11-1.0.0-scaladoc.jar.asc;c:\dev\kafka\kafka_2.11-1.0.0\libs\kafka_2.11-1.0.0-sources.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\kafka_2.11-1.0.0-sources.jar.asc;c:\dev\kafka\kafka_2.11-1.0.0\libs\kafka_2.11-1.0.0-test-sources.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\kafka_2.11-1.0.0-test-sources.jar.asc;c:\dev\kafka\kafka_2.11-1.0.0\libs\kafka_2.11-1.0.0-test.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\kafka_2.11-1.0.0-test.jar.asc;c:\dev\kafka\kafka_2.11-1.0.0\libs\kafka_2.11-1.0.0.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\kafka_2.11-1.0.0.jar.asc;c:\dev\kafka\kafka_2.11-1.0.0\libs\log4j-1.2.17.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\lz4-java-1.4.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\maven-artifact-3.5.0.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\metrics-core-2.2.0.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\osgi-resource-locator-1.0.1.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\plexus-utils-3.0.24.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\reflections-0.9.11.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\rocksdbjni-5.7.3.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\scala-library-2.11.11.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\slf4j-api-1.7.25.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\slf4j-log4j12-1.7.25.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\snappy-java-1.1.4.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\validation-api-1.1.0.Final.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\zkclient-0.10.jar;c:\dev\kafka\kafka_2.11-1.0.0\libs\zookeeper-3.4.10.jar (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-08 14:36:18,810] INFO Server environment:java.library.path=C:\dev\Java\jdk1.8.0_102\bin;C:\WINDOWS\Sun\Java\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\dev\utils\gradle-4.3.1\bin;C:\Program Files\MongoDB\Server\3.2\bin;C:\dev\utils\apache-maven-3.3.9\bin;C:\dev\Java\jdk1.8.0_102\bin;C:\Program Files (x86)\Intel\iCLS Client\;C:\Python27\;C:\Python27\Scripts;C:\ProgramData\Oracle\Java\javapath;C:\Program Files\Intel\iCLS Client\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Brackets\command;C:\Program Files (x86)\Microsoft SDKs\Azure\CLI\wbin;C:\Program Files (x86)\Heroku\bin;C:\Program Files (x86)\git\cmd;C:\Program Files (x86)\Nodist\bin;C:\Program Files\Git\cmd;C:\Program Files (x86)\GtkSharp\2.12\bin;C:\Program Files\Intel\WiFi\bin\;C:\Program Files\Common Files\Intel\WirelessCommon\;C:\Program Files\Cloud Foundry;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files\Intel\Intel(R) Management Engine Components\IPT;C:\Users\tworks55\AppData\Local\Microsoft\WindowsApps;;C:\Program Files\Intel\WiFi\bin\;C:\Program Files\Common Files\Intel\WirelessCommon\;C:\Program Files\Microsoft VS Code\bin;. (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-08 14:36:18,811] INFO Server environment:java.io.tmpdir=C:\Users\tworks55\AppData\Local\Temp\ (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-08 14:36:18,811] INFO Server environment:java.compiler=<NA> (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-08 14:36:18,812] INFO Server environment:os.name=Windows 10 (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-08 14:36:18,812] INFO Server environment:os.arch=amd64 (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-08 14:36:18,813] INFO Server environment:os.version=10.0 (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-08 14:36:18,813] INFO Server environment:user.name=tworks55 (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-08 14:36:18,814] INFO Server environment:user.home=C:\Users\tworks55 (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-08 14:36:18,815] INFO Server environment:user.dir=c:\dev\kafka\kafka_2.11-1.0.0\bin\windows (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-08 14:36:18,825] INFO tickTime set to 3000 (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-08 14:36:18,825] INFO minSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-08 14:36:18,826] INFO maxSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-08 14:36:18,892] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
※停止する場合は、CTRL+Cを使用します。
3.2 Kafkaの起動
DOS窓を立ち上げ、kafka-server-start.batを起動します。
C:\dev\kafka\kafka_2.11-1.0.0\bin\windows>kafka-server-start.bat ..\..\config\server.properties
[2018-01-08 15:20:07,707] INFO KafkaConfig values:
advertised.host.name = null
advertised.listeners = null
advertised.port = null
alter.config.policy.class.name = null
authorizer.class.name =
auto.create.topics.enable = true
auto.leader.rebalance.enable = true
background.threads = 10
<中略>
[2018-01-08 15:20:07,788] INFO Client environment:java.io.tmpdir=C:\Users\tworks55\AppData\Local\Temp\ (org.apache.zookeeper.ZooKeeper)
[2018-01-08 15:20:07,788] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2018-01-08 15:20:07,788] INFO Client environment:os.name=Windows 10 (org.apache.zookeeper.ZooKeeper)
[2018-01-08 15:20:07,789] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2018-01-08 15:20:07,789] INFO Client environment:os.version=10.0 (org.apache.zookeeper.ZooKeeper)
[2018-01-08 15:20:07,789] INFO Client environment:user.name=tworks55 (org.apache.zookeeper.ZooKeeper)
[2018-01-08 15:20:07,790] INFO Client environment:user.home=C:\Users\tworks55 (org.apache.zookeeper.ZooKeeper)
[2018-01-08 15:20:07,790] INFO Client environment:user.dir=C:\dev\kafka\kafka_2.11-1.0.0\bin\windows (org.apache.zookeeper.ZooKeeper)
[2018-01-08 15:20:07,791] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@130161f7 (org.apache.zookeeper.ZooKeeper)
[2018-01-08 15:20:07,857] INFO Waiting for keeper state SyncConnected (org.I0Itec.zkclient.ZkClient)
[2018-01-08 15:20:07,859] INFO Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2018-01-08 15:20:07,861] INFO Socket connection established to 127.0.0.1/127.0.0.1:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2018-01-08 15:20:07,871] INFO Session establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid = 0x160d444fbd70004, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2018-01-08 15:20:07,873] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2018-01-08 15:20:08,060] INFO Cluster ID = 4-5EtuXgRkuslEisFtCGTg (kafka.server.KafkaServer)
[2018-01-08 15:20:08,091] INFO [ThrottledRequestReaper-Fetch]: Starting (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2018-01-08 15:20:08,092] INFO [ThrottledRequestReaper-Produce]: Starting (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2018-01-08 15:20:08,093] INFO [ThrottledRequestReaper-Request]: Starting (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2018-01-08 15:20:08,126] INFO Loading logs. (kafka.log.LogManager)
[2018-01-08 15:20:08,145] INFO Logs loading complete in 8 ms. (kafka.log.LogManager)
[2018-01-08 15:20:08,196] WARN Error processing kafka.log:type=LogManager,name=LogDirectoryOffline,logDirectory=C:\tmp\kafka-logs (com.yammer.metrics.reporting.JmxReporter)
javax.management.MalformedObjectNameException: Invalid character ':' in value part of property
at javax.management.ObjectName.construct(ObjectName.java:618)
at javax.management.ObjectName.<init>(ObjectName.java:1382)
at com.yammer.metrics.reporting.JmxReporter.onMetricAdded(JmxReporter.java:395)
at com.yammer.metrics.core.MetricsRegistry.notifyMetricAdded(MetricsRegistry.java:516)
at com.yammer.metrics.core.MetricsRegistry.getOrAdd(MetricsRegistry.java:491)
at com.yammer.metrics.core.MetricsRegistry.newGauge(MetricsRegistry.java:79)
at kafka.metrics.KafkaMetricsGroup$class.newGauge(KafkaMetricsGroup.scala:74)
at kafka.log.LogManager.newGauge(LogManager.scala:50)
at kafka.log.LogManager$$anonfun$6.apply(LogManager.scala:117)
at kafka.log.LogManager$$anonfun$6.apply(LogManager.scala:116)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at kafka.log.LogManager.<init>(LogManager.scala:116)
at kafka.log.LogManager$.apply(LogManager.scala:799)
at kafka.server.KafkaServer.startup(KafkaServer.scala:222)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
at kafka.Kafka$.main(Kafka.scala:92)
at kafka.Kafka.main(Kafka.scala)
[2018-01-08 15:20:08,201] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2018-01-08 15:20:08,203] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2018-01-08 15:20:08,502] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
[2018-01-08 15:20:08,505] INFO [SocketServer brokerId=0] Started 1 acceptor threads (kafka.network.SocketServer)
[2018-01-08 15:20:08,523] INFO [ExpirationReaper-0-Produce]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-01-08 15:20:08,524] INFO [ExpirationReaper-0-Fetch]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-01-08 15:20:08,525] INFO [ExpirationReaper-0-DeleteRecords]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-01-08 15:20:08,532] INFO [LogDirFailureHandler]: Starting (kafka.server.ReplicaManager$LogDirFailureHandler)
[2018-01-08 15:20:08,565] INFO [ExpirationReaper-0-topic]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-01-08 15:20:08,569] INFO [ExpirationReaper-0-Heartbeat]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-01-08 15:20:08,571] INFO [ExpirationReaper-0-Rebalance]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-01-08 15:20:08,576] INFO Creating /controller (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2018-01-08 15:20:08,580] INFO [GroupCoordinator 0]: Starting up. (kafka.coordinator.group.GroupCoordinator)
[2018-01-08 15:20:08,581] INFO [GroupCoordinator 0]: Startup complete. (kafka.coordinator.group.GroupCoordinator)
[2018-01-08 15:20:08,583] INFO [GroupMetadataManager brokerId=0] Removed 0 expired offsets in 2 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2018-01-08 15:20:08,587] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2018-01-08 15:20:08,595] INFO [ProducerId Manager 0]: Acquired new producerId block (brokerId:0,blockStartProducerId:4000,blockEndProducerId:4999) by writing to Zk with path version 5 (kafka.coordinator.transaction.ProducerIdManager)
[2018-01-08 15:20:08,616] INFO [TransactionCoordinator id=0] Starting up. (kafka.coordinator.transaction.TransactionCoordinator)
[2018-01-08 15:20:08,618] INFO [Transaction Marker Channel Manager 0]: Starting (kafka.coordinator.transaction.TransactionMarkerChannelManager)
[2018-01-08 15:20:08,618] INFO [TransactionCoordinator id=0] Startup complete. (kafka.coordinator.transaction.TransactionCoordinator)
[2018-01-08 15:20:08,662] INFO Creating /brokers/ids/0 (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2018-01-08 15:20:08,671] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2018-01-08 15:20:08,672] INFO Registered broker 0 at path /brokers/ids/0 with addresses: EndPoint(DESKTOP-F0AFN64,9092,ListenerName(PLAINTEXT),PLAINTEXT) (kafka.utils.ZkUtils)
[2018-01-08 15:20:08,678] INFO Kafka version : 1.0.0 (org.apache.kafka.common.utils.AppInfoParser)
[2018-01-08 15:20:08,678] INFO Kafka commitId : aaa7af6d4a11b29d (org.apache.kafka.common.utils.AppInfoParser)
[2018-01-08 15:20:08,680] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)
[2018-01-08 15:20:08,196] WARN Error processing kafka.log:type=LogManager,name=LogDirectoryOffline,logDirectory=C:\tmp\kafka-logs (com.yammer.metrics.reporting.JmxReporter)
javax.management.MalformedObjectNameException: Invalid character ':' in value part of property
というエラーが表示されますがKAFKA-6156を見るとまだリリースされていない1.1.0では修正されているようです。
とりあえず今回は無視します。
4 Kafkaの動作確認
4.1 Topicの作成と確認
DOS窓を立ち上げ、kafka-topics.bat --createでTopicを作成し、kafka-topics.bat --listで確認してみます。
c:\dev\kafka\kafka_2.11-1.0.0\bin\windows>kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic hello-kafka
Created topic "hello-kafka".
c:\dev\kafka\kafka_2.11-1.0.0\bin\windows>kafka-topics.bat --list --zookeeper localhost:2181
hello-kafka
4.2 Producerの起動とメッセージの送信
kafka-console-producer.batでProducerを起動します。>マークが出てきたら、キーボードで適当なメッセージを打ち込みます。
c:\dev\kafka\kafka_2.11-1.0.0\bin\windows>kafka-console-producer.bat --broker-list localhost:9092 --topic hello-kafka
>Hello kafka!
>My name is tworks55
>
※停止する場合は、CTRL+Cを使用します。
4.3 Consumerの起動とメッセージの受信
DOS窓を立ち上げ、kafka-console-consumer.batを起動します。上記ProducerのDOS窓で入力したメッセージが受信できれば成功です。
c:\dev\kafka\kafka_2.11-1.0.0\bin\windows>kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic hello-kafka --from-beginning
Hello kafka!
My name is tworks55
*CTRL+Cで停止でき、再度同じコマンドでメッセージを再受信できます。--from-beginningを外すと同じメッセージは受信されません。
*Procucer側でメッセージを追記すると、リアルタイムで受信できます。
これでインストール完了です。
Kafkaを停止する際ですが、先にZooKeeperを止めてしまうとKafkaがエラーになります。
Consumer, Producer, Kafka, ZooKeeperという順で停止しましょう。
※いろいろな順番で停止してみるとそれぞれの依存関係の理解が深まると思います。
参考URL
https://nifi.apache.org/docs/nifi-docs/html/getting-started.html
https://howtoprogram.xyz/2017/08/03/install-apache-kafka-on-windows/
https://kafka.apache.org/quickstart