Edited at

Apache Kafka in Docker on Windows. (その2)


はじめに

Kafka Dockerとkafka-manager Dockerイメージを別々に起動してみる

Kafka Docker中でproducer、consumerプロセスを実行する


参考URL

https://zablo.net/blog/post/setup-apache-kafka-in-docker-on-windows

https://github.com/spotify/docker-kafka

https://hub.docker.com/r/mzagar/kafka-manager-docker/


バージョン確認、docker 起動

Apache Kafka in Docker on Windows.と同じ

https://qiita.com/0ashina0/items/e8b6a3ac4cf19805b15c


Kafka Docker


ダウンロード

URL:https://github.com/spotify/docker-kafka

「Clone or download」をクリックし、「Clone with HTTPS」の「Download ZIP」をクリックする

image.png


解凍

ダウンロードした「docker-kafka-master.zip」を解凍する。


docker起動

解凍したフォルダーdocker-kafka-master\kafkaで起動。(Dockerfileがあるフォルダー)

ADVERTISED_HOSTのIPは、ipconfigコマンドで確認(例:192.175.12.5)

docker run -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=192.175.12.5 --env ADVERTISED_PORT=9092 spotify/kafka

起動イメージ

> docker run -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=192.175.12.5 --env ADVERTISED_PORT=9092 spotify/kafka

/usr/lib/python2.7/dist-packages/supervisor/options.py:296: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
'Supervisord is running as root and it is searching '
2018-10-21 02:45:09,390 CRIT Supervisor running as root (no user in config file)
2018-10-21 02:45:09,391 WARN Included extra file "/etc/supervisor/conf.d/zookeeper.conf" during parsing
2018-10-21 02:45:09,391 WARN Included extra file "/etc/supervisor/conf.d/kafka.conf" during parsing
2018-10-21 02:45:09,403 INFO RPC interface 'supervisor' initialized
2018-10-21 02:45:09,403 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2018-10-21 02:45:09,403 INFO supervisord started with pid 1
2018-10-21 02:45:10,407 INFO spawned: 'zookeeper' with pid 8
2018-10-21 02:45:10,413 INFO spawned: 'kafka' with pid 9
2018-10-21 02:45:11,454 INFO success: zookeeper entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-10-21 02:45:11,454 INFO success: kafka entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)


kafka-manager Docker


ダウンロード

URL:https://github.com/mzagar/kafka-manager-docker

「Clone or download」をクリックし、「Clone with HTTPS」の「Download ZIP」をクリックする

image.png


解凍

ダウンロードした「docker-kafka-master.zip」を解凍する。


docker起動

解凍したフォルダーKafkaManagerで起動。(Dockerfileがあるフォルダー)

ZK_HOSTSのIPは、ipconfigコマンドで確認

docker run -it --rm  -p 9000:9000 -e ZK_HOSTS="192.175.12.5:2181" -e APPLICATION_SECRET=letmein sheepkiller/kafka-manager


Cluster、Topic作成

Apache Kafka in Docker on Windows.と同じ

https://qiita.com/0ashina0/items/e8b6a3ac4cf19805b15c


producer起動

> docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ce40c1bcd279 sheepkiller/kafka-manager "./start-kafka-manag…" 2 hours ago Up 2 hours 0.0.0.0:9000->9000/tcp distracted_wozniak
ae02d73f4bcd spotify/kafka "supervisord -n" 2 hours ago Up 2 hours 0.0.0.0:2181->2181/tcp, 0.0.0.0:9092->9092/tcp gallant_aryabhata
> docker exec -i -t ae02d73f4bcd /bin/bash

root@ae02d73f4bcd:/# cd /opt/kafka_2.11-0.10.1.0/bin/

root@ae02d73f4bcd:/opt/kafka_2.11-0.10.1.0/bin# export KAFKA=192.175.12.5:9092
root@ae02d73f4bcd:/opt/kafka_2.11-0.10.1.0/bin# ./kafka-console-producer.sh --broker-list $KAFKA --topic test


consumer起動

> docker exec -i -t ae02d73f4bcd  /bin/bash

root@ae02d73f4bcd:/# cd /opt/kafka_2.11-0.10.1.0/bin/

root@ae02d73f4bcd:/opt/kafka_2.11-0.10.1.0/bin# export ZOOKEEPER=192.175.12.5:2181
root@ae02d73f4bcd:/opt/kafka_2.11-0.10.1.0/bin# ./kafka-console-consumer.sh --zookeeper $ZOOKEEPER --topic test


producerからメッセージ送信

root@ae02d73f4bcd:/opt/kafka_2.11-0.10.1.0/bin# ./kafka-console-producer.sh --broker-list $KAFKA --topic test

messageA


consumerでメッセージ確認

root@ae02d73f4bcd:/opt/kafka_2.11-0.10.1.0/bin# ./kafka-console-consumer.sh --zookeeper $ZOOKEEPER --topic test

Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
messageA