LoginSignup
13
13

More than 5 years have passed since last update.

@AKB428

Mac OSXにApache Kafkaをインストールして使えるようにするまで

Last updated at Posted at 2015-06-04

Home brewでApache Kafkaインストール

gradleが入っていない場合gradleのインストールも始まる。
MacBookAirで10分以上時間かかったので注意

brew install kafka

==> Installing kafka
==> Downloading http://mirrors.ibiblio.org/apache/kafka/0.8.2.1/kafka-0.8.2.1-src.tgz
######################################################################## 100.0%
==> gradle
==> gradle jar


==> Caveats
To start Kafka, ensure that ZooKeeper is running and then execute:
  kafka-server-start.sh /usr/local/etc/kafka/server.properties

To have launchd start kafka at login:
    ln -sfv /usr/local/opt/kafka/*.plist ~/Library/LaunchAgents
Then to load kafka now:
    launchctl load ~/Library/LaunchAgents/homebrew.mxcl.kafka.plist
==> Summary
  /usr/local/Cellar/kafka/0.8.2.1: 3817 files, 38M, built in 12.9 minutes

Zookeeperもインストールする

brew intall zookeeper

Zookeeper 起動

zkServer start

kafka 起動

$ kafka-server-start.sh /usr/local/etc/kafka/server.properties
[2015-06-04 16:01:41,857] INFO Verifying properties (kafka.utils.VerifiableProperties)
[2015-06-04 16:01:41,930] INFO Property broker.id is overridden to 0 (kafka.utils.VerifiableProperties)
[2015-06-04 16:01:41,930] INFO Property log.cleaner.enable is overridden to false (kafka.utils.VerifiableProperties)
[2015-06-04 16:01:41,930] INFO Property log.dirs is overridden to /usr/local/var/lib/kafka-logs (kafka.utils.VerifiableProperties)
[2015-06-04 16:01:41,931] INFO Property log.retention.check.interval.ms is overridden to 300000 (kafka.utils.VerifiableProperties)
[2015-06-04 16:01:41,931] INFO Property log.retention.hours is overridden to 168 (kafka.utils.VerifiableProperties)
[2015-06-04 16:01:41,931] INFO Property log.segment.bytes is overridden to 1073741824 (kafka.utils.VerifiableProperties)
[2015-06-04 16:01:41,931] INFO Property num.io.threads is overridden to 8 (kafka.utils.VerifiableProperties)
[2015-06-04 16:01:41,931] INFO Property num.network.threads is overridden to 3 (kafka.utils.VerifiableProperties)
[2015-06-04 16:01:41,932] INFO Property num.partitions is overridden to 1 (kafka.utils.VerifiableProperties)
[2015-06-04 16:01:41,932] INFO Property num.recovery.threads.per.data.dir is overridden to 1 (kafka.utils.VerifiableProperties)
[2015-06-04 16:01:41,932] INFO Property port is overridden to 9092 (kafka.utils.VerifiableProperties)
[2015-06-04 16:01:41,932] INFO Property socket.receive.buffer.bytes is overridden to 102400 (kafka.utils.VerifiableProperties)
[2015-06-04 16:01:41,932] INFO Property socket.request.max.bytes is overridden to 104857600 (kafka.utils.VerifiableProperties)
[2015-06-04 16:01:41,933] INFO Property socket.send.buffer.bytes is overridden to 102400 (kafka.utils.VerifiableProperties)
[2015-06-04 16:01:41,933] INFO Property zookeeper.connect is overridden to localhost:2181 (kafka.utils.VerifiableProperties)
[2015-06-04 16:01:41,933] INFO Property zookeeper.connection.timeout.ms is overridden to 6000 (kafka.utils.VerifiableProperties)
[2015-06-04 16:01:42,002] INFO [Kafka Server 0], starting (kafka.server.KafkaServer)
[2015-06-04 16:01:42,006] INFO [Kafka Server 0], Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2015-06-04 16:01:42,022] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2015-06-04 16:01:42,036] INFO Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT (org.apache.zookeeper.ZooKeeper)
[2015-06-04 16:01:42,036] INFO Client environment:host.name=192.168.107.80 (org.apache.zookeeper.ZooKeeper)
[2015-06-04 16:01:42,036] INFO Client environment:java.version=1.7.0_79 (org.apache.zookeeper.ZooKeeper)
[2015-06-04 16:01:42,036] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2015-06-04 16:01:42,036] INFO Client environment:java.home=/Library/Java/JavaVirtualMachines/jdk1.7.0_79.jdk/Contents/Home/jre (org.apache.zookeeper.ZooKeeper)
[2015-06-04 16:01:42,036] INFO Client environment:java.class.path=:/usr/local/Cellar/kafka/0.8.2.1/libexec/bin/../core/build/dependant-libs-2.10.4/jopt-simple-3.2.jar:/usr/local/Cellar/kafka/0.8.2.1/libexec/bin/../core/build/dependant-libs-2.10.4/log4j-1.2.16.jar:/usr/local/Cellar/kafka/0.8.2.1/libexec/bin/../core/build/dependant-libs-2.10.4/lz4-1.2.0.jar:/usr/local/Cellar/kafka/0.8.2.1/libexec/bin/../core/build/dependant-libs-2.10.4/metrics-core-2.2.0.jar:/usr/local/Cellar/kafka/0.8.2.1/libexec/bin/../core/build/dependant-libs-2.10.4/scala-library-2.10.4.jar:/usr/local/Cellar/kafka/0.8.2.1/libexec/bin/../core/build/dependant-libs-2.10.4/slf4j-api-1.7.6.jar:/usr/local/Cellar/kafka/0.8.2.1/libexec/bin/../core/build/dependant-libs-2.10.4/slf4j-log4j12-1.7.6.jar:/usr/local/Cellar/kafka/0.8.2.1/libexec/bin/../core/build/dependant-libs-2.10.4/snappy-java-1.1.1.6.jar:/usr/local/Cellar/kafka/0.8.2.1/libexec/bin/../core/build/dependant-libs-2.10.4/zkclient-0.3.jar:/usr/local/Cellar/kafka/0.8.2.1/libexec/bin/../core/build/dependant-libs-2.10.4/zookeeper-3.4.6.jar:/usr/local/Cellar/kafka/0.8.2.1/libexec/bin/../examples/build/libs//kafka-examples-0.8.2.1.jar:/usr/local/Cellar/kafka/0.8.2.1/libexec/bin/../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer-0.8.2.1.jar:/usr/local/Cellar/kafka/0.8.2.1/libexec/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer-0.8.2.1.jar:/usr/local/Cellar/kafka/0.8.2.1/libexec/bin/../clients/build/libs/kafka-clients-0.8.2.1.jar:/usr/local/Cellar/kafka/0.8.2.1/libexec/bin/../libs/*.jar:/usr/local/Cellar/kafka/0.8.2.1/libexec/bin/../core/build/libs/kafka_2.10-0.8.2.1.jar (org.apache.zookeeper.ZooKeeper)
[2015-06-04 16:01:42,037] INFO Client environment:java.library.path=/Users/xxxx/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:. (org.apache.zookeeper.ZooKeeper)
[2015-06-04 16:01:42,037] INFO Client environment:java.io.tmpdir=/var/folders/xq/cjwpdzb90_zdsrhcfd4ckgyc0000gq/T/ (org.apache.zookeeper.ZooKeeper)
[2015-06-04 16:01:42,037] INFO Client environment:java.compiler= (org.apache.zookeeper.ZooKeeper)
[2015-06-04 16:01:42,037] INFO Client environment:os.name=Mac OS X (org.apache.zookeeper.ZooKeeper)
[2015-06-04 16:01:42,037] INFO Client environment:os.arch=x86_64 (org.apache.zookeeper.ZooKeeper)
[2015-06-04 16:01:42,037] INFO Client environment:os.version=10.9.5 (org.apache.zookeeper.ZooKeeper)
[2015-06-04 16:01:42,037] INFO Client environment:user.name=xxxx (org.apache.zookeeper.ZooKeeper)
[2015-06-04 16:01:42,037] INFO Client environment:user.home=/Users/xxx (org.apache.zookeeper.ZooKeeper)
[2015-06-04 16:01:42,037] INFO Client environment:user.dir=/Users/xxx (org.apache.zookeeper.ZooKeeper)
[2015-06-04 16:01:42,038] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@675c2785 (org.apache.zookeeper.ZooKeeper)
[2015-06-04 16:01:42,078] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2015-06-04 16:01:42,090] INFO Socket connection established to localhost/127.0.0.1:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2015-06-04 16:01:42,157] INFO Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x14dbd5f98f80000, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2015-06-04 16:01:42,158] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2015-06-04 16:01:42,312] INFO Log directory '/usr/local/var/lib/kafka-logs' not found, creating it. (kafka.log.LogManager)
[2015-06-04 16:01:42,329] INFO Loading logs. (kafka.log.LogManager)
[2015-06-04 16:01:42,339] INFO Logs loading complete. (kafka.log.LogManager)
[2015-06-04 16:01:42,340] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2015-06-04 16:01:42,347] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2015-06-04 16:01:42,390] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
[2015-06-04 16:01:42,391] INFO [Socket Server on Broker 0], Started (kafka.network.SocketServer)
[2015-06-04 16:01:42,515] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
[2015-06-04 16:01:42,576] INFO 0 successfully elected as leader (kafka.server.ZookeeperLeaderElector)
[2015-06-04 16:01:42,705] INFO Registered broker 0 at path /brokers/ids/0 with address 192.168.107.80:9092. (kafka.utils.ZkUtils$)
[2015-06-04 16:01:42,727] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
[2015-06-04 16:01:42,815] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
```

トピック作成

/usr/local/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

ログ
Created topic "test"

メッセージ送信

/usr/local/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
[2015-06-04 16:11:01,693] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
HELLO Kafka
second line
(^_^)
あいうえお

メッセージ受信

/usr/local/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
HELLO Kafka
second line
(^_^)
あいうえお
13
13
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
13
13