kafka partition reassign

ここではサーバーを２台から４台に増設すると想定

現状を確認

$ kafka-topics.sh --describe --zookeeper localhost:2181/kafka  --topic test_queue
Topic:test_queue	PartitionCount:64	ReplicationFactor:2	Configs:
	Topic: test_queue	Partition: 0	Leader: 1	Replicas: 0,1	Isr: 1,0
	Topic: test_queue	Partition: 1	Leader: 1	Replicas: 1,0	Isr: 1,0
...

0,1のみに割り振られている

config作成

topics-to-move.json

{"topics":
     [{"topic": "test_queue"}],
     "version":1
}

実行

$ kafka-reassign-partitions.sh  --zookeeper localhost:2181/kafka --topics-to-move-json-file test.json  --broker-list "0,1,2,3"  --generate

Current partition replica assignment
{"version":1,"partitions":...

Proposed partition reassignment configuration
{"version":1,"partitions":...

'Proposed partition reassignment configuration'以下のJSONをpartitions-to-move.jsonというファイルに保存

$ kafka-reassign-partitions.sh  --zookeeper localhost:2181/kafka --reassignment-json-file partitions-to-move.json  --execute

確認

$ kafka-topics.sh --describe --zookeeper localhost:2181/kafka  --topic test_queue
Topic:test_queue	PartitionCount:64	ReplicationFactor:2	Configs:
	Topic: test_queue	Partition: 0	Leader: 0	Replicas: 0,3	Isr: 0,3
	Topic: test_queue	Partition: 1	Leader: 1	Replicas: 1,0	Isr: 1,0
	Topic: test_queue	Partition: 2	Leader: 1	Replicas: 2,1	Isr: 1,2
...

Replicas, Isrが2,3にも割り振られていればOK

参考
https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignPartitionsTool