kafka partition reassign

ここではサーバーを２台から４台に増設すると想定

現状を確認

$ kafka-topics.sh --describe --zookeeper localhost:2181/kafka  --topic test_queue
Topic:test_queue    PartitionCount:64   ReplicationFactor:2 Configs:
    Topic: test_queue   Partition: 0    Leader: 1   Replicas: 0,1   Isr: 1,0
    Topic: test_queue   Partition: 1    Leader: 1   Replicas: 1,0   Isr: 1,0
...

0,1のみに割り振られている

config作成

topics-to-move.json

{"topics":
     [{"topic": "test_queue"}],
     "version":1
}

実行

$ kafka-reassign-partitions.sh  --zookeeper localhost:2181/kafka --topics-to-move-json-file test.json  --broker-list "0,1,2,3"  --generate

Current partition replica assignment
{"version":1,"partitions":...

Proposed partition reassignment configuration
{"version":1,"partitions":...

'Proposed partition reassignment configuration'以下のJSONをpartitions-to-move.jsonというファイルに保存

$ kafka-reassign-partitions.sh  --zookeeper localhost:2181/kafka --reassignment-json-file partitions-to-move.json  --execute

確認

$ kafka-topics.sh --describe --zookeeper localhost:2181/kafka  --topic test_queue
Topic:test_queue    PartitionCount:64   ReplicationFactor:2 Configs:
    Topic: test_queue   Partition: 0    Leader: 0   Replicas: 0,3   Isr: 0,3
    Topic: test_queue   Partition: 1    Leader: 1   Replicas: 1,0   Isr: 1,0
    Topic: test_queue   Partition: 2    Leader: 1   Replicas: 2,1   Isr: 1,2
...

Replicas, Isrが2,3にも割り振られていればOK

参考
https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignPartitionsTool