クラスタにノードが出入りするときの挙動について

普段の挙動

Riakにデータがコミットされると、どのvnodeに格納するかが計算されて、そのあとに実行される。
複数台構成を前提として、Riakの前にLBを置いてアクセスを分散させている場合、どのノードにいつデータがコミットされるか分からない状態だ。
該当のデータを持っていない（持つべきでない）ノードは、持っているノードに問い合わせることで処理を実行する。

riak-admin cluster leave

今回は、devrelで作成した開発環境でleaveコマンドを試してみる。
本当は物理4台でriakクラスタを構成し、LANケーブルをハサミで切って確かめたかったが、時間もお金もなかったので…

➜  riak git:(a12d5a2) dev/dev1/bin/riak-admin member-status
================================= Membership ==================================
Status     Ring    Pending    Node
-------------------------------------------------------------------------------
valid      25.0%      --      'dev1@127.0.0.1'
valid      25.0%      --      'dev3@127.0.0.1'
valid      25.0%      --      'dev4@127.0.0.1'
valid      25.0%      --      'dev5@127.0.0.1'
-------------------------------------------------------------------------------
Valid:4 / Leaving:0 / Exiting:0 / Joining:0 / Down:0

このような4台のクラスタだとする。
で、dev4を操作し、クラスタから離脱させる。

➜  riak git:(a12d5a2) dev/dev4/bin/riak-admin cluster leave
Success: staged leave request for 'dev4@127.0.0.1'
###############################################################################
                         After cluster transition 1/2
###############################################################################

================================= Membership ==================================
Status     Ring    Pending    Node
-------------------------------------------------------------------------------
leaving    25.0%      0.0%    'dev4@127.0.0.1'
valid      25.0%     34.4%    'dev1@127.0.0.1'
valid      25.0%     32.8%    'dev3@127.0.0.1'
valid      25.0%     32.8%    'dev5@127.0.0.1'
-------------------------------------------------------------------------------
Valid:3 / Leaving:1 / Exiting:0 / Joining:0 / Down:0

###############################################################################
                         After cluster transition 2/2
###############################################################################

================================= Membership ==================================
Status     Ring    Pending    Node
-------------------------------------------------------------------------------
valid      34.4%      --      'dev1@127.0.0.1'
valid      32.8%      --      'dev3@127.0.0.1'
valid      32.8%      --      'dev5@127.0.0.1'
-------------------------------------------------------------------------------
Valid:3 / Leaving:0 / Exiting:0 / Joining:0 / Down:0

そして変更をコミットさせると、データが移動していくのがわかる。
大体1分くらいで、移動が完了する。

➜  riak git:(a12d5a2) dev/dev1/bin/riak-admin cluster commit
Cluster changes committed

➜  riak git:(a12d5a2) dev/dev1/bin/riak-admin member-status
================================= Membership ==================================
Status     Ring    Pending    Node
-------------------------------------------------------------------------------
valid      25.0%      --      'dev1@127.0.0.1'
valid      25.0%      --      'dev3@127.0.0.1'
valid      25.0%      --      'dev4@127.0.0.1'
valid      25.0%      --      'dev5@127.0.0.1'
-------------------------------------------------------------------------------
Valid:4 / Leaving:0 / Exiting:0 / Joining:0 / Down:0
➜  riak git:(a12d5a2) dev/dev1/bin/riak-admin cluster commit
Cluster changes committed
➜  riak git:(a12d5a2) dev/dev1/bin/riak-admin member-status
================================= Membership ==================================
Status     Ring    Pending    Node
-------------------------------------------------------------------------------
leaving    21.9%      0.0%    'dev4@127.0.0.1'
valid      23.4%     34.4%    'dev1@127.0.0.1'
valid      25.0%     32.8%    'dev3@127.0.0.1'
valid      29.7%     32.8%    'dev5@127.0.0.1'
-------------------------------------------------------------------------------
Valid:3 / Leaving:1 / Exiting:0 / Joining:0 / Down:0
➜  riak git:(a12d5a2) dev/dev1/bin/riak-admin member-status
================================= Membership ==================================
Status     Ring    Pending    Node
-------------------------------------------------------------------------------
leaving    14.1%      0.0%    'dev4@127.0.0.1'
valid      25.0%     34.4%    'dev1@127.0.0.1'
valid      28.1%     32.8%    'dev3@127.0.0.1'
valid      32.8%     32.8%    'dev5@127.0.0.1'
-------------------------------------------------------------------------------
Valid:3 / Leaving:1 / Exiting:0 / Joining:0 / Down:0
➜  riak git:(a12d5a2) dev/dev1/bin/riak-admin member-status
================================= Membership ==================================
Status     Ring    Pending    Node
-------------------------------------------------------------------------------
leaving     9.4%      0.0%    'dev4@127.0.0.1'
valid      28.1%     34.4%    'dev1@127.0.0.1'
valid      29.7%     32.8%    'dev3@127.0.0.1'
valid      32.8%     32.8%    'dev5@127.0.0.1'
-------------------------------------------------------------------------------
Valid:3 / Leaving:1 / Exiting:0 / Joining:0 / Down:0
➜  riak git:(a12d5a2) dev/dev1/bin/riak-admin member-status
================================= Membership ==================================
Status     Ring    Pending    Node
-------------------------------------------------------------------------------
leaving     0.0%      --      'dev4@127.0.0.1'
valid      34.4%      --      'dev1@127.0.0.1'
valid      32.8%      --      'dev3@127.0.0.1'
valid      32.8%      --      'dev5@127.0.0.1'
-------------------------------------------------------------------------------
Valid:3 / Leaving:1 / Exiting:0 / Joining:0 / Down:0

このように、Leaveコマンドを使った場合は段階的にそのノードが持っているデータが他のクラスタに移されることがわかる。

追加したとき

追加するノードが動いていることを確認したら、クラスタにplanで確認してcommitsする。

➜  riak git:(a12d5a2) dev/dev4/bin/riak-admin cluster join dev1@127.0.0.1
Success: staged join request for 'dev4@127.0.0.1' to 'dev1@127.0.0.1'
➜  riak git:(a12d5a2) dev/dev1/bin/riak-admin cluster plan
=============================== Staged Changes ================================
Action         Details(s)
-------------------------------------------------------------------------------
join           'dev4@127.0.0.1'
-------------------------------------------------------------------------------


NOTE: Applying these changes will result in 1 cluster transition

###############################################################################
                         After cluster transition 1/1
###############################################################################

================================= Membership ==================================
Status     Ring    Pending    Node
-------------------------------------------------------------------------------
valid      34.4%     25.0%    'dev1@127.0.0.1'
valid      32.8%     25.0%    'dev3@127.0.0.1'
valid       0.0%     25.0%    'dev4@127.0.0.1'
valid      32.8%     25.0%    'dev5@127.0.0.1'
-------------------------------------------------------------------------------
Valid:4 / Leaving:0 / Exiting:0 / Joining:0 / Down:0

Transfers resulting from cluster changes: 47
  5 transfers from 'dev3@127.0.0.1' to 'dev4@127.0.0.1'
  5 transfers from 'dev1@127.0.0.1' to 'dev3@127.0.0.1'
  5 transfers from 'dev5@127.0.0.1' to 'dev1@127.0.0.1'
  5 transfers from 'dev3@127.0.0.1' to 'dev5@127.0.0.1'
  5 transfers from 'dev1@127.0.0.1' to 'dev4@127.0.0.1'
  5 transfers from 'dev5@127.0.0.1' to 'dev3@127.0.0.1'
  5 transfers from 'dev3@127.0.0.1' to 'dev1@127.0.0.1'
  6 transfers from 'dev1@127.0.0.1' to 'dev5@127.0.0.1'
  6 transfers from 'dev5@127.0.0.1' to 'dev4@127.0.0.1'

➜  riak git:(a12d5a2) dev/dev1/bin/riak-admin cluster commit
Cluster changes committed
➜  riak git:(a12d5a2) dev/dev1/bin/riak-admin member-status
================================= Membership ==================================
Status     Ring    Pending    Node
-------------------------------------------------------------------------------
valid      34.4%     25.0%    'dev1@127.0.0.1'
valid      32.8%     25.0%    'dev3@127.0.0.1'
valid       0.0%     25.0%    'dev4@127.0.0.1'
valid      32.8%     25.0%    'dev5@127.0.0.1'
-------------------------------------------------------------------------------
Valid:4 / Leaving:0 / Exiting:0 / Joining:0 / Down:0

あとは先ほどと同様の挙動を示す。

➜  riak git:(a12d5a2) dev/dev1/bin/riak-admin cluster commit
Cluster changes committed
➜  riak git:(a12d5a2) dev/dev1/bin/riak-admin member-status
================================= Membership ==================================
Status     Ring    Pending    Node
-------------------------------------------------------------------------------
valid      34.4%     25.0%    'dev1@127.0.0.1'
valid      32.8%     25.0%    'dev3@127.0.0.1'
valid       0.0%     25.0%    'dev4@127.0.0.1'
valid      32.8%     25.0%    'dev5@127.0.0.1'
-------------------------------------------------------------------------------
Valid:4 / Leaving:0 / Exiting:0 / Joining:0 / Down:0
➜  riak git:(a12d5a2) dev/dev1/bin/riak-admin member-status
================================= Membership ==================================
Status     Ring    Pending    Node
-------------------------------------------------------------------------------
valid      31.3%     25.0%    'dev1@127.0.0.1'
valid      32.8%     25.0%    'dev3@127.0.0.1'
valid       3.1%     25.0%    'dev4@127.0.0.1'
valid      32.8%     25.0%    'dev5@127.0.0.1'
-------------------------------------------------------------------------------
Valid:4 / Leaving:0 / Exiting:0 / Joining:0 / Down:0
➜  riak git:(a12d5a2) dev/dev1/bin/riak-admin member-status
================================= Membership ==================================
Status     Ring    Pending    Node
-------------------------------------------------------------------------------
valid      25.0%      --      'dev1@127.0.0.1'
valid      25.0%      --      'dev3@127.0.0.1'
valid      25.0%      --      'dev4@127.0.0.1'
valid      25.0%      --      'dev5@127.0.0.1'
-------------------------------------------------------------------------------
Valid:4 / Leaving:0 / Exiting:0 / Joining:0 / Down:0