LoginSignup
0
0

More than 3 years have passed since last update.

@fujita1203

etcd on docker でクラスタ構成を試してみる

Posted at

目的(確認したいこと)

  • etcdをクラスタ構成で動作させるるときのdocker run のパラメータなどの書き方と起動方法。
  • 3台がクラスタ化されているかの確認。
  • リーダー(マスタと)なる1台を停止したときに、他の2台のどちらかにリーダーが引き継がれるか。
  • 停止した1台を再開したときに、クラスタの仲間に戻るか。

環境と構成

  • 動作環境は、GCPのVMを1台。
  • etcd001, etcd002, etcd003と3台のetcd(on docker)をクラスタ構成でVM上に立てる。
  • etcdのコンテナはこちらを利用。(gcr.io/etcd-development/etcd:v3.4.7)

Dockerネットワークの作成

コンテナ間で通信させるための土俵として、
「base-network」という名称で、docker(etcd)が所属するネットワークを作成。

docker network create \
    --driver bridge \
    --subnet=172.19.0.0/24 \
    --gateway=172.19.0.1 \
    base-network

「gateway=172.19.0.1」は、VMのIPアドレス。subnetは、上記に合わせて適宜設定する。

1号機(etcd001)の起動

IPアドレス : 172.19.0.10
ポート番号 : docker内で、etcdがサービスとして使うポートは「2379と2380」の二つ。VM側から参照させるためにそのまま外だしするすると、3台同じ番号になり被るので「1」号機なので「1」をつけてVM側に公開しています。(1号機:2379->23791,2380->23801)(2号機:2379->23792,2380->23802)
データ : データを永続化(dockerを落としてもデータが消えないように)させるため、事前にVM側にディレクトリを作ってマウント /tmp/etcd-data-001.tmp/ しています。

分かりやすく標準出力にログを出しているので、サービス起動時はターミナルを占拠します。

docker run \
  --network base-network \
  --ip 172.19.0.10 \
  -p 23791:2379 \
  -p 23801:2380 \
  --mount type=bind,source=/tmp/etcd-data-001.tmp,destination=/etcd-data \
  --name etcd001 \
  gcr.io/etcd-development/etcd:v3.4.7 \
  /usr/local/bin/etcd \
  --name etcd001 \
  --data-dir /etcd-data \
  --listen-client-urls http://172.19.0.10:23791,http://127.0.0.1:2379 \
  --advertise-client-urls http://172.19.0.10:23791 \
  --listen-peer-urls http://172.19.0.10:23801 \
  --initial-advertise-peer-urls http://172.19.0.10:23801 \
  --initial-cluster etcd001=http://172.19.0.10:23801,etcd002=http://172.19.0.20:23802,etcd003=http://172.19.0.30:23803 \
  --initial-cluster-token tkn \
  --initial-cluster-state new \
  --log-level info \
  --logger zap \
  --log-outputs stderr

2号機(etcd002)の起動

IPアドレス: 172.19.0.20
ポート番号: 2379->23792,2380->23802
データ: /tmp/etcd-data-002.tmp/

docker run \
  --network base-network \
  --ip 172.19.0.20 \
  -p 23792:2379 \
  -p 23802:2380 \
  --mount type=bind,source=/tmp/etcd-data-002.tmp,destination=/etcd-data \
  --name etcd002 \
  gcr.io/etcd-development/etcd:v3.4.7 \
  /usr/local/bin/etcd \
  --name etcd002 \
  --data-dir /etcd-data \
  --listen-client-urls http://172.19.0.20:23792,http://127.0.0.1:2379 \
  --advertise-client-urls http://172.19.0.20:23792 \
  --listen-peer-urls http://172.19.0.20:23802 \
  --initial-advertise-peer-urls http://172.19.0.20:23802 \
  --initial-cluster etcd001=http://172.19.0.10:23801,etcd002=http://172.19.0.20:23802,etcd003=http://172.19.0.30:23803 \
  --initial-cluster-token tkn \
  --initial-cluster-state new \
  --log-level info \
  --logger zap \
  --log-outputs stderr

3号機(etcd003)の起動

IPアドレス: 172.19.0.30
ポート番号: 2379->23793,2380->23803
データ: /tmp/etcd-data-003.tmp/

docker run \
  --network base-network \
  --ip 172.19.0.30 \
  -p 23793:2379 \
  -p 23803:2380 \
  --mount type=bind,source=/tmp/etcd-data-003.tmp,destination=/etcd-data \
  --name etcd003 \
  gcr.io/etcd-development/etcd:v3.4.7 \
  /usr/local/bin/etcd \
  --name etcd003 \
  --data-dir /etcd-data \
  --listen-client-urls http://172.19.0.30:23793,http://127.0.0.1:2379 \
  --advertise-client-urls http://172.19.0.30:23793 \
  --listen-peer-urls http://172.19.0.30:23803 \
  --initial-advertise-peer-urls http://172.19.0.30:23803 \
  --initial-cluster etcd001=http://172.19.0.10:23801,etcd002=http://172.19.0.20:23802,etcd003=http://172.19.0.30:23803 \
  --initial-cluster-token tkn \
  --initial-cluster-state new \
  --log-level info \
  --logger zap \
  --log-outputs stderr

3台がクラスタ化されてるかなの確認

お。できてるぽい。
そして1号機がリーダー(マスター)のようだ。(endpoint statusのIS LEADERで確認)

・member list

[fujita1203@instance-1 ~]$ docker exec etcd001 /bin/sh -c "ETCDCTL_API=3 && /usr/local/bin/etcdctl -w table member list"
+------------------+---------+---------+--------------------------+--------------------------+------------+
|        ID        | STATUS  |  NAME   |        PEER ADDRS        |       CLIENT ADDRS       | IS LEARNER |
+------------------+---------+---------+--------------------------+--------------------------+------------+
| a5a9dff0272bf242 | started | etcd003 | http://172.19.0.30:23803 | http://172.19.0.30:23793 |      false |
| a9f542562b5fed3d | started | etcd001 | http://172.19.0.10:23801 | http://172.19.0.10:23791 |      false |
| db0290dd0457c29f | started | etcd002 | http://172.19.0.20:23802 | http://172.19.0.20:23792 |      false |
+------------------+---------+---------+--------------------------+--------------------------+------------+

・endpoint health

[fujita1203@instance-1 ~]$ docker exec etcd003 /bin/sh -c "ETCDCTL_API=3 /usr/local/bin/etcdctl -w table --endpoints http://172.19.0.10:23791,http://172.19.0.20:23792,http://172.19.0.30:23793 endpoint health"
+--------------------------+--------+--------------+-------+
|         ENDPOINT         | HEALTH |     TOOK     | ERROR |
+--------------------------+--------+--------------+-------+
| http://172.19.0.30:23793 |   true | 310.139671ms |       |
| http://172.19.0.20:23792 |   true |  310.08538ms |       |
| http://172.19.0.10:23791 |   true | 310.206383ms |       |
+--------------------------+--------+--------------+-------+

・endpoint status

[fujita1203@instance-1 ~]$ docker exec etcd003 /bin/sh -c "ETCDCTL_API=3 /usr/local/bin/etcdctl -w table --endpoints http://172.19.0.10:23791,http://172.19.0.20:23792,http://172.19.0.30:23793 endpoint status"
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|         ENDPOINT         |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| http://172.19.0.10:23791 | a9f542562b5fed3d |   3.4.7 |   20 kB |      true |      false |        15 |          9 |                  9 |        |
| http://172.19.0.20:23792 | db0290dd0457c29f |   3.4.7 |   20 kB |     false |      false |        15 |          9 |                  9 |        |
| http://172.19.0.30:23793 | a5a9dff0272bf242 |   3.4.7 |   20 kB |     false |      false |        15 |          9 |                  9 |        |
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
  • 最初は値が入ってないことを確認
[fujita1203@instance-1 ~]$ curl -L http://172.19.0.10:23801/v3/kv/range -X POST -d '{"key": "Zm9v"}'
404 page not found
[fujita1203@instance-1 ~]$ curl -L http://172.19.0.20:23802/v3/kv/range -X POST -d '{"key": "Zm9v"}'
404 page not found
[fujita1203@instance-1 ~]$ curl -L http://172.19.0.30:23803/v3/kv/range -X POST -d '{"key": "Zm9v"}'
404 page not found
  • 3号機に入れたものが、1,2,3号機から取得できるか確認

コマンド

curl -L http://172.19.0.30:23793/v3/kv/put -X POST -d '{"key": "Zm9v", "value": "YmFy"}' | python -mjson.tool
curl -L http://172.19.0.10:23791/v3/kv/range -X POST -d '{"key": "Zm9v"}' | python -mjson.tool
curl -L http://172.19.0.20:23792/v3/kv/range -X POST -d '{"key": "Zm9v"}' | python -mjson.tool
curl -L http://172.19.0.30:23793/v3/kv/range -X POST -d '{"key": "Zm9v"}' | python -mjson.tool

とれたとれた。(長いので1号機だけ) 

[fujita1203@instance-1 ~]$ curl -L http://172.19.0.10:23791/v3/kv/range -X POST -d '{"key": "Zm9v"}' | python -mjson.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   234  100   219  100    15  19527   1337 --:--:-- --:--:-- --:--:-- 21900
{
    "count": "1",
    "header": {
        "cluster_id": "3282096315814289506",
        "member_id": "12246767699566259517",
        "raft_term": "16",
        "revision": "2"
    },
    "kvs": [
        {
            "create_revision": "2",
            "key": "Zm9v",
            "mod_revision": "2",
            "value": "YmFy",
            "version": "1"
        }
    ]
}

1号機(リーダー)を停止したときの挙動と、復帰したときの挙動の確認

  • リーダーを強制終了(ctr+c)してみる (リーダーがetcd002になってる)
[fujita1203@instance-1 ~]$ docker exec etcd003 /bin/sh -c "ETCDCTL_API=3 /usr/local/bin/etcdctl -w table --endpoints http://172.19.0.10:23791,http://172.19.0.20:23792,http://172.19.0.30:23793 endpoint status"
{"level":"warn","ts":"2020-04-12T23:19:41.068Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"passthrough:///http://172.19.0.10:23791","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Failed to get the status of endpoint http://172.19.0.10:23791 (context deadline exceeded)
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|         ENDPOINT         |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| http://172.19.0.20:23792 | db0290dd0457c29f |   3.4.7 |   20 kB |      true |      false |        16 |         11 |                 11 |        |
| http://172.19.0.30:23793 | a5a9dff0272bf242 |   3.4.7 |   20 kB |     false |      false |        16 |         11 |                 11 |        |
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
[fujita1203@instance-1 ~]$ docker exec etcd003 /bin/sh -c "ETCDCTL_API=3 /usr/local/bin/etcdctl -w table member list"
+------------------+---------+---------+--------------------------+--------------------------+------------+
|        ID        | STATUS  |  NAME   |        PEER ADDRS        |       CLIENT ADDRS       | IS LEARNER |
+------------------+---------+---------+--------------------------+--------------------------+------------+
| a5a9dff0272bf242 | started | etcd003 | http://172.19.0.30:23803 | http://172.19.0.30:23793 |      false |
| a9f542562b5fed3d | started | etcd001 | http://172.19.0.10:23801 | http://172.19.0.10:23791 |      false |
| db0290dd0457c29f | started | etcd002 | http://172.19.0.20:23802 | http://172.19.0.20:23792 |      false |
+------------------+---------+---------+--------------------------+--------------------------+------------+
[fujita1203@instance-1 ~]$ docker exec etcd003 /bin/sh -c "ETCDCTL_API=3 /usr/local/bin/etcdctl -w table --endpoints http://172.19.0.10:23791,http://172.19.0.20:23792,http://172.19.0.30:23793 endpoint health"
+--------------------------+--------+--------------+---------------------------+
|         ENDPOINT         | HEALTH |     TOOK     |           ERROR           |
+--------------------------+--------+--------------+---------------------------+
| http://172.19.0.30:23793 |   true | 270.115572ms |                           |
| http://172.19.0.20:23792 |   true |   269.9296ms |                           |
| http://172.19.0.10:23791 |  false | 5.000427627s | context deadline exceeded |
+--------------------------+--------+--------------+---------------------------+
{"level":"warn","ts":"2020-04-12T23:22:39.902Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-b4e8b2e6-5e62-4210-b353-dd518d9051a3/172.19.0.10:23791","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = \"transport: Error while dialing dial tcp 172.19.0.10:23791: connect: no route to host\""}
Error: unhealthy cluster
[fujita1203@instance-1 ~]$
  • 1号機を起動させる (クラスタに仲間入りした。2号機がリーダーのまま。1号機から値がとれる。)
[fujita1203@instance-1 ~]$ docker exec etcd003 /bin/sh -c "ETCDCTL_API=3 /usr/local/bin/etcdctl -w table --endpoints http://172.19.0.10:23791,http://172.19.0.20:23792,http://172.19.0.30:23793 endpoint status"
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|         ENDPOINT         |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| http://172.19.0.10:23791 | a9f542562b5fed3d |   3.4.7 |   20 kB |     false |      false |        16 |         12 |                 12 |        |
| http://172.19.0.20:23792 | db0290dd0457c29f |   3.4.7 |   20 kB |      true |      false |        16 |         12 |                 12 |        |
| http://172.19.0.30:23793 | a5a9dff0272bf242 |   3.4.7 |   20 kB |     false |      false |        16 |         12 |                 12 |        |
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
[fujita1203@instance-1 ~]$ docker exec etcd003 /bin/sh -c "ETCDCTL_API=3 /usr/local/bin/etcdctl -w table member list"
+------------------+---------+---------+--------------------------+--------------------------+------------+
|        ID        | STATUS  |  NAME   |        PEER ADDRS        |       CLIENT ADDRS       | IS LEARNER |
+------------------+---------+---------+--------------------------+--------------------------+------------+
| a5a9dff0272bf242 | started | etcd003 | http://172.19.0.30:23803 | http://172.19.0.30:23793 |      false |
| a9f542562b5fed3d | started | etcd001 | http://172.19.0.10:23801 | http://172.19.0.10:23791 |      false |
| db0290dd0457c29f | started | etcd002 | http://172.19.0.20:23802 | http://172.19.0.20:23792 |      false |
+------------------+---------+---------+--------------------------+--------------------------+------------+
[fujita1203@instance-1 ~]$ docker exec etcd003 /bin/sh -c "ETCDCTL_API=3 /usr/local/bin/etcdctl -w table --endpoints http://172.19.0.10:23791,http://172.19.0.20:23792,http://172.19.0.30:23793 endpoint health"
+--------------------------+--------+--------------+-------+
|         ENDPOINT         | HEALTH |     TOOK     | ERROR |
+--------------------------+--------+--------------+-------+
| http://172.19.0.30:23793 |   true | 145.849727ms |       |
| http://172.19.0.10:23791 |   true |  145.79547ms |       |
| http://172.19.0.20:23792 |   true | 145.736273ms |       |
+--------------------------+--------+--------------+-------+
[fujita1203@instance-1 ~]$ curl -L http://172.19.0.10:23791/v3/kv/range -X POST -d '{"key": "Zm9v"}' | python -mjson.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   234  100   219  100    15  19527   1337 --:--:-- --:--:-- --:--:-- 21900
{
    "count": "1",
    "header": {
        "cluster_id": "3282096315814289506",
        "member_id": "12246767699566259517",
        "raft_term": "16",
        "revision": "2"
    },
    "kvs": [
        {
            "create_revision": "2",
            "key": "Zm9v",
            "mod_revision": "2",
            "value": "YmFy",
            "version": "1"
        }
    ]
}

一通り今回確認したかったことは確認できたかな。

参考

https://tech-lab.sios.jp/archives/13221
http://pj-doaa.hatenablog.com/entry/2019/09/04/144520
https://qiita.com/pocket8137/items/ef44ca68ffc0f4e70995
https://qiita.com/hana_shin/items/602f98bd9b153d22e50c

そもそもetcdって何状態だったので、とても参考になりました。
ありがとうございました。

0
0
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
0
0