概要
kubernetesでredis cluster構築してみた備忘録
https://github.com/kelseyhightower/kubernetes-redis-cluster
ほぼこちらをそのまま使っている
- Redis server version、3.2.0
- データの永続化はしない
- redisクラスタ基本master×3にレプリカ用 slave×3
redis (loadbalancer)
|
|- [ redis-1(master) ] - [redis-4(slave)]
|- [ redis-2(master) ] - [redis-5(slave)]
|- [ redis-3(master) ] - [redis-6(slave)]
-
maxmemory-policyは、デフォルトvolatile-lru
`volatile-lru` remove the key with an expire set using an LRU algorithm
セッション保存時に、有効期限を設定する想定
redis-1 ~ redis6もreplicasetでpod作成しているので設定を変えれば複数pod起動可能。
1podずつで以下構築と確認を行う。
ソース
構築
- kubernetesでreplicasetとserviceでpodとloadbalancerを起動する
$ kubectl create configmap redis-conf --from-file=redis.conf
$ kubectl create -f replicasets
$ kubectl create -f services
- podをredis clusterに繋ぐ
redis cluster作成コマンド実行用podを起動する
$ kubectl run -i --tty ubuntu --image=ubuntu --restart=Never /bin/bash
必要なライブラリー類をインストールする
$ apt-get update
$ apt-get install ruby vim wget redis-tools
$ gem install redis
$ wget http://download.redis.io/redis-stable/src/redis-trib.rb
$ chmod 755 redis-trib.rb
redis clusterを作成する
$ ./redis-trib.rb create --replicas 1 \
10.107.255.1:6379 \
10.107.255.2:6379 \
10.107.255.3:6379 \
10.107.255.4:6379 \
10.107.255.5:6379 \
10.107.255.6:6379
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
10.107.255.1:6379
10.107.255.2:6379
10.107.255.3:6379
Adding replica 10.107.255.4:6379 to 10.107.255.1:6379
Adding replica 10.107.255.5:6379 to 10.107.255.2:6379
Adding replica 10.107.255.6:6379 to 10.107.255.3:6379
M: d4dfb6366289f581e395428811af4bb6e1f07e2d 10.107.255.1:6379 10.104.1.34
slots:0-5460 (5461 slots) master
M: 4b23c907ef408e5efd6225fffd11a6cdff5de337 10.107.255.2:6379 10.104.1.35
slots:5461-10922 (5462 slots) master
M: 189ee8e87202326675adef4f507a80b5b5873369 10.107.255.3:6379 10.104.1.36
slots:10923-16383 (5461 slots) master
S: 37b2b6b625d8bfba7c8997abfe894df099761937 10.107.255.4:6379 10.104.1.37
replicates d4dfb6366289f581e395428811af4bb6e1f07e2d
S: f2229c73056a819f3cee6521d231e65ce1f4c180 10.107.255.5:6379 10.104.0.14
replicates 4b23c907ef408e5efd6225fffd11a6cdff5de337
S: ad1aa4c73849622068c96976998f778673a2ce4c 10.107.255.6:6379 10.104.1.38
replicates 189ee8e87202326675adef4f507a80b5b5873369
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join..
>>> Performing Cluster Check (using node 10.107.255.1:6379)
M: d4dfb6366289f581e395428811af4bb6e1f07e2d 10.107.255.1:6379
slots:0-5460 (5461 slots) master
1 additional replica(s)
M: 189ee8e87202326675adef4f507a80b5b5873369 10.104.1.36:6379
slots:10923-16383 (5461 slots) master
1 additional replica(s)
M: 4b23c907ef408e5efd6225fffd11a6cdff5de337 10.104.1.35:6379
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: ad1aa4c73849622068c96976998f778673a2ce4c 10.104.1.38:6379
slots: (0 slots) slave
replicates 189ee8e87202326675adef4f507a80b5b5873369
S: f2229c73056a819f3cee6521d231e65ce1f4c180 10.104.0.14:6379
slots: (0 slots) slave
replicates 4b23c907ef408e5efd6225fffd11a6cdff5de337
S: 37b2b6b625d8bfba7c8997abfe894df099761937 10.104.1.37:6379
slots: (0 slots) slave
replicates d4dfb6366289f581e395428811af4bb6e1f07e2d
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
確認
dockerを落としても他のredisノードから値が帰ってくる事を確認する
- masterを1pod止めて、自動起動するか確認する
pod一覧
$ kubectl get pods --show-all
NAME READY STATUS RESTARTS AGE
redis-1-2pg4v 1/1 Running 0 56m
redis-2-fbzks 1/1 Running 0 56m
redis-3-4fs93 1/1 Running 0 56m
redis-4-pjxt1 1/1 Running 0 56m
redis-5-bt1wm 1/1 Running 0 56m
redis-6-qpvvl 1/1 Running 0 56m
ubuntu 1/1 Running 0 20m
redis-1(master)podを止める
$ kubectl stop pod redis-1-2pg4v
Command "stop" is deprecated, use "delete" instead.
pod "redis-1-2pg4v" deleted
pod一覧
$ kubectl get pods --show-all
NAME READY STATUS RESTARTS AGE
redis-1-b7blk 1/1 Running 0 3m ← redis-1(master)のpodが新たに起動された
redis-2-fbzks 1/1 Running 0 1h
redis-3-4fs93 1/1 Running 0 1h
redis-4-pjxt1 1/1 Running 0 1h
redis-5-bt1wm 1/1 Running 0 1h
redis-6-qpvvl 1/1 Running 0 1h
ubuntu 1/1 Running 0 24m
- dockerを止めても正常にデータ取得できるか
a12948@gke-satsuma-redis-default-pool-f99bcf1a-cft1 ~ $ docker ps -a | grep redis-server
4b1cac0124e9 redis:3.2.0-alpine "redis-server /etc/re" 8 minutes ago Up 8 minutes k8s_redis.638d2d73_redis-1-b7blk_default_2df04edf-e91c-11e6-999b-42010af00135_72d2a8a6
6fef91ca8ca5 redis:3.2.0-alpine "redis-server /etc/re" 5 hours ago Exited (0) 8 minutes ago k8s_redis.638d2d73_redis-1-b7blk_default_2df04edf-e91c-11e6-999b-42010af00135_74b5a100
$ kubectl exec -it ubuntu sh
# redis-cli -h 10.107.244.37 -c
10.107.244.37:6379> get hoge
"fuga"
10.107.244.37:6379> get hogehoge
"fugafuga"
ノードを落としてもredisノードから値が帰ってくる事を確認する
！！container clusterのnodeが1つになってしまうと自動起動でいないpodができてしまうため、
少なくても2つのnodeが必要である。
nodeが1つ落ちてしまった時を考慮し、クラスタの設定を以下にする
- node最小サイズ3
- container cluster 自動スケーリング
ON
$ kubectl get nodes
NAME STATUS AGE
gke-satsuma-redis-default-pool-f99bcf1a-cft1 Ready 2d
gke-satsuma-redis-default-pool-f99bcf1a-lj8d Ready 9m
$ kubectl delete node gke-satsuma-redis-default-pool-f99bcf1a-cft1
node "gke-satsuma-redis-default-pool-f99bcf1a-cft1" deleted
$ kubectl get nodes
NAME STATUS AGE
gke-satsuma-redis-default-pool-f99bcf1a-lj8d Ready 10m
$ kubectl exec -it ubuntu sh
# redis-cli -h 10.107.244.37 -c
10.107.244.37:6379> get hoge
-> Redirected to slot [1525] located at 10.104.0.20:6379
"fuga"
10.104.0.20:6379> get hogehoge
"fugafuga"
10.104.0.20:6379> get hoge
"fuga"
10.104.0.20:6379> get hogehoge
"fugafuga"
serviceに紐付けて全redisノードにランダムアクセスするも、値を正常に取得できる事を確認する
- service
redisがredis-1〜redis-6endpointに向いているか確認
$ kubectl get services
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes 10.107.240.1 <none> 443/TCP 2d
redis 10.107.244.37 <none> 6379/TCP,16379/TCP 52m
redis-1 10.107.255.1 <none> 6379/TCP,16379/TCP 52m
redis-2 10.107.255.2 <none> 6379/TCP,16379/TCP 52m
redis-3 10.107.255.3 <none> 6379/TCP,16379/TCP 52m
redis-4 10.107.255.4 <none> 6379/TCP,16379/TCP 52m
redis-5 10.107.255.5 <none> 6379/TCP,16379/TCP 52m
redis-6 10.107.255.6 <none> 6379/TCP,16379/TCP 52m
$ kubectl describe service redis
Name: redis
Namespace: default
Labels: <none>
Selector: app=redis
Type: ClusterIP
IP: 10.107.244.37
Port: redis 6379/TCP
Endpoints: 10.104.0.18:6379,10.104.0.19:6379,10.104.0.20:6379 + 3 more...
Port: cluster 16379/TCP
Endpoints: 10.104.0.18:16379,10.104.0.19:16379,10.104.0.20:16379 + 3 more...
Session Affinity: None
- masterでデータを挿入し、loadbalancerからデータが正しく取得できるか確認する
redis-1(master)にてデータを挿入する
$ kubectl exec -it redis-1-2pg4v -- sh
/data # redis-cli -c
127.0.0.1:6379> set hoge fuga
OK
redis-3(master)にてデータを挿入する
$ kubectl exec -it redis-3-4fs93 -- sh
/data # redis-cli -c
127.0.0.1:6379> set hogehoge fugafuga
-> Redirected to slot [694] located at 10.104.1.44:6379
OK
redis(loadalancer)にてデータを取得してみる
$ kubectl exec -it ubuntu -- sh
# redis-cli -h 10.107.244.37 -c
10.107.244.37:6379> get hoge
-> Redirected to slot [1525] located at 10.104.1.44:6379
"fuga"
10.104.1.44:6379> get hogehoge
"fugafuga"
redis-2(master)にて取得してみる
$ kubectl exec -it redis-2-fbzks sh (feature/redis✱)
/data # redis-cli -c
127.0.0.1:6379> get hoge
-> Redirected to slot [1525] located at 10.104.1.46:6379
"fuga"
10.104.1.46:6379> get hogehoge
"fugafuga"
redis-5(slave)にて取得してみる
$ kubectl exec -it redis-5-bt1wm sh
/data # redis-cli -c
127.0.0.1:6379> get hoge
-> Redirected to slot [1525] located at 10.104.1.46:6379
"fuga"
10.104.1.46:6379> get hogehoge
"fugafuga"
検証
- gcp redis上で1000000万件データ作成
$ redis-cli debug populate 1000000
10.104.0.6:6379> mget key:0
1) "value:0"
10.104.1.4:6379> mget key:99999
-> Redirected to slot [2036] located at 10.104.0.6:6379
1) "value:99999"
- 使用memory確認
$ redis-cli info
# Memory
used_memory:2278256
used_memory_human:2.17M
used_memory_rss:3805184
used_memory_peak:2278256
used_memory_peak_human:2.17M
used_memory_lua:36864
mem_fragmentation_ratio:1.67
mem_allocator:libc
保存するデータの文字列の長さで予想されるデータ件数を見積もり、設定したメモリで耐えるか検証が別途必要である