[IBMCloud] IKS(Kubernetes) Cordon/Drain時の挙動確認

テスト環境

東京リージョンの各ZoneにVSI for VPCのLinuxサーバを配置

• VSI for VPC(CentOS) TOK01 x 1台
• VSI for VPC(CentOS) TOK02 x 1台
• VSI for VPC(CentOS) TOK03 x 1台

サーバ側はIKS(Kubernetes)にてNginxのPodを各ゾーンに配置し、前段にIngressALBを配置

• Application load balancer for VPC x 1(TOK02/TOK03)
• IBM Cloud Kubernetes Service x 1Cluster(TOK02/TOK03)
  • Nginx Pod(TOK02) x 1台
  • Nginx Pod(TOK03) x 1台
    

テスト内容

今回はWorkerNodeのcordonとPodのdrain実施時の通信状態について確認する

事前確認

PODの稼働状態、各WorkerNodeで1台づつNginxPodが動いていることを確認

% kubectl get pod -o wide
NAME                        READY   STATUS    RESTARTS   AGE   IP              NODE            NOMINATED NODE   READINESS GATES
iks-nginx-7b789b9b4-29bnv   1/1     Running   0          29s   172.17.4.111    10.244.64.28    <none>           <none>
iks-nginx-7b789b9b4-6rvcn   1/1     Running   0          29s   172.17.22.245   10.244.128.21   <none>           <none>

スケジューリングの停止(Cordon)

「10.244.128.21」のノードに対して新しいポッドのスケジューリングを停止する
作業手順はこちら

スケジュールが有効になっていることを確認

STATUSがReadyとなっていることを確認する
STATUSはSchedulingDisabledが表示されている場合、スケジューリング対象から外れている

% kubectl get nodes

NAME            STATUS   ROLES    AGE   VERSION
10.244.128.21   Ready    <none>   17h   v1.28.11+IKS
10.244.64.28    Ready    <none>   17h   v1.28.11+IKS

Cordonの実行

Cordonの実施

% kubectl cordon 10.244.128.21
node/10.244.128.21 cordoned

スケジュールが無効になっていることを確認

% kubectl get node
NAME            STATUS                     ROLES    AGE   VERSION
10.244.128.21   Ready,SchedulingDisabled   <none>   17h   v1.28.11+IKS
10.244.64.28    Ready                      <none>   17h   v1.28.11+IKS

Cordon時のPODの状態を確認
Cordon前と同様に、各ノードでそれぞれPODが１つづつ動いている状態

% kubectl get pod -o wide
NAME                        READY   STATUS    RESTARTS   AGE   IP              NODE            NOMINATED NODE   READINESS GATES
iks-nginx-7b789b9b4-29bnv   1/1     Running   0          29s   172.17.4.111    10.244.64.28    <none>           <none>
iks-nginx-7b789b9b4-6rvcn   1/1     Running   0          29s   172.17.22.245   10.244.128.21   <none>           <none>

VPCのサーバからPODへの通信の停止なし、POD分散も正常の状態

[root@tok01-sv ~]# while true; do curl --max-time 1 -s http://bf9b5d7d-jp-tok.lb.appdomain.cloud:8080 || echo "Request timed out or failed."; sleep 1; done
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:47:00
Hostname: iks-nginx-7b789b9b4-6rvcn Current Time: 2024-06-27 01:47:01
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:47:02
Hostname: iks-nginx-7b789b9b4-6rvcn Current Time: 2024-06-27 01:47:03

[root@tok02-sv ~]# while true; do curl --max-time 1 -s http://bf9b5d7d-jp-tok.lb.appdomain.cloud:8080 || echo "Request timed out or failed."; sleep 1; done
Hostname: iks-nginx-7b789b9b4-6rvcn Current Time: 2024-06-27 01:47:02
Hostname: iks-nginx-7b789b9b4-6rvcn Current Time: 2024-06-27 01:47:03
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:47:04
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:47:05

[root@tok03-sv ~]# while true; do curl --max-time 1 -s http://bf9b5d7d-jp-tok.lb.appdomain.cloud:8080 || echo "Request timed out or failed."; sleep 1; done
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:47:04
Hostname: iks-nginx-7b789b9b4-6rvcn Current Time: 2024-06-27 01:47:05
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:47:06
Hostname: iks-nginx-7b789b9b4-6rvcn Current Time: 2024-06-27 01:47:07

uncordonにする場合は下記の通り実施する

kubectl uncordon 10.244.128.21   

drainの実行

ノード上のすべてのPODは停止され、他のノードに再スケジュールされる
今回は、「10.244.128.21」にあるPOD「iks-nginx-7fbc7dfc94-6x4s9」が「10.244.64.28」で再スケジュールされる

kubectl drain を実行すると、ノード上のポッドは停止されるため、その間は一時的にポッドへの通信が中断される。ただし、ポッドが再スケジュールされ、新しいノードで稼働を開始すると、通信は再び確立される。

%  kubectl drain 10.244.128.21 --ignore-daemonsets --delete-emptydir-data
node/10.244.128.21 cordoned

Warning: ignoring DaemonSet-managed Pods: calico-system/calico-node-mm4jx, ibm-object-s3fs/ibmcloud-object-storage-driver-q4gbh, ibm-observe/sysdig-agent-node-analyzer-bmhpj, kube-system/ibm-vpc-block-csi-node-cx9fb, kube-system/ibmcloud-iks-debug-daemonset-769d2, kube-system/konnectivity-agent-g9zxm
evicting pod kube-system/coredns-jp-tok-3-6fb86b959c-rwhhc
evicting pod kube-system/ibm-vpc-block-csi-controller-0
evicting pod kube-system/private-crco9j451t0t7uths6kb10-alb2-5586d5d9ff-f88lw
evicting pod kube-system/coredns-jp-tok-3-6fb86b959c-2d6nt
evicting pod kube-system/coredns-jp-tok-3-6fb86b959c-j5mnv
evicting pod default/iks-nginx-7b789b9b4-6rvcn
evicting pod kube-system/coredns-autoscaler-jp-tok-3-5c77b4959c-z8jf2
evicting pod kube-system/public-crco9j451t0t7uths6kb10-alb2-78678bc946-kzp28
error when evicting pods/"private-crco9j451t0t7uths6kb10-alb2-5586d5d9ff-f88lw" -n "kube-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
error when evicting pods/"public-crco9j451t0t7uths6kb10-alb2-78678bc946-kzp28" -n "kube-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
pod/coredns-autoscaler-jp-tok-3-5c77b4959c-z8jf2 evicted
pod/ibm-vpc-block-csi-controller-0 evicted
evicting pod kube-system/private-crco9j451t0t7uths6kb10-alb2-5586d5d9ff-f88lw
evicting pod kube-system/public-crco9j451t0t7uths6kb10-alb2-78678bc946-kzp28
error when evicting pods/"private-crco9j451t0t7uths6kb10-alb2-5586d5d9ff-f88lw" -n "kube-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
error when evicting pods/"public-crco9j451t0t7uths6kb10-alb2-78678bc946-kzp28" -n "kube-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod kube-system/private-crco9j451t0t7uths6kb10-alb2-5586d5d9ff-f88lw
evicting pod kube-system/public-crco9j451t0t7uths6kb10-alb2-78678bc946-kzp28
error when evicting pods/"private-crco9j451t0t7uths6kb10-alb2-5586d5d9ff-f88lw" -n "kube-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
error when evicting pods/"public-crco9j451t0t7uths6kb10-alb2-78678bc946-kzp28" -n "kube-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
pod/coredns-jp-tok-3-6fb86b959c-2d6nt evicted
pod/coredns-jp-tok-3-6fb86b959c-j5mnv evicted
pod/coredns-jp-tok-3-6fb86b959c-rwhhc evicted

Drain時の通信状態

Drain開始時間

Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:58:25

PodがDrainされ、方系のPodのみとなり続けて新しいPodが作られたことがわかる

DrainされたPod:6rvcn
新しく作成されたPod:76wp7
方系で動き続けていたPod:29bnv

sv1 -> IngressALB -> Node -> Pod

Hostname: iks-nginx-7b789b9b4-6rvcn Current Time: 2024-06-27 01:58:54
Hostname: iks-nginx-7b789b9b4-6rvcn Current Time: 2024-06-27 01:58:55
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:58:56
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:58:57
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:58:58
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:58:59
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:59:00
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:59:01
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:59:02
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:59:03
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:59:04
Hostname: iks-nginx-7b789b9b4-76wp7 Current Time: 2024-06-27 01:59:05
Hostname: iks-nginx-7b789b9b4-76wp7 Current Time: 2024-06-27 01:59:06

sv2 -> IngressALB -> Node -> Pod

Hostname: iks-nginx-7b789b9b4-6rvcn Current Time: 2024-06-27 01:58:54
Hostname: iks-nginx-7b789b9b4-6rvcn Current Time: 2024-06-27 01:58:55
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:58:56
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:58:57
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:58:58
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:58:59
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:59:00
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:59:01
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:59:02
Hostname: iks-nginx-7b789b9b4-76wp7 Current Time: 2024-06-27 01:59:03
Hostname: iks-nginx-7b789b9b4-76wp7 Current Time: 2024-06-27 01:59:04
Hostname: iks-nginx-7b789b9b4-76wp7 Current Time: 2024-06-27 01:59:05
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:59:06

sv3 -> IngressALB -> Node -> Pod

Hostname: iks-nginx-7b789b9b4-6rvcn Current Time: 2024-06-27 01:58:54
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:58:56
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:58:57
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:58:58
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:58:59
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:59:00
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:59:01
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:59:02
Hostname: iks-nginx-7b789b9b4-76wp7 Current Time: 2024-06-27 01:59:03
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:59:04
Hostname: iks-nginx-7b789b9b4-29bnv Current Time: 2024-06-27 01:59:05

今回の検証ではPodへの通信の停止は見られなかった