はじめに
これはEKSのManaged Worker Nodeのアップデートの挙動を確認する記事です。
Managed Worker Node自体のドキュメントはこちらです
https://aws.amazon.com/jp/about-aws/whats-new/2019/11/amazon-eks-adds-support-for-provisioning-and-managing-kubernetes-worker-nodes/
https://docs.aws.amazon.com/ja_jp/eks/latest/userguide/managed-node-groups.html
前提条件
Managed Worker Node自体は現時点でAMIを指定して作成することはできません。
つまり、その次点での最新AMIが自動的に選択されるため、アップデートの挙動確認はすぐにはできません。
今回検証する環境は予め別の検証目的で作っていたものです。
たまたまAMIの更新が降ってきたので、アップデート検証記事を書くことにしました。
アップデート前の状態
EKSのGUI
こんな感じで表示されます
Updateの動作について
https://docs.aws.amazon.com/ja_jp/eks/latest/userguide/managed-node-update-behavior.html
ここに記載されている通りです。
簡単に書くと、
- AutoScalingの起動テンプレートを最新AMIにして更新し、EC2インスタンスの数を１つ増やす。
増やすことで、EC2インスタンスが１つ新規で起動する、かつ、それは新しいAMIで起動する。 -
eks.amazonaws.com/nodegroup-imagelabelを見て最新のAMIになっていない、Worker Node すべてを cordon して、Podがスケジュールされないようにする。 - EKSが、そのcordonされたNodeの中から１つ選び、drainを実行する(drainのtimeoutは15分、その15分後の挙動については、エラーを発生させるか、nodeを強制deleteさせるか選ぶことが可能) drainが正常に完了した場合は、nodeを削除する
- そうするとAutoScalingにより新規でNodeが起動する（かつ新しいAMIで）ので、上記の処理を繰り返して、すべてのNodeが更新されたら、EC2インスタンス数を１つ減らして元の起動テンプレートの状態に戻す
この動作を検証したいと思います。
現在の状態
k get node
NAME STATUS ROLES AGE VERSION
ip-10-111-0-241.eu-west-1.compute.internal Ready <none> 20d v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal Ready <none> 18d v1.14.7-eks-1861c5
こんな感じでnodeが２台あります。
ラベルはこんな感じです。
k describe node ip-10-111-0-241.eu-west-1.compute.internal
Name: ip-10-111-0-241.eu-west-1.compute.internal
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=t3.medium
beta.kubernetes.io/os=linux
eks.amazonaws.com/nodegroup=managed1
eks.amazonaws.com/nodegroup-image=ami-059c6874350e63ca9
failure-domain.beta.kubernetes.io/region=eu-west-1
failure-domain.beta.kubernetes.io/zone=eu-west-1a
kubernetes.io/arch=amd64
kubernetes.io/hostname=ip-10-111-0-241.eu-west-1.compute.internal
kubernetes.io/os=linux
Annotations: node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Mon, 17 Feb 2020 14:43:40 +0000
Taints: <none>
Unschedulable: false
eks.amazonaws.com/nodegroup-image=ami-059c6874350e63ca9 の部分でAMIが記載されていることが分かります。
Taintsもついていません
こんな感じでPodもdaemonsetも動かしています。
pod名やその役割には特に意味はないです。
k get all --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default pod/awscli-7bc86c4488-2qj2d 1/1 Running 0 85s
default pod/awscli-7bc86c4488-7xvcn 1/1 Running 0 85s
default pod/awscli-7bc86c4488-868ln 1/1 Running 0 85s
default pod/awscli-7bc86c4488-bd5bl 1/1 Running 0 17d
default pod/awscli-7bc86c4488-jh6v4 1/1 Running 0 85s
default pod/awscli-7bc86c4488-q2l7m 1/1 Running 0 85s
default pod/awscli-7bc86c4488-qnfg5 1/1 Running 0 85s
default pod/awscli-7bc86c4488-qtvcq 1/1 Running 0 85s
default pod/awscli-7bc86c4488-t5zkl 1/1 Running 0 85s
default pod/awscli-7bc86c4488-z9tps 1/1 Running 0 85s
default pod/awscli2-654c7776dd-zrmxh 1/1 Running 0 17d
kube-system pod/aws-node-w57dk 1/1 Running 0 18d
kube-system pod/aws-node-z8jpf 1/1 Running 0 20d
kube-system pod/coredns-9c7df9cf9-2gzdg 1/1 Running 0 25m
kube-system pod/coredns-9c7df9cf9-rmpn9 1/1 Running 0 25m
kube-system pod/kube-proxy-69mw4 1/1 Running 0 20d
kube-system pod/kube-proxy-96kb2 1/1 Running 0 18d
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 172.20.0.1 <none> 443/TCP 23d
kube-system service/kube-dns ClusterIP 172.20.0.10 <none> 53/UDP,53/TCP 23d
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system daemonset.apps/aws-node 2 2 2 2 2 <none> 23d
kube-system daemonset.apps/kube-proxy 2 2 2 2 2 <none> 23d
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
default deployment.apps/awscli 10/10 10 10 17d
default deployment.apps/awscli2 1/1 1 1 17d
kube-system deployment.apps/coredns 2/2 2 2 23d
NAMESPACE NAME DESIRED CURRENT READY AGE
default replicaset.apps/awscli-7bc86c4488 10 10 10 17d
default replicaset.apps/awscli2-654c7776dd 1 1 1 17d
kube-system replicaset.apps/coredns-9c7df9cf9 2 2 2 23d
確認したいこと
-
起動テンプレートの設定が代わること
-
新規で起動したEC2 nodeのラベルが代わること
-
古いEC2たちが、cordonされること
-
そのうちの一台がdrainされること
-
そしてそれが、terminaleされること
-
そしてそれが、繰り返されることと
-
アップデートが完了すること
アップデートの実行
GUIで実行。ポチッと
起動テンプレートのAMIが更新された
これは想定通り
Nodeの状態
だいぶ分かりにくいが
- 最初は2台 => Ready,SchedulingDisabledになった。つまり、新規NodeがReadyになることを待たずに、cordonが実行されていることが分かる
- そしてそのあと、なぜが5台以上のEC2インスタンスが起動されていることが分かる
k get node -w
NAME STATUS ROLES AGE VERSION
ip-10-111-0-241.eu-west-1.compute.internal Ready <none> 20d v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal Ready <none> 18d v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal Ready <none> 18d v1.14.7-eks-1861c5
ip-10-111-0-241.eu-west-1.compute.internal Ready <none> 20d v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal Ready <none> 18d v1.14.7-eks-1861c5
ip-10-111-0-241.eu-west-1.compute.internal Ready <none> 20d v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal Ready <none> 18d v1.14.7-eks-1861c5
ip-10-111-0-241.eu-west-1.compute.internal Ready <none> 20d v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal Ready <none> 18d v1.14.7-eks-1861c5
ip-10-111-0-241.eu-west-1.compute.internal Ready <none> 20d v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal Ready <none> 18d v1.14.7-eks-1861c5
ip-10-111-0-241.eu-west-1.compute.internal Ready <none> 20d v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal Ready <none> 18d v1.14.7-eks-1861c5
ip-10-111-0-241.eu-west-1.compute.internal Ready,SchedulingDisabled <none> 20d v1.14.7-eks-1861c5
ip-10-111-0-241.eu-west-1.compute.internal Ready,SchedulingDisabled <none> 20d v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal Ready,SchedulingDisabled <none> 18d v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal Ready,SchedulingDisabled <none> 18d v1.14.7-eks-1861c5
ip-10-111-0-241.eu-west-1.compute.internal Ready,SchedulingDisabled <none> 20d v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal Ready,SchedulingDisabled <none> 18d v1.14.7-eks-1861c5
ip-10-111-0-121.eu-west-1.compute.internal NotReady <none> 0s v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal NotReady <none> 0s v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal NotReady <none> 0s v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal NotReady <none> 0s v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal NotReady <none> 0s v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal NotReady <none> 0s v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal NotReady <none> 1s v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal NotReady <none> 1s v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal NotReady <none> 1s v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal NotReady <none> 0s v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal NotReady <none> 0s v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal NotReady <none> 0s v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal NotReady <none> 0s v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal NotReady <none> 0s v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal NotReady <none> 0s v1.14.8-eks-b8860f
ip-10-111-0-241.eu-west-1.compute.internal Ready,SchedulingDisabled <none> 20d v1.14.7-eks-1861c5
ip-10-111-1-42.eu-west-1.compute.internal NotReady <none> 2s v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal NotReady <none> 0s v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal NotReady <none> 4s v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal NotReady <none> 0s v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal NotReady <none> 0s v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal NotReady <none> 0s v1.14.8-eks-b8860f
ip-10-111-1-170.eu-west-1.compute.internal Ready,SchedulingDisabled <none> 18d v1.14.7-eks-1861c5
ip-10-111-0-121.eu-west-1.compute.internal Ready <none> 10s v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal Ready <none> 10s v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal Ready <none> 10s v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal Ready <none> 10s v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal Ready <none> 11s v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal Ready <none> 11s v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal Ready <none> 10s v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal Ready <none> 10s v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal Ready <none> 10s v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal Ready <none> 10s v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal Ready <none> 14s v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal Ready <none> 12s v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal Ready <none> 10s v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal NotReady <none> 9s v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal Ready <none> 10s v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal Ready <none> 10s v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal Ready <none> 14s v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal Ready <none> 30s v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal Ready <none> 30s v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal Ready <none> 31s v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal Ready <none> 30s v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal Ready <none> 30s v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal Ready <none> 30s v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal Ready <none> 60s v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal Ready <none> 60s v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal Ready <none> 61s v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal Ready <none> 60s v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal Ready <none> 60s v1.14.8-eks-b8860f
ip-10-111-0-241.eu-west-1.compute.internal Ready,SchedulingDisabled <none> 20d v1.14.7-eks-1861c5
ip-10-111-2-224.eu-west-1.compute.internal Ready <none> 60s v1.14.8-eks-b8860f
ip-10-111-1-170.eu-west-1.compute.internal Ready,SchedulingDisabled <none> 18d v1.14.7-eks-1861c5
ip-10-111-0-121.eu-west-1.compute.internal Ready <none> 2m v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal Ready <none> 2m v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal Ready <none> 2m1s v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal Ready <none> 2m v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal Ready <none> 2m v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal Ready <none> 2m v1.14.8-eks-b8860f
ip-10-111-1-170.eu-west-1.compute.internal Ready,SchedulingDisabled <none> 18d v1.14.7-eks-1861c5
ip-10-111-0-241.eu-west-1.compute.internal NotReady,SchedulingDisabled <none> 20d v1.14.7-eks-1861c5
ip-10-111-0-241.eu-west-1.compute.internal NotReady,SchedulingDisabled <none> 20d v1.14.7-eks-1861c5
ip-10-111-1-43.eu-west-1.compute.internal NotReady <none> 0s v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal NotReady <none> 0s v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal NotReady <none> 0s v1.14.8-eks-b8860f
ip-10-111-0-241.eu-west-1.compute.internal NotReady,SchedulingDisabled <none> 20d v1.14.7-eks-1861c5
ip-10-111-1-43.eu-west-1.compute.internal NotReady <none> 1s v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal Ready <none> 10s v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal Ready <none> 10s v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal Ready <none> 11s v1.14.8-eks-b8860f
ip-10-111-0-241.eu-west-1.compute.internal NotReady,SchedulingDisabled <none> 20d v1.14.7-eks-1861c5
ip-10-111-0-121.eu-west-1.compute.internal Ready <none> 3m v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal Ready <none> 3m v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal Ready <none> 3m1s v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal Ready <none> 3m v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal Ready <none> 3m v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal Ready <none> 3m v1.14.8-eks-b8860f
ip-10-111-1-170.eu-west-1.compute.internal Ready,SchedulingDisabled <none> 18d v1.14.7-eks-1861c5
ip-10-111-1-43.eu-west-1.compute.internal Ready <none> 30s v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal Ready <none> 3m30s v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal Ready <none> 60s v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal Ready <none> 4m v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal Ready <none> 4m1s v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal Ready <none> 4m v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal Ready <none> 4m v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal Ready <none> 4m v1.14.8-eks-b8860f
ip-10-111-1-170.eu-west-1.compute.internal Ready,SchedulingDisabled <none> 18d v1.14.7-eks-1861c5
ip-10-111-2-210.eu-west-1.compute.internal Ready <none> 4m30s v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal Ready <none> 2m v1.14.8-eks-b8860f
ip-10-111-1-170.eu-west-1.compute.internal NotReady,SchedulingDisabled <none> 18d v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal NotReady,SchedulingDisabled <none> 18d v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal NotReady,SchedulingDisabled <none> 18d v1.14.7-eks-1861c5
ip-10-111-0-121.eu-west-1.compute.internal Ready <none> 5m v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal Ready <none> 5m1s v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal Ready <none> 5m v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal NotReady <none> 0s v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal NotReady <none> 0s v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal NotReady <none> 0s v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal Ready <none> 5m v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal NotReady <none> 1s v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal Ready <none> 5m v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal Ready <none> 10s v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal Ready <none> 10s v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal Ready <none> 10s v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal Ready <none> 5m30s v1.14.8-eks-b8860f
ip-10-111-1-170.eu-west-1.compute.internal NotReady,SchedulingDisabled <none> 18d v1.14.7-eks-1861c5
ip-10-111-0-40.eu-west-1.compute.internal Ready <none> 30s v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal Ready <none> 3m v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal Ready <none> 6m v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal Ready <none> 6m1s v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal Ready <none> 6m v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal Ready,SchedulingDisabled <none> 5m59s v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal Ready,SchedulingDisabled <none> 5m59s v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal Ready,SchedulingDisabled <none> 6m4s v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal Ready,SchedulingDisabled <none> 6m4s v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal Ready,SchedulingDisabled <none> 6m v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal Ready <none> 6m v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal Ready,SchedulingDisabled <none> 6m8s v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal Ready,SchedulingDisabled <none> 6m8s v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal Ready,SchedulingDisabled <none> 6m11s v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal Ready,SchedulingDisabled <none> 6m11s v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal Ready,SchedulingDisabled <none> 6m11s v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal Ready,SchedulingDisabled <none> 6m11s v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal Ready,SchedulingDisabled <none> 6m12s v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal Ready,SchedulingDisabled <none> 6m12s v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal Ready,SchedulingDisabled <none> 6m30s v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal Ready <none> 90s v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal Ready <none> 4m v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal Ready,SchedulingDisabled <none> 7m v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal Ready,SchedulingDisabled <none> 7m1s v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal Ready,SchedulingDisabled <none> 7m v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal Ready,SchedulingDisabled <none> 7m1s v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal Ready,SchedulingDisabled <none> 7m v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal Ready <none> 2m30s v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal Ready <none> 5m v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal NotReady,SchedulingDisabled <none> 7m45s v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal NotReady,SchedulingDisabled <none> 7m45s v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal NotReady,SchedulingDisabled <none> 7m46s v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal NotReady,SchedulingDisabled <none> 7m46s v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal NotReady,SchedulingDisabled <none> 7m50s v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal NotReady,SchedulingDisabled <none> 7m51s v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal NotReady,SchedulingDisabled <none> 7m47s v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal NotReady,SchedulingDisabled <none> 7m55s v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal NotReady,SchedulingDisabled <none> 7m55s v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal NotReady,SchedulingDisabled <none> 7m55s v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal NotReady,SchedulingDisabled <none> 7m55s v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal NotReady,SchedulingDisabled <none> 7m53s v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal NotReady,SchedulingDisabled <none> 7m53s v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal NotReady,SchedulingDisabled <none> 7m56s v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal NotReady,SchedulingDisabled <none> 7m54s v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal NotReady,SchedulingDisabled <none> 8m v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal NotReady,SchedulingDisabled <none> 8m7s v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal NotReady,SchedulingDisabled <none> 8m5s v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal NotReady,SchedulingDisabled <none> 8m5s v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal NotReady,SchedulingDisabled <none> 8m7s v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal Ready <none> 3m30s v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal Ready <none> 6m v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal Ready <none> 4m30s v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal Ready <none> 7m1s v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal Ready <none> 5m30s v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal Ready <none> 8m1s v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal Ready <none> 6m30s v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal Ready <none> 9m1s v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal Ready <none> 7m30s v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal Ready <none> 10m v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal Ready <none> 8m31s v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal Ready <none> 11m v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal Ready <none> 9m31s v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal Ready <none> 12m v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal Ready <none> 10m v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal Ready <none> 13m v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal Ready <none> 11m v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal Ready <none> 14m v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal Ready <none> 12m v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal Ready <none> 15m v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal Ready <none> 13m v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal Ready <none> 16m v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal Ready <none> 14m v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal Ready <none> 17m v1.14.8-eks-b8860f
最終的に一時的に9台のEC2が起動する状態となり、最後に初期状態の2台に落ち着いた
8台のときに取得したスクショ。その後一台増えて、9台となった
daemonsetもこんな感じで推移
k get daemonset -n kube-system -w
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
aws-node 2 2 2 2 2 <none> 23d
kube-proxy 2 2 2 2 2 <none> 23d
kube-proxy 3 3 2 3 2 <none> 23d
aws-node 3 3 2 3 2 <none> 23d
aws-node 4 4 2 4 2 <none> 23d
kube-proxy 4 4 2 4 2 <none> 23d
kube-proxy 5 4 2 4 2 <none> 23d
aws-node 5 5 2 5 2 <none> 23d
kube-proxy 5 5 2 5 2 <none> 23d
kube-proxy 6 6 2 6 2 <none> 23d
aws-node 6 5 2 5 2 <none> 23d
aws-node 6 6 2 6 2 <none> 23d
aws-node 7 7 2 7 2 <none> 23d
kube-proxy 7 6 2 6 2 <none> 23d
kube-proxy 7 7 2 7 2 <none> 23d
aws-node 8 7 2 7 2 <none> 23d
kube-proxy 8 7 2 7 2 <none> 23d
aws-node 8 8 2 8 2 <none> 23d
kube-proxy 8 8 2 8 2 <none> 23d
kube-proxy 8 8 3 8 3 <none> 23d
kube-proxy 8 8 4 8 4 <none> 23d
kube-proxy 8 8 5 8 5 <none> 23d
aws-node 8 8 3 8 3 <none> 23d
aws-node 8 8 4 8 4 <none> 23d
aws-node 8 8 5 8 5 <none> 23d
kube-proxy 8 8 6 8 6 <none> 23d
kube-proxy 8 8 7 8 7 <none> 23d
aws-node 8 8 6 8 6 <none> 23d
kube-proxy 8 8 8 8 8 <none> 23d
aws-node 8 8 7 8 7 <none> 23d
aws-node 8 8 8 8 8 <none> 23d
aws-node 8 8 7 8 7 <none> 23d
kube-proxy 8 8 7 8 7 <none> 23d
kube-proxy 9 9 7 9 7 <none> 23d
aws-node 9 9 7 9 7 <none> 23d
kube-proxy 9 9 8 9 8 <none> 23d
aws-node 9 9 8 9 8 <none> 23d
aws-node 8 8 8 8 8 <none> 23d
kube-proxy 8 8 8 8 8 <none> 23d
aws-node 8 8 7 8 7 <none> 23d
kube-proxy 8 8 7 8 7 <none> 23d
aws-node 9 9 7 9 7 <none> 23d
kube-proxy 9 9 7 9 7 <none> 23d
kube-proxy 9 9 8 9 8 <none> 23d
aws-node 9 9 8 9 8 <none> 23d
aws-node 8 8 8 8 8 <none> 23d
kube-proxy 8 8 8 8 8 <none> 23d
aws-node 8 8 7 8 7 <none> 23d
kube-proxy 8 8 7 8 7 <none> 23d
aws-node 8 8 6 8 6 <none> 23d
kube-proxy 8 8 6 8 6 <none> 23d
aws-node 6 6 6 6 6 <none> 23d
kube-proxy 6 6 6 6 6 <none> 23d
aws-node 6 6 5 6 5 <none> 23d
kube-proxy 6 6 5 6 5 <none> 23d
aws-node 6 6 4 6 4 <none> 23d
kube-proxy 6 6 4 6 4 <none> 23d
aws-node 6 6 3 6 3 <none> 23d
kube-proxy 6 6 3 6 3 <none> 23d
aws-node 3 3 2 3 2 <none> 23d
kube-proxy 3 3 2 3 2 <none> 23d
kube-proxy 2 2 2 2 2 <none> 23d
aws-node 2 2 2 2 2 <none> 23d
podもこんな感じで推移
k get pod -w
NAME READY STATUS RESTARTS AGE
awscli-7bc86c4488-2qj2d 1/1 Running 0 3m15s
awscli-7bc86c4488-7xvcn 1/1 Running 0 3m15s
awscli-7bc86c4488-868ln 1/1 Running 0 3m15s
awscli-7bc86c4488-bd5bl 1/1 Running 0 17d
awscli-7bc86c4488-jh6v4 1/1 Running 0 3m15s
awscli-7bc86c4488-q2l7m 1/1 Running 0 3m15s
awscli-7bc86c4488-qnfg5 1/1 Running 0 3m15s
awscli-7bc86c4488-qtvcq 1/1 Running 0 3m15s
awscli-7bc86c4488-t5zkl 1/1 Running 0 3m15s
awscli-7bc86c4488-z9tps 1/1 Running 0 3m15s
awscli2-654c7776dd-zrmxh 1/1 Running 0 17d
awscli-7bc86c4488-2qj2d 1/1 Terminating 0 12m
awscli-7bc86c4488-tv2f6 0/1 Pending 0 0s
awscli-7bc86c4488-tv2f6 0/1 Pending 0 0s
awscli-7bc86c4488-tv2f6 0/1 ContainerCreating 0 1s
awscli-7bc86c4488-7xvcn 1/1 Terminating 0 12m
awscli-7bc86c4488-nss6s 0/1 Pending 0 0s
awscli-7bc86c4488-nss6s 0/1 Pending 0 0s
awscli-7bc86c4488-nss6s 0/1 ContainerCreating 0 0s
awscli-7bc86c4488-868ln 1/1 Terminating 0 12m
awscli-7bc86c4488-5vxdk 0/1 Pending 0 0s
awscli-7bc86c4488-5vxdk 0/1 Pending 0 0s
awscli-7bc86c4488-5vxdk 0/1 ContainerCreating 0 0s
awscli-7bc86c4488-bd5bl 1/1 Terminating 0 17d
awscli-7bc86c4488-wb259 0/1 Pending 0 0s
awscli-7bc86c4488-wb259 0/1 Pending 0 0s
awscli-7bc86c4488-wb259 0/1 ContainerCreating 0 0s
awscli-7bc86c4488-qtvcq 1/1 Terminating 0 12m
awscli-7bc86c4488-dwtk8 0/1 Pending 0 0s
awscli-7bc86c4488-dwtk8 0/1 Pending 0 0s
awscli-7bc86c4488-dwtk8 0/1 ContainerCreating 0 0s
awscli-7bc86c4488-wb259 1/1 Running 0 10s
awscli-7bc86c4488-5vxdk 1/1 Running 0 11s
awscli-7bc86c4488-tv2f6 1/1 Running 0 12s
awscli-7bc86c4488-nss6s 1/1 Running 0 11s
awscli-7bc86c4488-dwtk8 1/1 Running 0 11s
awscli-7bc86c4488-2qj2d 0/1 Terminating 0 12m
awscli-7bc86c4488-7xvcn 0/1 Terminating 0 12m
awscli-7bc86c4488-bd5bl 0/1 Terminating 0 17d
awscli-7bc86c4488-868ln 0/1 Terminating 0 12m
awscli-7bc86c4488-qtvcq 0/1 Terminating 0 12m
awscli-7bc86c4488-qtvcq 0/1 Terminating 0 12m
awscli-7bc86c4488-7xvcn 0/1 Terminating 0 12m
awscli-7bc86c4488-7xvcn 0/1 Terminating 0 12m
awscli-7bc86c4488-2qj2d 0/1 Terminating 0 12m
awscli-7bc86c4488-2qj2d 0/1 Terminating 0 12m
awscli-7bc86c4488-qtvcq 0/1 Terminating 0 12m
awscli-7bc86c4488-qtvcq 0/1 Terminating 0 12m
awscli-7bc86c4488-bd5bl 0/1 Terminating 0 17d
awscli-7bc86c4488-bd5bl 0/1 Terminating 0 17d
awscli-7bc86c4488-868ln 0/1 Terminating 0 12m
awscli-7bc86c4488-868ln 0/1 Terminating 0 12m
awscli-7bc86c4488-jh6v4 1/1 Terminating 0 14m
awscli-7bc86c4488-dd679 0/1 Pending 0 0s
awscli-7bc86c4488-dd679 0/1 Pending 0 0s
awscli-7bc86c4488-dd679 0/1 ContainerCreating 0 0s
awscli-7bc86c4488-q2l7m 1/1 Terminating 0 14m
awscli-7bc86c4488-5jvch 0/1 Pending 0 0s
awscli-7bc86c4488-5jvch 0/1 Pending 0 0s
awscli-7bc86c4488-5jvch 0/1 ContainerCreating 0 0s
awscli-7bc86c4488-qnfg5 1/1 Terminating 0 14m
awscli-7bc86c4488-5knbq 0/1 Pending 0 0s
awscli-7bc86c4488-5knbq 0/1 Pending 0 0s
awscli-7bc86c4488-5knbq 0/1 ContainerCreating 0 0s
awscli-7bc86c4488-t5zkl 1/1 Terminating 0 14m
awscli-7bc86c4488-zfhns 0/1 Pending 0 0s
awscli-7bc86c4488-zfhns 0/1 Pending 0 0s
awscli-7bc86c4488-zfhns 0/1 ContainerCreating 0 0s
awscli-7bc86c4488-z9tps 1/1 Terminating 0 14m
awscli-7bc86c4488-jffnr 0/1 Pending 0 0s
awscli-7bc86c4488-jffnr 0/1 Pending 0 0s
awscli-7bc86c4488-jffnr 0/1 ContainerCreating 0 0s
awscli2-654c7776dd-zrmxh 1/1 Terminating 0 17d
awscli2-654c7776dd-8p7mx 0/1 Pending 0 0s
awscli2-654c7776dd-8p7mx 0/1 Pending 0 0s
awscli2-654c7776dd-8p7mx 0/1 ContainerCreating 0 0s
awscli-7bc86c4488-zfhns 1/1 Running 0 1s
awscli-7bc86c4488-5knbq 1/1 Running 0 2s
awscli-7bc86c4488-jffnr 1/1 Running 0 2s
awscli2-654c7776dd-8p7mx 1/1 Running 0 2s
awscli-7bc86c4488-dd679 1/1 Running 0 10s
awscli-7bc86c4488-5jvch 1/1 Running 0 10s
awscli-7bc86c4488-jh6v4 0/1 Terminating 0 15m
awscli-7bc86c4488-qnfg5 0/1 Terminating 0 15m
awscli-7bc86c4488-q2l7m 0/1 Terminating 0 15m
awscli-7bc86c4488-jh6v4 0/1 Terminating 0 15m
awscli-7bc86c4488-jh6v4 0/1 Terminating 0 15m
awscli-7bc86c4488-z9tps 0/1 Terminating 0 15m
awscli-7bc86c4488-t5zkl 0/1 Terminating 0 15m
awscli2-654c7776dd-zrmxh 0/1 Terminating 0 17d
awscli-7bc86c4488-t5zkl 0/1 Terminating 0 15m
awscli-7bc86c4488-t5zkl 0/1 Terminating 0 15m
awscli-7bc86c4488-q2l7m 0/1 Terminating 0 15m
awscli-7bc86c4488-q2l7m 0/1 Terminating 0 15m
awscli2-654c7776dd-zrmxh 0/1 Terminating 0 17d
awscli2-654c7776dd-zrmxh 0/1 Terminating 0 17d
awscli-7bc86c4488-z9tps 0/1 Terminating 0 15m
awscli-7bc86c4488-z9tps 0/1 Terminating 0 15m
awscli-7bc86c4488-qnfg5 0/1 Terminating 0 15m
awscli-7bc86c4488-qnfg5 0/1 Terminating 0 15m
awscli-7bc86c4488-dwtk8 1/1 Terminating 0 5m23s
awscli-7bc86c4488-pd5c9 0/1 Pending 0 0s
awscli-7bc86c4488-pd5c9 0/1 Pending 0 0s
awscli-7bc86c4488-pd5c9 0/1 ContainerCreating 0 0s
awscli-7bc86c4488-5knbq 1/1 Terminating 0 2m57s
awscli-7bc86c4488-ddx8r 0/1 Pending 0 0s
awscli-7bc86c4488-ddx8r 0/1 Pending 0 0s
awscli2-654c7776dd-8p7mx 1/1 Terminating 0 2m55s
awscli-7bc86c4488-ddx8r 0/1 ContainerCreating 0 0s
awscli2-654c7776dd-rh59c 0/1 Pending 0 0s
awscli2-654c7776dd-rh59c 0/1 Pending 0 0s
awscli2-654c7776dd-rh59c 0/1 ContainerCreating 0 0s
awscli-7bc86c4488-nss6s 1/1 Terminating 0 5m25s
awscli-7bc86c4488-n2thk 0/1 Pending 0 0s
awscli-7bc86c4488-n2thk 0/1 Pending 0 0s
awscli-7bc86c4488-n2thk 0/1 ContainerCreating 0 0s
awscli-7bc86c4488-pd5c9 1/1 Running 0 2s
awscli2-654c7776dd-rh59c 1/1 Running 0 2s
awscli-7bc86c4488-n2thk 1/1 Running 0 2s
awscli-7bc86c4488-jffnr 1/1 Terminating 0 3m3s
awscli-7bc86c4488-9x9ws 0/1 Pending 0 1s
awscli-7bc86c4488-9x9ws 0/1 Pending 0 1s
awscli-7bc86c4488-9x9ws 0/1 ContainerCreating 0 1s
awscli-7bc86c4488-wb259 1/1 Terminating 0 5m31s
awscli-7bc86c4488-5tthr 0/1 Pending 0 0s
awscli-7bc86c4488-5tthr 0/1 Pending 0 0s
awscli-7bc86c4488-5tthr 0/1 ContainerCreating 0 0s
awscli-7bc86c4488-5vxdk 1/1 Terminating 0 5m32s
awscli-7bc86c4488-5vrhj 0/1 Pending 0 0s
awscli-7bc86c4488-5vrhj 0/1 Pending 0 0s
awscli-7bc86c4488-5vrhj 0/1 ContainerCreating 0 0s
awscli-7bc86c4488-5jvch 1/1 Terminating 0 3m4s
awscli-7bc86c4488-md55k 0/1 Pending 0 0s
awscli-7bc86c4488-md55k 0/1 Pending 0 0s
awscli-7bc86c4488-md55k 0/1 ContainerCreating 0 0s
awscli-7bc86c4488-zfhns 1/1 Terminating 0 3m3s
awscli-7bc86c4488-2plnt 0/1 Pending 0 0s
awscli-7bc86c4488-2plnt 0/1 Pending 0 0s
awscli-7bc86c4488-2plnt 0/1 ContainerCreating 0 0s
awscli-7bc86c4488-pd5c9 1/1 Terminating 0 8s
awscli-7bc86c4488-8mbgr 0/1 Pending 0 0s
awscli-7bc86c4488-8mbgr 0/1 Pending 0 0s
awscli-7bc86c4488-8mbgr 0/1 ContainerCreating 0 0s
awscli-7bc86c4488-9x9ws 1/1 Running 0 2s
awscli-7bc86c4488-5vrhj 1/1 Running 0 1s
awscli-7bc86c4488-md55k 1/1 Running 0 2s
awscli-7bc86c4488-8mbgr 1/1 Running 0 2s
awscli-7bc86c4488-ddx8r 1/1 Running 0 10s
awscli-7bc86c4488-2plnt 1/1 Running 0 4s
awscli-7bc86c4488-5tthr 1/1 Running 0 4s
awscli-7bc86c4488-5vrhj 1/1 Terminating 0 7s
awscli-7bc86c4488-cldgf 0/1 Pending 0 0s
awscli-7bc86c4488-cldgf 0/1 Pending 0 0s
awscli-7bc86c4488-cldgf 0/1 ContainerCreating 0 0s
awscli-7bc86c4488-8mbgr 1/1 Terminating 0 7s
awscli-7bc86c4488-cx62z 0/1 Pending 0 0s
awscli-7bc86c4488-cx62z 0/1 Pending 0 0s
awscli-7bc86c4488-cx62z 0/1 ContainerCreating 0 0s
awscli-7bc86c4488-n2thk 1/1 Terminating 0 15s
awscli-7bc86c4488-c4zdj 0/1 Pending 0 0s
awscli-7bc86c4488-c4zdj 0/1 Pending 0 0s
awscli-7bc86c4488-c4zdj 0/1 ContainerCreating 0 0s
awscli-7bc86c4488-tv2f6 1/1 Terminating 0 5m41s
awscli-7bc86c4488-cm54v 0/1 Pending 0 0s
awscli-7bc86c4488-cm54v 0/1 Pending 0 0s
awscli-7bc86c4488-cm54v 0/1 ContainerCreating 0 0s
awscli-7bc86c4488-cldgf 1/1 Running 0 2s
awscli-7bc86c4488-cx62z 1/1 Running 0 2s
awscli-7bc86c4488-cm54v 1/1 Running 0 1s
awscli-7bc86c4488-c4zdj 1/1 Running 0 2s
awscli-7bc86c4488-5knbq 0/1 Terminating 0 3m27s
awscli-7bc86c4488-nss6s 0/1 Terminating 0 5m55s
awscli-7bc86c4488-dwtk8 0/1 Terminating 0 5m55s
awscli2-654c7776dd-8p7mx 0/1 Terminating 0 3m26s
awscli2-654c7776dd-8p7mx 0/1 Terminating 0 3m26s
awscli-7bc86c4488-nss6s 0/1 Terminating 0 5m59s
awscli-7bc86c4488-nss6s 0/1 Terminating 0 5m59s
awscli-7bc86c4488-5knbq 0/1 Terminating 0 3m31s
awscli-7bc86c4488-5knbq 0/1 Terminating 0 3m31s
awscli-7bc86c4488-jffnr 0/1 Terminating 0 3m33s
awscli-7bc86c4488-wb259 0/1 Terminating 0 6m2s
awscli-7bc86c4488-5vxdk 0/1 Terminating 0 6m3s
awscli-7bc86c4488-zfhns 0/1 Terminating 0 3m34s
awscli-7bc86c4488-5jvch 0/1 Terminating 0 3m35s
awscli-7bc86c4488-pd5c9 0/1 Terminating 0 39s
awscli-7bc86c4488-dwtk8 0/1 Terminating 0 6m2s
awscli-7bc86c4488-dwtk8 0/1 Terminating 0 6m3s
awscli2-654c7776dd-8p7mx 0/1 Terminating 0 3m34s
awscli2-654c7776dd-8p7mx 0/1 Terminating 0 3m34s
awscli-7bc86c4488-5jvch 0/1 Terminating 0 3m36s
awscli-7bc86c4488-5jvch 0/1 Terminating 0 3m36s
awscli-7bc86c4488-pd5c9 0/1 Terminating 0 40s
awscli-7bc86c4488-pd5c9 0/1 Terminating 0 41s
awscli-7bc86c4488-5vrhj 0/1 Terminating 0 37s
awscli-7bc86c4488-wb259 0/1 Terminating 0 6m9s
awscli-7bc86c4488-wb259 0/1 Terminating 0 6m9s
awscli-7bc86c4488-jffnr 0/1 Terminating 0 3m41s
awscli-7bc86c4488-jffnr 0/1 Terminating 0 3m41s
awscli-7bc86c4488-tv2f6 0/1 Terminating 0 6m11s
awscli-7bc86c4488-n2thk 0/1 Terminating 0 46s
awscli-7bc86c4488-8mbgr 0/1 Terminating 0 39s
awscli-7bc86c4488-5vrhj 0/1 Terminating 0 39s
awscli-7bc86c4488-5vrhj 0/1 Terminating 0 39s
awscli-7bc86c4488-5vxdk 0/1 Terminating 0 6m11s
awscli-7bc86c4488-5vxdk 0/1 Terminating 0 6m11s
awscli-7bc86c4488-zfhns 0/1 Terminating 0 3m42s
awscli-7bc86c4488-zfhns 0/1 Terminating 0 3m42s
awscli-7bc86c4488-tv2f6 0/1 Terminating 0 6m15s
awscli-7bc86c4488-tv2f6 0/1 Terminating 0 6m15s
awscli-7bc86c4488-n2thk 0/1 Terminating 0 49s
awscli-7bc86c4488-n2thk 0/1 Terminating 0 49s
awscli-7bc86c4488-8mbgr 0/1 Terminating 0 42s
awscli-7bc86c4488-8mbgr 0/1 Terminating 0 42s
古いNodeは全部Cordonが実行されて、こんな状態になった
k describe node ip-10-111-1-170.eu-west-1.compute.internal
Name: ip-10-111-1-170.eu-west-1.compute.internal
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=t3.medium
beta.kubernetes.io/os=linux
eks.amazonaws.com/nodegroup=managed1
eks.amazonaws.com/nodegroup-image=ami-059c6874350e63ca9
failure-domain.beta.kubernetes.io/region=eu-west-1
failure-domain.beta.kubernetes.io/zone=eu-west-1b
kubernetes.io/arch=amd64
kubernetes.io/hostname=ip-10-111-1-170.eu-west-1.compute.internal
kubernetes.io/os=linux
Annotations: node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Wed, 19 Feb 2020 05:25:17 +0000
Taints: node.kubernetes.io/unschedulable:NoSchedule
Unschedulable: true
新しいNodeは tainsはなく、nodegroup-imageが更新されていることが分かる
k describe node ip-10-111-1-43.eu-west-1.compute.internal
Name: ip-10-111-1-43.eu-west-1.compute.internal
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=t3.medium
beta.kubernetes.io/os=linux
eks.amazonaws.com/nodegroup=managed1
eks.amazonaws.com/nodegroup-image=ami-0b9d2c11b47bd8264
failure-domain.beta.kubernetes.io/region=eu-west-1
failure-domain.beta.kubernetes.io/zone=eu-west-1b
kubernetes.io/arch=amd64
kubernetes.io/hostname=ip-10-111-1-43.eu-west-1.compute.internal
kubernetes.io/os=linux
Annotations: node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Sun, 08 Mar 2020 16:19:10 +0000
Taints: <none>
Unschedulable: false
まとめ
最後はすごい散らかってしまいましたが、
- AutoScalingの起動テンプレートを最新AMIにして更新し、EC2インスタンスの数を１つ増やす。
増やすことで、EC2インスタンスが１つ新規で起動する、かつ、それは新しいAMIで起動する - １つ増えるのではなく、一気に増えるようです。今回の検証では古いNodeは２つで、Max9Nodeになったので、7つ増えた？この原因は、ドキュメントの理解が足りない可能性があるので時間があるときにもう一度検証してみようと思います。
-
eks.amazonaws.com/nodegroup-imagelabelを見て最新のAMIになっていない、Worker Node すべてを cordon して、Podがスケジュールされないようにする。 - これはそのとおりでした
- EKSが、そのcordonされたNodeの中から１つ選び、drainを実行する(drainのtimeoutは15分、その15分後の挙動については、エラーを発生させるか、nodeを強制deleteさせるか選ぶことが可能) drainが正常に完了した場合は、nodeを削除する
- 異常系は試していないのですが、drainされたnodeが削除されたのは事実です。
- そうするとAutoScalingにより新規でNodeが起動する（かつ新しいAMIで）ので、上記の処理を繰り返して、すべてのNodeが更新されたら、EC2インスタンス数を１つ減らして元の起動テンプレートの状態に戻す
- １つずつ減らしたかどうかは見ていませんが、もとのEC2インスタンス数に戻ったのは確かです。
まとめ
damonsetがあっても問題なくdrainしてくれました。本番環境だと、PDBが必須だと思うので、これも時間があるときに（次回のアップデートが降ってきたときに）PDBと組み合わせて検証してみようと思います。