はじめに

これはEKSのManaged Worker Nodeのアップデートの挙動を確認する記事です。

Managed Worker Node自体のドキュメントはこちらです
https://aws.amazon.com/jp/about-aws/whats-new/2019/11/amazon-eks-adds-support-for-provisioning-and-managing-kubernetes-worker-nodes/
https://docs.aws.amazon.com/ja_jp/eks/latest/userguide/managed-node-groups.html

前提条件

Managed Worker Node自体は現時点でAMIを指定して作成することはできません。
つまり、その次点での最新AMIが自動的に選択されるため、アップデートの挙動確認はすぐにはできません。

今回検証する環境は予め別の検証目的で作っていたものです。
たまたまAMIの更新が降ってきたので、アップデート検証記事を書くことにしました。

アップデート前の状態

EKSのGUI

こんな感じで表示されます

Updateの動作について

https://docs.aws.amazon.com/ja_jp/eks/latest/userguide/managed-node-update-behavior.html
ここに記載されている通りです。

簡単に書くと、

AutoScalingの起動テンプレートを最新AMIにして更新し、EC2インスタンスの数を１つ増やす。
増やすことで、EC2インスタンスが１つ新規で起動する、かつ、それは新しいAMIで起動する。
eks.amazonaws.com/nodegroup-image labelを見て最新のAMIになっていない、Worker Node すべてを　cordon して、Podがスケジュールされないようにする。
EKSが、そのcordonされたNodeの中から１つ選び、drainを実行する(drainのtimeoutは15分、その15分後の挙動については、エラーを発生させるか、nodeを強制deleteさせるか選ぶことが可能) drainが正常に完了した場合は、nodeを削除する
そうするとAutoScalingにより新規でNodeが起動する（かつ新しいAMIで）ので、上記の処理を繰り返して、すべてのNodeが更新されたら、EC2インスタンス数を１つ減らして元の起動テンプレートの状態に戻す

この動作を検証したいと思います。

現在の状態

k get node
NAME                                         STATUS   ROLES    AGE   VERSION
ip-10-111-0-241.eu-west-1.compute.internal   Ready    <none>   20d   v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal   Ready    <none>   18d   v1.14.7-eks-1861c5

こんな感じでnodeが２台あります。
ラベルはこんな感じです。

k describe node ip-10-111-0-241.eu-west-1.compute.internal  
Name:               ip-10-111-0-241.eu-west-1.compute.internal
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=t3.medium
                    beta.kubernetes.io/os=linux
                    eks.amazonaws.com/nodegroup=managed1
                    eks.amazonaws.com/nodegroup-image=ami-059c6874350e63ca9
                    failure-domain.beta.kubernetes.io/region=eu-west-1
                    failure-domain.beta.kubernetes.io/zone=eu-west-1a
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=ip-10-111-0-241.eu-west-1.compute.internal
                    kubernetes.io/os=linux
Annotations:        node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Mon, 17 Feb 2020 14:43:40 +0000
Taints:             <none>
Unschedulable:      false

eks.amazonaws.com/nodegroup-image=ami-059c6874350e63ca9 の部分でAMIが記載されていることが分かります。
Taintsもついていません

こんな感じでPodもdaemonsetも動かしています。
pod名やその役割には特に意味はないです。

k get all --all-namespaces
NAMESPACE     NAME                           READY   STATUS    RESTARTS   AGE
default       pod/awscli-7bc86c4488-2qj2d    1/1     Running   0          85s
default       pod/awscli-7bc86c4488-7xvcn    1/1     Running   0          85s
default       pod/awscli-7bc86c4488-868ln    1/1     Running   0          85s
default       pod/awscli-7bc86c4488-bd5bl    1/1     Running   0          17d
default       pod/awscli-7bc86c4488-jh6v4    1/1     Running   0          85s
default       pod/awscli-7bc86c4488-q2l7m    1/1     Running   0          85s
default       pod/awscli-7bc86c4488-qnfg5    1/1     Running   0          85s
default       pod/awscli-7bc86c4488-qtvcq    1/1     Running   0          85s
default       pod/awscli-7bc86c4488-t5zkl    1/1     Running   0          85s
default       pod/awscli-7bc86c4488-z9tps    1/1     Running   0          85s
default       pod/awscli2-654c7776dd-zrmxh   1/1     Running   0          17d
kube-system   pod/aws-node-w57dk             1/1     Running   0          18d
kube-system   pod/aws-node-z8jpf             1/1     Running   0          20d
kube-system   pod/coredns-9c7df9cf9-2gzdg    1/1     Running   0          25m
kube-system   pod/coredns-9c7df9cf9-rmpn9    1/1     Running   0          25m
kube-system   pod/kube-proxy-69mw4           1/1     Running   0          20d
kube-system   pod/kube-proxy-96kb2           1/1     Running   0          18d

NAMESPACE     NAME                 TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)         AGE
default       service/kubernetes   ClusterIP   172.20.0.1    <none>        443/TCP         23d
kube-system   service/kube-dns     ClusterIP   172.20.0.10   <none>        53/UDP,53/TCP   23d

NAMESPACE     NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
kube-system   daemonset.apps/aws-node     2         2         2       2            2           <none>          23d
kube-system   daemonset.apps/kube-proxy   2         2         2       2            2           <none>          23d

NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
default       deployment.apps/awscli    10/10   10           10          17d
default       deployment.apps/awscli2   1/1     1            1           17d
kube-system   deployment.apps/coredns   2/2     2            2           23d

NAMESPACE     NAME                                 DESIRED   CURRENT   READY   AGE
default       replicaset.apps/awscli-7bc86c4488    10        10        10      17d
default       replicaset.apps/awscli2-654c7776dd   1         1         1       17d
kube-system   replicaset.apps/coredns-9c7df9cf9    2         2         2       23d

確認したいこと

起動テンプレートの設定が代わること
新規で起動したEC2 nodeのラベルが代わること
古いEC2たちが、cordonされること
そのうちの一台がdrainされること
そしてそれが、terminaleされること
そしてそれが、繰り返されることと
アップデートが完了すること

アップデートの実行

GUIで実行。ポチッと

起動テンプレートのAMIが更新された

これは想定通り

Nodeの状態

だいぶ分かりにくいが

最初は2台 => Ready,SchedulingDisabledになった。つまり、新規NodeがReadyになることを待たずに、cordonが実行されていることが分かる
そしてそのあと、なぜが5台以上のEC2インスタンスが起動されていることが分かる

k get node -w
NAME                                         STATUS   ROLES    AGE   VERSION
ip-10-111-0-241.eu-west-1.compute.internal   Ready    <none>   20d   v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal   Ready    <none>   18d   v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal   Ready   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-0-241.eu-west-1.compute.internal   Ready   <none>   20d   v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal   Ready   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-0-241.eu-west-1.compute.internal   Ready   <none>   20d   v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal   Ready   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-0-241.eu-west-1.compute.internal   Ready   <none>   20d   v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal   Ready   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-0-241.eu-west-1.compute.internal   Ready   <none>   20d   v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal   Ready   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-0-241.eu-west-1.compute.internal   Ready   <none>   20d   v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal   Ready   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-0-241.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   20d   v1.14.7-eks-1861c5
ip-10-111-0-241.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   20d   v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-0-241.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   20d   v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-0-121.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   NotReady   <none>   1s    v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   NotReady   <none>   1s    v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   NotReady   <none>   1s    v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-0-241.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   20d   v1.14.7-eks-1861c5
ip-10-111-1-42.eu-west-1.compute.internal   NotReady   <none>   2s    v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal   NotReady   <none>   4s    v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-1-170.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-0-121.eu-west-1.compute.internal   Ready   <none>   10s   v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal   Ready   <none>   10s   v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   Ready   <none>   10s   v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   Ready   <none>   10s   v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   Ready   <none>   11s   v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   Ready   <none>   11s   v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   Ready   <none>   10s   v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   Ready   <none>   10s   v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   Ready   <none>   10s   v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   Ready   <none>   10s   v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal   Ready   <none>   14s   v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   Ready   <none>   12s   v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   Ready   <none>   10s   v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   NotReady   <none>   9s    v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   Ready   <none>   10s   v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   Ready   <none>   10s   v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   Ready   <none>   14s   v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal   Ready   <none>   30s   v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   Ready   <none>   30s   v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   Ready   <none>   31s   v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   Ready   <none>   30s   v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   Ready   <none>   30s   v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   Ready   <none>   30s   v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal   Ready   <none>   60s   v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   Ready   <none>   60s   v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   Ready   <none>   61s   v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   Ready   <none>   60s   v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   Ready   <none>   60s   v1.14.8-eks-b8860f
ip-10-111-0-241.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   20d   v1.14.7-eks-1861c5
ip-10-111-2-224.eu-west-1.compute.internal   Ready   <none>   60s   v1.14.8-eks-b8860f
ip-10-111-1-170.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-0-121.eu-west-1.compute.internal   Ready   <none>   2m    v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   Ready   <none>   2m    v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   Ready   <none>   2m1s   v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   Ready   <none>   2m    v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   Ready   <none>   2m    v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   Ready   <none>   2m    v1.14.8-eks-b8860f
ip-10-111-1-170.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-0-241.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   20d   v1.14.7-eks-1861c5
ip-10-111-0-241.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   20d   v1.14.7-eks-1861c5
ip-10-111-1-43.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-0-241.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   20d   v1.14.7-eks-1861c5
ip-10-111-1-43.eu-west-1.compute.internal   NotReady   <none>   1s    v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   10s   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   10s   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   11s   v1.14.8-eks-b8860f
ip-10-111-0-241.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   20d   v1.14.7-eks-1861c5
ip-10-111-0-121.eu-west-1.compute.internal   Ready   <none>   3m    v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   Ready   <none>   3m    v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   Ready   <none>   3m1s   v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   Ready   <none>   3m    v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   Ready   <none>   3m    v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   Ready   <none>   3m    v1.14.8-eks-b8860f
ip-10-111-1-170.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   30s   v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   Ready   <none>   3m30s   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   60s   v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal   Ready   <none>   4m    v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   Ready   <none>   4m1s   v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   Ready   <none>   4m    v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   Ready   <none>   4m    v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   Ready   <none>   4m    v1.14.8-eks-b8860f
ip-10-111-1-170.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-2-210.eu-west-1.compute.internal   Ready   <none>   4m30s   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   2m    v1.14.8-eks-b8860f
ip-10-111-1-170.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-1-170.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-0-121.eu-west-1.compute.internal   Ready   <none>   5m    v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   Ready   <none>   5m1s   v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   Ready   <none>   5m    v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   NotReady   <none>   0s    v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   Ready   <none>   5m    v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   NotReady   <none>   1s    v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   Ready   <none>   5m    v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   10s   v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   10s   v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   10s   v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   Ready   <none>   5m30s   v1.14.8-eks-b8860f
ip-10-111-1-170.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   18d   v1.14.7-eks-1861c5
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   30s   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   3m    v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal   Ready   <none>   6m    v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   Ready   <none>   6m1s   v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   Ready   <none>   6m    v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   5m59s   v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   5m59s   v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   6m4s   v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   6m4s   v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   6m    v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   Ready   <none>   6m    v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   6m8s   v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   6m8s   v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   6m11s   v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   6m11s   v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   6m11s   v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   6m11s   v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   6m12s   v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   6m12s   v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   6m30s   v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   90s   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   4m    v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   7m    v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   7m1s   v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   7m    v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   7m1s   v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   Ready,SchedulingDisabled   <none>   7m    v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   2m30s   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   5m    v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   7m45s   v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   7m45s   v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   7m46s   v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   7m46s   v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   7m50s   v1.14.8-eks-b8860f
ip-10-111-0-121.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   7m51s   v1.14.8-eks-b8860f
ip-10-111-2-148.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   7m47s   v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   7m55s   v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   7m55s   v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   7m55s   v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   7m55s   v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   7m53s   v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   7m53s   v1.14.8-eks-b8860f
ip-10-111-0-103.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   7m56s   v1.14.8-eks-b8860f
ip-10-111-1-42.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   7m54s   v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   8m    v1.14.8-eks-b8860f
ip-10-111-2-210.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   8m7s   v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   8m5s   v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   8m5s   v1.14.8-eks-b8860f
ip-10-111-2-224.eu-west-1.compute.internal   NotReady,SchedulingDisabled   <none>   8m7s   v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   3m30s   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   6m    v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   4m30s   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   7m1s   v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   5m30s   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   8m1s   v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   6m30s   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   9m1s   v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   7m30s   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   10m   v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   8m31s   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   11m   v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   9m31s   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   12m   v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   10m   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   13m   v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   11m   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   14m   v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   12m   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   15m   v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   13m   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   16m   v1.14.8-eks-b8860f
ip-10-111-0-40.eu-west-1.compute.internal   Ready   <none>   14m   v1.14.8-eks-b8860f
ip-10-111-1-43.eu-west-1.compute.internal   Ready   <none>   17m   v1.14.8-eks-b8860f

最終的に一時的に9台のEC2が起動する状態となり、最後に初期状態の2台に落ち着いた

8台のときに取得したスクショ。その後一台増えて、9台となった

daemonsetもこんな感じで推移

k get daemonset -n kube-system -w
NAME         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
aws-node     2         2         2       2            2           <none>          23d
kube-proxy   2         2         2       2            2           <none>          23d
kube-proxy   3     3     2     3     2     <none>   23d
aws-node   3     3     2     3     2     <none>   23d
aws-node   4     4     2     4     2     <none>   23d
kube-proxy   4     4     2     4     2     <none>   23d
kube-proxy   5     4     2     4     2     <none>   23d
aws-node   5     5     2     5     2     <none>   23d
kube-proxy   5     5     2     5     2     <none>   23d
kube-proxy   6     6     2     6     2     <none>   23d
aws-node   6     5     2     5     2     <none>   23d
aws-node   6     6     2     6     2     <none>   23d
aws-node   7     7     2     7     2     <none>   23d
kube-proxy   7     6     2     6     2     <none>   23d
kube-proxy   7     7     2     7     2     <none>   23d
aws-node   8     7     2     7     2     <none>   23d
kube-proxy   8     7     2     7     2     <none>   23d
aws-node   8     8     2     8     2     <none>   23d
kube-proxy   8     8     2     8     2     <none>   23d
kube-proxy   8     8     3     8     3     <none>   23d
kube-proxy   8     8     4     8     4     <none>   23d
kube-proxy   8     8     5     8     5     <none>   23d
aws-node   8     8     3     8     3     <none>   23d
aws-node   8     8     4     8     4     <none>   23d
aws-node   8     8     5     8     5     <none>   23d
kube-proxy   8     8     6     8     6     <none>   23d
kube-proxy   8     8     7     8     7     <none>   23d
aws-node   8     8     6     8     6     <none>   23d
kube-proxy   8     8     8     8     8     <none>   23d
aws-node   8     8     7     8     7     <none>   23d
aws-node   8     8     8     8     8     <none>   23d
aws-node   8     8     7     8     7     <none>   23d
kube-proxy   8     8     7     8     7     <none>   23d
kube-proxy   9     9     7     9     7     <none>   23d
aws-node   9     9     7     9     7     <none>   23d
kube-proxy   9     9     8     9     8     <none>   23d
aws-node   9     9     8     9     8     <none>   23d
aws-node   8     8     8     8     8     <none>   23d
kube-proxy   8     8     8     8     8     <none>   23d
aws-node   8     8     7     8     7     <none>   23d
kube-proxy   8     8     7     8     7     <none>   23d
aws-node   9     9     7     9     7     <none>   23d
kube-proxy   9     9     7     9     7     <none>   23d
kube-proxy   9     9     8     9     8     <none>   23d
aws-node   9     9     8     9     8     <none>   23d
aws-node   8     8     8     8     8     <none>   23d
kube-proxy   8     8     8     8     8     <none>   23d
aws-node   8     8     7     8     7     <none>   23d
kube-proxy   8     8     7     8     7     <none>   23d
aws-node   8     8     6     8     6     <none>   23d
kube-proxy   8     8     6     8     6     <none>   23d
aws-node   6     6     6     6     6     <none>   23d
kube-proxy   6     6     6     6     6     <none>   23d
aws-node   6     6     5     6     5     <none>   23d
kube-proxy   6     6     5     6     5     <none>   23d
aws-node   6     6     4     6     4     <none>   23d
kube-proxy   6     6     4     6     4     <none>   23d
aws-node   6     6     3     6     3     <none>   23d
kube-proxy   6     6     3     6     3     <none>   23d
aws-node   3     3     2     3     2     <none>   23d
kube-proxy   3     3     2     3     2     <none>   23d
kube-proxy   2     2     2     2     2     <none>   23d
aws-node   2     2     2     2     2     <none>   23d

podもこんな感じで推移

k get pod -w
NAME                       READY   STATUS    RESTARTS   AGE
awscli-7bc86c4488-2qj2d    1/1     Running   0          3m15s
awscli-7bc86c4488-7xvcn    1/1     Running   0          3m15s
awscli-7bc86c4488-868ln    1/1     Running   0          3m15s
awscli-7bc86c4488-bd5bl    1/1     Running   0          17d
awscli-7bc86c4488-jh6v4    1/1     Running   0          3m15s
awscli-7bc86c4488-q2l7m    1/1     Running   0          3m15s
awscli-7bc86c4488-qnfg5    1/1     Running   0          3m15s
awscli-7bc86c4488-qtvcq    1/1     Running   0          3m15s
awscli-7bc86c4488-t5zkl    1/1     Running   0          3m15s
awscli-7bc86c4488-z9tps    1/1     Running   0          3m15s
awscli2-654c7776dd-zrmxh   1/1     Running   0          17d
awscli-7bc86c4488-2qj2d   1/1   Terminating   0     12m
awscli-7bc86c4488-tv2f6   0/1   Pending   0     0s
awscli-7bc86c4488-tv2f6   0/1   Pending   0     0s
awscli-7bc86c4488-tv2f6   0/1   ContainerCreating   0     1s
awscli-7bc86c4488-7xvcn   1/1   Terminating   0     12m
awscli-7bc86c4488-nss6s   0/1   Pending   0     0s
awscli-7bc86c4488-nss6s   0/1   Pending   0     0s
awscli-7bc86c4488-nss6s   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-868ln   1/1   Terminating   0     12m
awscli-7bc86c4488-5vxdk   0/1   Pending   0     0s
awscli-7bc86c4488-5vxdk   0/1   Pending   0     0s
awscli-7bc86c4488-5vxdk   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-bd5bl   1/1   Terminating   0     17d
awscli-7bc86c4488-wb259   0/1   Pending   0     0s
awscli-7bc86c4488-wb259   0/1   Pending   0     0s
awscli-7bc86c4488-wb259   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-qtvcq   1/1   Terminating   0     12m
awscli-7bc86c4488-dwtk8   0/1   Pending   0     0s
awscli-7bc86c4488-dwtk8   0/1   Pending   0     0s
awscli-7bc86c4488-dwtk8   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-wb259   1/1   Running   0     10s
awscli-7bc86c4488-5vxdk   1/1   Running   0     11s
awscli-7bc86c4488-tv2f6   1/1   Running   0     12s
awscli-7bc86c4488-nss6s   1/1   Running   0     11s
awscli-7bc86c4488-dwtk8   1/1   Running   0     11s
awscli-7bc86c4488-2qj2d   0/1   Terminating   0     12m
awscli-7bc86c4488-7xvcn   0/1   Terminating   0     12m
awscli-7bc86c4488-bd5bl   0/1   Terminating   0     17d
awscli-7bc86c4488-868ln   0/1   Terminating   0     12m
awscli-7bc86c4488-qtvcq   0/1   Terminating   0     12m
awscli-7bc86c4488-qtvcq   0/1   Terminating   0     12m
awscli-7bc86c4488-7xvcn   0/1   Terminating   0     12m
awscli-7bc86c4488-7xvcn   0/1   Terminating   0     12m
awscli-7bc86c4488-2qj2d   0/1   Terminating   0     12m
awscli-7bc86c4488-2qj2d   0/1   Terminating   0     12m
awscli-7bc86c4488-qtvcq   0/1   Terminating   0     12m
awscli-7bc86c4488-qtvcq   0/1   Terminating   0     12m
awscli-7bc86c4488-bd5bl   0/1   Terminating   0     17d
awscli-7bc86c4488-bd5bl   0/1   Terminating   0     17d
awscli-7bc86c4488-868ln   0/1   Terminating   0     12m
awscli-7bc86c4488-868ln   0/1   Terminating   0     12m
awscli-7bc86c4488-jh6v4   1/1   Terminating   0     14m
awscli-7bc86c4488-dd679   0/1   Pending   0     0s
awscli-7bc86c4488-dd679   0/1   Pending   0     0s
awscli-7bc86c4488-dd679   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-q2l7m   1/1   Terminating   0     14m
awscli-7bc86c4488-5jvch   0/1   Pending   0     0s
awscli-7bc86c4488-5jvch   0/1   Pending   0     0s
awscli-7bc86c4488-5jvch   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-qnfg5   1/1   Terminating   0     14m
awscli-7bc86c4488-5knbq   0/1   Pending   0     0s
awscli-7bc86c4488-5knbq   0/1   Pending   0     0s
awscli-7bc86c4488-5knbq   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-t5zkl   1/1   Terminating   0     14m
awscli-7bc86c4488-zfhns   0/1   Pending   0     0s
awscli-7bc86c4488-zfhns   0/1   Pending   0     0s
awscli-7bc86c4488-zfhns   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-z9tps   1/1   Terminating   0     14m
awscli-7bc86c4488-jffnr   0/1   Pending   0     0s
awscli-7bc86c4488-jffnr   0/1   Pending   0     0s
awscli-7bc86c4488-jffnr   0/1   ContainerCreating   0     0s
awscli2-654c7776dd-zrmxh   1/1   Terminating   0     17d
awscli2-654c7776dd-8p7mx   0/1   Pending   0     0s
awscli2-654c7776dd-8p7mx   0/1   Pending   0     0s
awscli2-654c7776dd-8p7mx   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-zfhns   1/1   Running   0     1s
awscli-7bc86c4488-5knbq   1/1   Running   0     2s
awscli-7bc86c4488-jffnr   1/1   Running   0     2s
awscli2-654c7776dd-8p7mx   1/1   Running   0     2s
awscli-7bc86c4488-dd679   1/1   Running   0     10s
awscli-7bc86c4488-5jvch   1/1   Running   0     10s
awscli-7bc86c4488-jh6v4   0/1   Terminating   0     15m
awscli-7bc86c4488-qnfg5   0/1   Terminating   0     15m
awscli-7bc86c4488-q2l7m   0/1   Terminating   0     15m
awscli-7bc86c4488-jh6v4   0/1   Terminating   0     15m
awscli-7bc86c4488-jh6v4   0/1   Terminating   0     15m
awscli-7bc86c4488-z9tps   0/1   Terminating   0     15m
awscli-7bc86c4488-t5zkl   0/1   Terminating   0     15m
awscli2-654c7776dd-zrmxh   0/1   Terminating   0     17d
awscli-7bc86c4488-t5zkl   0/1   Terminating   0     15m
awscli-7bc86c4488-t5zkl   0/1   Terminating   0     15m
awscli-7bc86c4488-q2l7m   0/1   Terminating   0     15m
awscli-7bc86c4488-q2l7m   0/1   Terminating   0     15m
awscli2-654c7776dd-zrmxh   0/1   Terminating   0     17d
awscli2-654c7776dd-zrmxh   0/1   Terminating   0     17d
awscli-7bc86c4488-z9tps   0/1   Terminating   0     15m
awscli-7bc86c4488-z9tps   0/1   Terminating   0     15m
awscli-7bc86c4488-qnfg5   0/1   Terminating   0     15m
awscli-7bc86c4488-qnfg5   0/1   Terminating   0     15m
awscli-7bc86c4488-dwtk8   1/1   Terminating   0     5m23s
awscli-7bc86c4488-pd5c9   0/1   Pending   0     0s
awscli-7bc86c4488-pd5c9   0/1   Pending   0     0s
awscli-7bc86c4488-pd5c9   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-5knbq   1/1   Terminating   0     2m57s
awscli-7bc86c4488-ddx8r   0/1   Pending   0     0s
awscli-7bc86c4488-ddx8r   0/1   Pending   0     0s
awscli2-654c7776dd-8p7mx   1/1   Terminating   0     2m55s
awscli-7bc86c4488-ddx8r   0/1   ContainerCreating   0     0s
awscli2-654c7776dd-rh59c   0/1   Pending   0     0s
awscli2-654c7776dd-rh59c   0/1   Pending   0     0s
awscli2-654c7776dd-rh59c   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-nss6s   1/1   Terminating   0     5m25s
awscli-7bc86c4488-n2thk   0/1   Pending   0     0s
awscli-7bc86c4488-n2thk   0/1   Pending   0     0s
awscli-7bc86c4488-n2thk   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-pd5c9   1/1   Running   0     2s
awscli2-654c7776dd-rh59c   1/1   Running   0     2s
awscli-7bc86c4488-n2thk   1/1   Running   0     2s
awscli-7bc86c4488-jffnr   1/1   Terminating   0     3m3s
awscli-7bc86c4488-9x9ws   0/1   Pending   0     1s
awscli-7bc86c4488-9x9ws   0/1   Pending   0     1s
awscli-7bc86c4488-9x9ws   0/1   ContainerCreating   0     1s
awscli-7bc86c4488-wb259   1/1   Terminating   0     5m31s
awscli-7bc86c4488-5tthr   0/1   Pending   0     0s
awscli-7bc86c4488-5tthr   0/1   Pending   0     0s
awscli-7bc86c4488-5tthr   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-5vxdk   1/1   Terminating   0     5m32s
awscli-7bc86c4488-5vrhj   0/1   Pending   0     0s
awscli-7bc86c4488-5vrhj   0/1   Pending   0     0s
awscli-7bc86c4488-5vrhj   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-5jvch   1/1   Terminating   0     3m4s
awscli-7bc86c4488-md55k   0/1   Pending   0     0s
awscli-7bc86c4488-md55k   0/1   Pending   0     0s
awscli-7bc86c4488-md55k   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-zfhns   1/1   Terminating   0     3m3s
awscli-7bc86c4488-2plnt   0/1   Pending   0     0s
awscli-7bc86c4488-2plnt   0/1   Pending   0     0s
awscli-7bc86c4488-2plnt   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-pd5c9   1/1   Terminating   0     8s
awscli-7bc86c4488-8mbgr   0/1   Pending   0     0s
awscli-7bc86c4488-8mbgr   0/1   Pending   0     0s
awscli-7bc86c4488-8mbgr   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-9x9ws   1/1   Running   0     2s
awscli-7bc86c4488-5vrhj   1/1   Running   0     1s
awscli-7bc86c4488-md55k   1/1   Running   0     2s
awscli-7bc86c4488-8mbgr   1/1   Running   0     2s
awscli-7bc86c4488-ddx8r   1/1   Running   0     10s
awscli-7bc86c4488-2plnt   1/1   Running   0     4s
awscli-7bc86c4488-5tthr   1/1   Running   0     4s
awscli-7bc86c4488-5vrhj   1/1   Terminating   0     7s
awscli-7bc86c4488-cldgf   0/1   Pending   0     0s
awscli-7bc86c4488-cldgf   0/1   Pending   0     0s
awscli-7bc86c4488-cldgf   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-8mbgr   1/1   Terminating   0     7s
awscli-7bc86c4488-cx62z   0/1   Pending   0     0s
awscli-7bc86c4488-cx62z   0/1   Pending   0     0s
awscli-7bc86c4488-cx62z   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-n2thk   1/1   Terminating   0     15s
awscli-7bc86c4488-c4zdj   0/1   Pending   0     0s
awscli-7bc86c4488-c4zdj   0/1   Pending   0     0s
awscli-7bc86c4488-c4zdj   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-tv2f6   1/1   Terminating   0     5m41s
awscli-7bc86c4488-cm54v   0/1   Pending   0     0s
awscli-7bc86c4488-cm54v   0/1   Pending   0     0s
awscli-7bc86c4488-cm54v   0/1   ContainerCreating   0     0s
awscli-7bc86c4488-cldgf   1/1   Running   0     2s
awscli-7bc86c4488-cx62z   1/1   Running   0     2s
awscli-7bc86c4488-cm54v   1/1   Running   0     1s
awscli-7bc86c4488-c4zdj   1/1   Running   0     2s
awscli-7bc86c4488-5knbq   0/1   Terminating   0     3m27s
awscli-7bc86c4488-nss6s   0/1   Terminating   0     5m55s
awscli-7bc86c4488-dwtk8   0/1   Terminating   0     5m55s
awscli2-654c7776dd-8p7mx   0/1   Terminating   0     3m26s
awscli2-654c7776dd-8p7mx   0/1   Terminating   0     3m26s
awscli-7bc86c4488-nss6s   0/1   Terminating   0     5m59s
awscli-7bc86c4488-nss6s   0/1   Terminating   0     5m59s
awscli-7bc86c4488-5knbq   0/1   Terminating   0     3m31s
awscli-7bc86c4488-5knbq   0/1   Terminating   0     3m31s
awscli-7bc86c4488-jffnr   0/1   Terminating   0     3m33s
awscli-7bc86c4488-wb259   0/1   Terminating   0     6m2s
awscli-7bc86c4488-5vxdk   0/1   Terminating   0     6m3s
awscli-7bc86c4488-zfhns   0/1   Terminating   0     3m34s
awscli-7bc86c4488-5jvch   0/1   Terminating   0     3m35s
awscli-7bc86c4488-pd5c9   0/1   Terminating   0     39s
awscli-7bc86c4488-dwtk8   0/1   Terminating   0     6m2s
awscli-7bc86c4488-dwtk8   0/1   Terminating   0     6m3s
awscli2-654c7776dd-8p7mx   0/1   Terminating   0     3m34s
awscli2-654c7776dd-8p7mx   0/1   Terminating   0     3m34s
awscli-7bc86c4488-5jvch   0/1   Terminating   0     3m36s
awscli-7bc86c4488-5jvch   0/1   Terminating   0     3m36s
awscli-7bc86c4488-pd5c9   0/1   Terminating   0     40s
awscli-7bc86c4488-pd5c9   0/1   Terminating   0     41s
awscli-7bc86c4488-5vrhj   0/1   Terminating   0     37s
awscli-7bc86c4488-wb259   0/1   Terminating   0     6m9s
awscli-7bc86c4488-wb259   0/1   Terminating   0     6m9s
awscli-7bc86c4488-jffnr   0/1   Terminating   0     3m41s
awscli-7bc86c4488-jffnr   0/1   Terminating   0     3m41s
awscli-7bc86c4488-tv2f6   0/1   Terminating   0     6m11s
awscli-7bc86c4488-n2thk   0/1   Terminating   0     46s
awscli-7bc86c4488-8mbgr   0/1   Terminating   0     39s
awscli-7bc86c4488-5vrhj   0/1   Terminating   0     39s
awscli-7bc86c4488-5vrhj   0/1   Terminating   0     39s
awscli-7bc86c4488-5vxdk   0/1   Terminating   0     6m11s
awscli-7bc86c4488-5vxdk   0/1   Terminating   0     6m11s
awscli-7bc86c4488-zfhns   0/1   Terminating   0     3m42s
awscli-7bc86c4488-zfhns   0/1   Terminating   0     3m42s
awscli-7bc86c4488-tv2f6   0/1   Terminating   0     6m15s
awscli-7bc86c4488-tv2f6   0/1   Terminating   0     6m15s
awscli-7bc86c4488-n2thk   0/1   Terminating   0     49s
awscli-7bc86c4488-n2thk   0/1   Terminating   0     49s
awscli-7bc86c4488-8mbgr   0/1   Terminating   0     42s
awscli-7bc86c4488-8mbgr   0/1   Terminating   0     42s

古いNodeは全部Cordonが実行されて、こんな状態になった

k describe node ip-10-111-1-170.eu-west-1.compute.internal 
Name:               ip-10-111-1-170.eu-west-1.compute.internal
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=t3.medium
                    beta.kubernetes.io/os=linux
                    eks.amazonaws.com/nodegroup=managed1
                    eks.amazonaws.com/nodegroup-image=ami-059c6874350e63ca9
                    failure-domain.beta.kubernetes.io/region=eu-west-1
                    failure-domain.beta.kubernetes.io/zone=eu-west-1b
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=ip-10-111-1-170.eu-west-1.compute.internal
                    kubernetes.io/os=linux
Annotations:        node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Wed, 19 Feb 2020 05:25:17 +0000
Taints:             node.kubernetes.io/unschedulable:NoSchedule
Unschedulable:      true

新しいNodeは tainsはなく、nodegroup-imageが更新されていることが分かる

k describe node ip-10-111-1-43.eu-west-1.compute.internal 
                                                                                    Name:               ip-10-111-1-43.eu-west-1.compute.internal
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=t3.medium
                    beta.kubernetes.io/os=linux
                    eks.amazonaws.com/nodegroup=managed1
                    eks.amazonaws.com/nodegroup-image=ami-0b9d2c11b47bd8264
                    failure-domain.beta.kubernetes.io/region=eu-west-1
                    failure-domain.beta.kubernetes.io/zone=eu-west-1b
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=ip-10-111-1-43.eu-west-1.compute.internal
                    kubernetes.io/os=linux
Annotations:        node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Sun, 08 Mar 2020 16:19:10 +0000
Taints:             <none>
Unschedulable:      false

まとめ

最後はすごい散らかってしまいましたが、

AutoScalingの起動テンプレートを最新AMIにして更新し、EC2インスタンスの数を１つ増やす。
増やすことで、EC2インスタンスが１つ新規で起動する、かつ、それは新しいAMIで起動する
１つ増えるのではなく、一気に増えるようです。今回の検証では古いNodeは２つで、Max9Nodeになったので、7つ増えた？この原因は、ドキュメントの理解が足りない可能性があるので時間があるときにもう一度検証してみようと思います。
eks.amazonaws.com/nodegroup-image labelを見て最新のAMIになっていない、Worker Node すべてを　cordon して、Podがスケジュールされないようにする。
これはそのとおりでした
EKSが、そのcordonされたNodeの中から１つ選び、drainを実行する(drainのtimeoutは15分、その15分後の挙動については、エラーを発生させるか、nodeを強制deleteさせるか選ぶことが可能) drainが正常に完了した場合は、nodeを削除する
異常系は試していないのですが、drainされたnodeが削除されたのは事実です。
そうするとAutoScalingにより新規でNodeが起動する（かつ新しいAMIで）ので、上記の処理を繰り返して、すべてのNodeが更新されたら、EC2インスタンス数を１つ減らして元の起動テンプレートの状態に戻す
１つずつ減らしたかどうかは見ていませんが、もとのEC2インスタンス数に戻ったのは確かです。

まとめ

damonsetがあっても問題なくdrainしてくれました。本番環境だと、PDBが必須だと思うので、これも時間があるときに（次回のアップデートが降ってきたときに）PDBと組み合わせて検証してみようと思います。