問題

Kubernetes再インストールのためにkubeadm reset後にkubeadm　initするが上手くいかない。

kubeadm init --pod-network-cidr=10.244.0.0/16
...(中略)...
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.

        Unfortunately, an error has occurred:
                timed out waiting for the condition

        This error is likely caused by:
                - The kubelet is not running
                - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

        If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
                - 'systemctl status kubelet'
                - 'journalctl -xeu kubelet'

        Additionally, a control plane component may have crashed or exited when started by the container runtime.
        To troubleshoot, list all containers using your preferred container runtimes CLI.

        Here is one example how you may list all Kubernetes containers running in docker:
                - 'docker ps -a | grep kube | grep -v pause'
                Once you have found the failing container, you can inspect its logs with:
                - 'docker logs CONTAINERID'

error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher

原因

dockerとのcgroup driverの不整合により、
kubeletが起動に失敗している。

journalctl -xeu kubelet によると、

"Failed to run kubelet" err="failed to run Kubelet: misconfiguration: kubelet cgroup driver: \"systemd\" is different from docker cgroup driver: “cgroupfs”

Kubernetes v1.21から、kubeadmの標準のcgroupのdriverがcgroupfsからsystemdに変更されたらしい

解決策

dockerのdriverを確認する

docker info | grep Cgroup
 Cgroup Driver: cgroupfs
 Cgroup Version: 1

dockerのdriverをsystemdに変更する

 vim /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}

daemonをリロードしてdockerを再起動

systemctl daemon-reload
systemctl restart docker
docker info | grep Cgroup
 Cgroup Driver: systemd
 Cgroup Version: 1

再び、kubeadam initすると今後は成功した。

kubeadm init --pod-network-cidr=10.244.0.0/16

解決

環境

ソフト	バージョン
OS	Ubuntu 20.04.2LTS
Docker	20.10.7
kubernetes	v1.22.2

感想

Versionを上げるときは変更事項を確認してから上げる必要があると反省