Kubernetes 1.24からDockershimが廃止となりcriを使う必要がでてきました。
上に乗せるコンテナのビルド自体はDockerでやってよいと思いますが、クラスタはcri-docker or cri-oで構築する必要があります。
ちなみに別環境でcontainerdを利用している1.23をアップデートする際にcri-oに同時に上げようとしましたが、うまくいきませんでした。（わからん）

今回は、Rockey Linux 9にKubernetes v1.25.5, cri-oを新規セットアップしました。

ここではオレオレk8s環境を、minikubeなどを使わずにkubeadmを使ってサーバにセットアップしていきます。
公式の手順を参考にしながら進めましょう。
https://kubernetes.io/ja/docs/setup/independent/install-kubeadm/

[root@localhost tmp]# uname -a
Linux localhost.localdomain 5.14.0-162.6.1.el9_1.0.1.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Nov 28 18:44:09 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
[root@localhost tmp]# cat /etc/os-release 
NAME="Rocky Linux"
VERSION="9.1 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.1"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Rocky Linux 9.1 (Blue Onyx)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:9::baseos"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9"
ROCKY_SUPPORT_PRODUCT_VERSION="9.1"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.1"

TL;DR

オレオレk8sクラスタ(1台)を立ててkubectl applyできるようにする。

OSの準備

公式の手順を参考にしながら進めましょう。
https://kubernetes.io/ja/docs/setup/independent/install-kubeadm/

swapをoff

始める前にの項にswapをOFFにしておけよってあるのでします。
Swapがオフであること。kubeletが正常に動作するためにはswapは必ずオフでなければなりません。

[root@localhost tmp]# sudo swapoff -a
[root@localhost tmp]# free -h
               total        used        free      shared  buff/cache   available
Mem:           7.5Gi       1.4Gi       3.9Gi        10Mi       2.5Gi       6.1Gi
Swap:             0B          0B          0B

sudo swapoff -aしてswap領域がなくなっていることを確認すればよろしいかと。

その他

そのほかにMACアドレスやホスト名がユニークであること、product_uuidがほかサーバと重複していないか確認すべし！とあります。
firewallによって利用するポートを許可してください。とりあえず今回はお試しなので systemctl stop firewalld しました。（外に出ているサーバの場合には絶対にしないように注意）

Install CRI-O

こちらを参考に勧めていきます。
https://kubernetes.io/ja/docs/setup/production-environment/container-runtimes/#cri-o

networkのブリッジの設定

書いてあるとおりにコピペします。ネットワークのパケットをブリッジ＆ルーティングできるようにします。
k8sはiptablesを利用してパケットを転送します。

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

# sysctl params required by setup, params persist across reboots
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

# Apply sysctl params without reboot
sudo sysctl --system

リポジトリ追加

下記にRockey用のリポジトリがないため、CentOS_8_Streamで代用します。（ビルドしたほうがよいのかも）

export OS=CentOS_8_Stream
export VERSION=1.25
curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8_Stream/devel:kubic:libcontainers:stable.repo
curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo

dnf install cri-o
dnf install containernetworking-plugins

crio.conf調整

/etc/crio/crio.confに以下を追加（コメントアウトされてるので外すのでもおｋ

[crio.runtime.runtimes.runc]
runtime_path = "" 
runtime_type = "oci" 
runtime_root = "/run/runc"

cri-o動かす。

systemctlで起動させてあげます。

sudo systemctl daemon-reload
sudo systemctl enable crio
sudo systemctl start crio

[root@k8s-node01 crio]# systemctl status crio
● crio.service - Container Runtime Interface for OCI (CRI-O)
     Loaded: loaded (/usr/lib/systemd/system/crio.service; enabled; vendor preset: disabled)
     Active: active (running) since Thu 2022-12-15 03:55:24 EST; 5s ago
       Docs: https://github.com/cri-o/cri-o
   Main PID: 12123 (crio)
      Tasks: 28
     Memory: 38.2M
        CPU: 351ms
     CGroup: /system.slice/crio.service
             └─12123 /usr/bin/crio

Dec 15 03:55:24 k8s-node01.ceres.local crio[12123]: time="2022-12-15 03:55:24.006674834-05:00" level=info msg="RDT not available in the host system" 
Dec 15 03:55:24 k8s-node01.ceres.local crio[12123]: time="2022-12-15 03:55:24.011036842-05:00" level=info msg="Conmon does support the --sync option" 
Dec 15 03:55:24 k8s-node01.ceres.local crio[12123]: time="2022-12-15 03:55:24.011088556-05:00" level=info msg="Conmon does support the --log-global-size-max option" 
Dec 15 03:55:24 k8s-node01.ceres.local crio[12123]: time="2022-12-15 03:55:24.018631810-05:00" level=info msg="Found CNI network crio (type=bridge) at /etc/cni/net.d/100-crio-bridge>
Dec 15 03:55:24 k8s-node01.ceres.local crio[12123]: time="2022-12-15 03:55:24.025073153-05:00" level=info msg="Found CNI network 200-loopback.conf (type=loopback) at /etc/cni/net.d/>
Dec 15 03:55:24 k8s-node01.ceres.local crio[12123]: time="2022-12-15 03:55:24.025121761-05:00" level=info msg="Updated default CNI network name to crio" 
Dec 15 03:55:24 k8s-node01.ceres.local crio[12123]: time="2022-12-15 03:55:24.044838541-05:00" level=info msg="Serving metrics on :9537 via HTTP" 
Dec 15 03:55:24 k8s-node01.ceres.local crio[12123]: time="2022-12-15 03:55:24.045259409-05:00" level=error msg="Writing clean shutdown supported file: open /var/lib/crio/clean.shutd>
Dec 15 03:55:24 k8s-node01.ceres.local crio[12123]: time="2022-12-15 03:55:24.045361471-05:00" level=error msg="Failed to sync parent directory of clean shutdown file: open /var/lib>
Dec 15 03:55:24 k8s-node01.ceres.local systemd[1]: Started Container Runtime Interface for OCI (CRI-O).

インストール

kubeadm、kubelet、kubectlのインストールを参考にyumリポジトリを追加します。

公式のままですね。

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF

selinuxを調整して、インストールします。公式ではyumコマンドを利用していますが、fedoraさんらしくdnfでインストールします。

# sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
# dnf install -y kubelet-1.25.5-0 kubeadm-1.25.5-0 kubectl-1.25.5-0 --disableexcludes=kubernetes
systemctl enable --now kubelet

packageをインストールしたら一度rebootしておいてください。selinuxをdisableにしておきたいので。

kubeadmによる初期設定

Creating a single control-plane cluster with kubeadm

dnf updateなどはしてあるので飛ばします。

kubeadm init <args>をします。<args>にはオプションを入れられます。次の項目でCalicoというコンテナネットワークプラグインを使うので、--pod-network-cidr=10.244.0.0/16を追加しておきます。後述のcalico側のマニフェストファイル側を変えてもいい。

[root@localhost ~]# kubeadm init --pod-network-cidr=10.244.0.0/16
I0119 02:55:04.007666   12858 version.go:256] remote version is much newer: v1.26.1; falling back to: stable-1.25
[init] Using Kubernetes version: v1.25.6
[preflight] Running pre-flight checks
        [WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
        [WARNING SystemVerification]: missing optional cgroups: blkio
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local localhost.localdomain] and IPs [10.96.0.1 172.xx.12.62]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost localhost.localdomain] and IPs [172.xx.12.62 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost localhost.localdomain] and IPs [172.xx.12.62 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 17.501654 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node localhost.localdomain as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node localhost.localdomain as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: 815j1e.wy5xkrhs0fkwkkcx
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.xx.12.62:6443 --token 815j1e.wy5xkrhs0fkwkkcx \
        --discovery-token-ca-cert-hash sha256:88abcd03f98035ef780d5a2455d89c6a3c8fc860bf2baf285396a78e349499f2

上にあるように、設定ファイルをhomeディレクトリに持ってきます。

# mkdir -p $HOME/.kube
# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
# chown $(id -u):$(id -g) $HOME/.kube/config
# cat ~/.kube/config
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ながい
    server: https://172.xx.12.62:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubernetes-admin
  name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
  user:
    client-certificate-data: ながい
    client-key-data: ながい

これでkubectlで見れるようになるはず。

[root@localhost lib]# kubectl get all -A
NAMESPACE     NAME                                                READY   STATUS    RESTARTS   AGE
kube-system   pod/coredns-565d847f94-b9p5t                        0/1     Pending   0          38s
kube-system   pod/coredns-565d847f94-tm6lw                        0/1     Pending   0          38s
kube-system   pod/etcd-localhost.localdomain                      1/1     Running   2          52s
kube-system   pod/kube-apiserver-localhost.localdomain            1/1     Running   2          54s
kube-system   pod/kube-controller-manager-localhost.localdomain   1/1     Running   2          53s
kube-system   pod/kube-proxy-64t26                                1/1     Running   0          38s
kube-system   pod/kube-scheduler-localhost.localdomain            1/1     Running   2          52s

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  54s
kube-system   service/kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   53s

NAMESPACE     NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system   daemonset.apps/kube-proxy   1         1         1       1            1           kubernetes.io/os=linux   53s

NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/coredns   0/2     2            0           53s

NAMESPACE     NAME                                 DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/coredns-565d847f94   2         2         0       39s

現段階ではcorednsのpodは動かないよ。

CNIのセットアップ

pod同士のネットワークはプラグインで入れる必要があるので、Calicoを入れる。
何種類か存在する。ontainer Network Interface (CNI) の略。

この手順通りにやればおｋ
https://projectcalico.docs.tigera.io/getting-started/kubernetes/self-managed-onprem/onpremises
manifestを直接読み込ませる方法とoperatorのpodがやってくれる方法と2種類あるが、今回は前者で。

※2023/04/11 tigera-operator.yamlを使う方法でも問題ありませんでした。custom-resources.yaml にipPoolsを設定します。calico:v3.25.1, k8s:v1.26.3

canalというcalicoとFlannelを組み合わせたものもあるが、calicoにVXLAN機能が内報されたので特段理由がなければ使わなくてよさそう。

calico.yaml

calico.yaml をダウンロードしてCIDRのところを編集する必要がある。
昔は10.244.0.0/16が例に書いてあった気がするが、192.168.0.0/16になっていたので、DLして編集しよう。

# curl https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/calico.yaml -O

CALICO_IPV4POOL_CIDRのところのコメントを外して192.168.0.0/16から10.244.0.0/16に変更。

            # The default IPv4 pool to create on startup if none exists. Pod IPs will be
            # chosen from this range. Changing this value after installation will have
            # no effect. This should fall within `--cluster-cidr`.
            # - name: CALICO_IPV4POOL_CIDR
            #   value: "192.168.0.0/16"

を下のようにする。

            # The default IPv4 pool to create on startup if none exists. Pod IPs will be
            # chosen from this range. Changing this value after installation will have
            # no effect. This should fall within `--cluster-cidr`.
            - name: CALICO_IPV4POOL_CIDR
              value: "10.244.0.0/16"

編集したマニフェストをapply。

[root@localhost ~]# kubectl apply -f ./calico.yaml
poddisruptionbudget.policy/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
serviceaccount/calico-node created
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
deployment.apps/calico-kube-controllers created

作成中。

[root@localhost ~]# kubectl get all -A
NAMESPACE     NAME                                                READY   STATUS              RESTARTS   AGE
kube-system   pod/calico-kube-controllers-74677b4c5f-7mm8p        0/1     ContainerCreating   0          17s
kube-system   pod/calico-node-bpznj                               0/1     Init:2/3            0          17s
kube-system   pod/coredns-565d847f94-b9p5t                        0/1     ContainerCreating   0          8m51s
kube-system   pod/coredns-565d847f94-tm6lw                        0/1     ContainerCreating   0          8m51s
kube-system   pod/etcd-localhost.localdomain                      1/1     Running             2          9m5s
kube-system   pod/kube-apiserver-localhost.localdomain            1/1     Running             2          9m7s
kube-system   pod/kube-controller-manager-localhost.localdomain   1/1     Running             2          9m6s
kube-system   pod/kube-proxy-64t26                                1/1     Running             0          8m51s
kube-system   pod/kube-scheduler-localhost.localdomain            1/1     Running             2          9m5s

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  9m7s
kube-system   service/kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   9m6s

NAMESPACE     NAME                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system   daemonset.apps/calico-node   1         1         0       1            0           kubernetes.io/os=linux   17s
kube-system   daemonset.apps/kube-proxy    1         1         1       1            1           kubernetes.io/os=linux   9m6s

NAMESPACE     NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/calico-kube-controllers   0/1     1            0           17s
kube-system   deployment.apps/coredns                   0/2     2            0           9m6s

NAMESPACE     NAME                                                 DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/calico-kube-controllers-74677b4c5f   1         1         0       17s
kube-system   replicaset.apps/coredns-565d847f94                   2         2         0       8m52s

すべてのpodがrunningになる。

[root@localhost ~]# kubectl get all -A
NAMESPACE     NAME                                                READY   STATUS    RESTARTS   AGE
kube-system   pod/calico-kube-controllers-74677b4c5f-7mm8p        1/1     Running   0          45s
kube-system   pod/calico-node-bpznj                               1/1     Running   0          45s
kube-system   pod/coredns-565d847f94-b9p5t                        1/1     Running   0          9m19s
kube-system   pod/coredns-565d847f94-tm6lw                        1/1     Running   0          9m19s
kube-system   pod/etcd-localhost.localdomain                      1/1     Running   2          9m33s
kube-system   pod/kube-apiserver-localhost.localdomain            1/1     Running   2          9m35s
kube-system   pod/kube-controller-manager-localhost.localdomain   1/1     Running   2          9m34s
kube-system   pod/kube-proxy-64t26                                1/1     Running   0          9m19s
kube-system   pod/kube-scheduler-localhost.localdomain            1/1     Running   2          9m33s

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  9m35s
kube-system   service/kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   9m34s

NAMESPACE     NAME                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system   daemonset.apps/calico-node   1         1         1       1            1           kubernetes.io/os=linux   45s
kube-system   daemonset.apps/kube-proxy    1         1         1       1            1           kubernetes.io/os=linux   9m34s

NAMESPACE     NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/calico-kube-controllers   1/1     1            1           45s
kube-system   deployment.apps/coredns                   2/2     2            2           9m34s

NAMESPACE     NAME                                                 DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/calico-kube-controllers-74677b4c5f   1         1         1       45s
kube-system   replicaset.apps/coredns-565d847f94                   2         2         2       9m20s

できた。

ip aコマンドをするとNICが増えているのがわかる。tunl0とcaliなんちゃらがそれ。

[root@localhost etc]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:ae:20:df brd ff:ff:ff:ff:ff:ff
    inet 172.xx.12.62/24 brd 172.xx.12.255 scope global noprefixroute enp1s0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:feae:20df/64 scope link 
       valid_lft forever preferred_lft forever
3: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
    inet 10.244.102.128/32 scope global tunl0
       valid_lft forever preferred_lft forever
6: cali9cc271e60ca@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns 38b565cd-5099-43f6-a21e-82ccd68eda6c
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link 
       valid_lft forever preferred_lft forever
7: cali015e1fca632@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns 86d781d2-7ad6-4f84-9a1b-febd17732b48
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link 
       valid_lft forever preferred_lft forever
8: cali2db521aeade@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns c1ac37af-9a8d-45a0-93ae-4f6a0cdda07f
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link 
       valid_lft forever preferred_lft forever

Control Planeでpodを動かす場合はnodeのtaintを消して、podをスケジューリングできるようにする。（デフォルトでControl Planeでは動かないようになっている。）

[root@localhost ~]# kubectl taint nodes --all node-role.kubernetes.io/control-plane-
node/localhost.localdomain untainted

動作確認

ためしにnginxをデプロイしてみます。

cat <<EOF | kubectl apply -f -
kind: Deployment
apiVersion: apps/v1
metadata:
  name: nginx-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-test
  template:
    metadata:
      labels:
        app: nginx-test
    spec:
      containers:
        - args:
          image: nginx:latest
          imagePullPolicy: IfNotPresent
          name: nginx-test
          ports:
            - containerPort: 80
              protocol: TCP

---
kind: Service
apiVersion: v1
metadata:
  name: nginx-test-svc
spec:
  ports:
  - name: "http-port"
    protocol: TCP
    port: 8080
    targetPort: 80
  selector:
    app: nginx-test

EOF
deployment.apps/nginx-test created
service/nginx-test-svc created

できた。

[root@localhost ~]# kubectl get all 
NAME                              READY   STATUS    RESTARTS   AGE
pod/nginx-test-54cdc496f7-zbg6p   1/1     Running   0          40s

NAME                     TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/kubernetes       ClusterIP   10.96.0.1      <none>        443/TCP    17m
service/nginx-test-svc   ClusterIP   10.96.28.133   <none>        8080/TCP   5m45s

NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-test   1/1     1            1           5m45s

NAME                                    DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-test-54cdc496f7   1         1         1       40s
replicaset.apps/nginx-test-64d5bd95d7   0         0         0       5m45s

確認

[murata@localhost ~]$ curl 10.96.28.133:8080 -I
HTTP/1.1 200 OK
Server: nginx/1.23.3
Date: Thu, 19 Jan 2023 09:41:05 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Tue, 13 Dec 2022 15:53:53 GMT
Connection: keep-alive
ETag: "6398a011-267"
Accept-Ranges: bytes

ちゃんとレスポンスも受け取れる。

まとめ。

まだIngress（ロードバランサ）の設定をしていないため外からの通信はできないけど、kubeadmを使って構築はこんな感じになる。
クラスタにノード(サーバ)を追加する際にはkubeadm joinのコマンドをやれば勝手につないでくれる。