はじめに
CRIランタイムとしてcontainerdを使用するk8sのインストールを初めて実施するので、作業ログとして本記事を残します。
環境
- ベアメタル
- ubutnu 20.04.2
- プロキシ環境
インストール対象
- containerd v1.3.3
- kubernetes v1.20.4
- Calico v3.18.0
サーバー構成
作業ログ
control planeのインストール
containerdのインストール
k8sのドキュメントに従って実施する。
Install and configure prerequisites:
controlplane:~$ cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
> overlay
> br_netfilter
> EOF
overlay
br_netfilter
controlplane:~$ sudo modprobe overlay
controlplane:~$ sudo modprobe br_netfilter
controlplane:~$ cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
> net.bridge.bridge-nf-call-iptables = 1
> net.ipv4.ip_forward = 1
> net.bridge.bridge-nf-call-ip6tables = 1
> EOF
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
controlplane:~$ sudo sysctl --system
* Applying /etc/sysctl.d/10-console-messages.conf ...
kernel.printk = 4 4 1 7
* Applying /etc/sysctl.d/10-ipv6-privacy.conf ...
net.ipv6.conf.all.use_tempaddr = 2
net.ipv6.conf.default.use_tempaddr = 2
* Applying /etc/sysctl.d/10-kernel-hardening.conf ...
kernel.kptr_restrict = 1
* Applying /etc/sysctl.d/10-link-restrictions.conf ...
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
* Applying /etc/sysctl.d/10-magic-sysrq.conf ...
kernel.sysrq = 176
* Applying /etc/sysctl.d/10-network-security.conf ...
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.all.rp_filter = 2
* Applying /etc/sysctl.d/10-ptrace.conf ...
kernel.yama.ptrace_scope = 1
* Applying /etc/sysctl.d/10-zeropage.conf ...
vm.mmap_min_addr = 65536
* Applying /usr/lib/sysctl.d/50-default.conf ...
net.ipv4.conf.default.promote_secondaries = 1
sysctl: setting key "net.ipv4.conf.all.promote_secondaries": Invalid argument
net.ipv4.ping_group_range = 0 2147483647
net.core.default_qdisc = fq_codel
fs.protected_regular = 1
fs.protected_fifos = 1
* Applying /usr/lib/sysctl.d/50-pid-max.conf ...
kernel.pid_max = 4194304
* Applying /etc/sysctl.d/99-kubernetes-cri.conf ...
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
* Applying /etc/sysctl.d/99-sysctl.conf ...
* Applying /usr/lib/sysctl.d/protect-links.conf ...
fs.protected_fifos = 1
fs.protected_hardlinks = 1
fs.protected_regular = 2
fs.protected_symlinks = 1
* Applying /etc/sysctl.conf ...
aptのプロキシ設定
controlplane:~$ export http_proxy=http://<proxy-host>:<proxy-port>/
controlplane:~$ cat <<EOF | sudo tee /etc/apt/apt.conf.d/90proxy
> Acquire::http::Proxy "${http_proxy}";
> Acquire::https::Proxy "${http_proxy}";
> EOF
... snip ...
Install containerd:
controlplane:~$ sudo apt-get update && sudo apt-get install -y containerd
（中略）
controlplane:~$ sudo mkdir -p /etc/containerd
controlplane:~$ containerd config default | sudo tee /etc/containerd/config.toml
（中略）
containerdのプロキシ設定
controlplane:~$ export no_proxy=10.0.0.0/8
controlplane:~$ sudo mkdir /etc/systemd/system/containerd.service.d
controlplane:~$ cat <<EOF | sudo tee /etc/systemd/system/containerd.service.d/http-proxy.conf
> [Service]
> Environment="HTTP_PROXY=${http_proxy}" "HTTPS_PROXY=${http_proxy}" "NO_PROXY=${no_proxy}"
> EOF
[Service]
Environment="HTTP_PROXY=http://xx.xx.xx.xx:xxxx" "HTTPS_PROXY=http://xx.xx.xx.xx:xxxx/" "NO_PROXY=10.0.0.0/8"
controlplane:~$ sudo systemctl daemon-reload
controlplane:~$ sudo systemctl restart containerd.service
controlplane:~$ unset http_proxy
controlplane:~$ unset no_proxy
CRIランタイムとkubeletのcgroup driverにsystemdを指定するのが推奨のようである。
https://kubernetes.io/docs/setup/production-environment/container-runtimes/#cgroup-drivers
しかし、kubeletにcgroup driverを指定するとkubadm initに失敗する問題に遭遇したため、今回はcontanerdのデフォルト（cgroupfs）のまま使用する。
イメージのpullができるかを確認するために、crictlをインストール。
controlplane:~$ VERSION="v1.20.0"
controlplane:~$ wget https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-$VERSION-linux-amd64.tar.gz
（中略）
controlplane:~$ sudo tar zxvf crictl-$VERSION-linux-amd64.tar.gz -C /usr/local/bin
crictl
controlplane:~$ rm -f crictl-$VERSION-linux-amd64.tar.gz
crictlの定義の作成
controlplane:~$ cat <<EOF | sudo tee /etc/crictl.yaml
> runtime-endpoint: unix:///run/containerd/containerd.sock
> image-endpoint: unix:///run/containerd/containerd.sock
> EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
controlplane:~$ sudo crictl version
Version: 0.1.0
RuntimeName: containerd
RuntimeVersion: 1.3.3-0ubuntu2.2
RuntimeApiVersion: v1alpha2
コンテナイメージがpullできるかを確認。
controlplane:~$ sudo crictl pull hello-world
Image is up to date for sha256:bf756fb1ae65adf866bd8c456593cd24beb6a0a061dedf42b26a993176745f6b
controlplane:~$ sudo crictl start hello-world
FATA[0000] starting the container "hello-world": rpc error: code = NotFound desc = an error occurred when try to find container "hello-world": not found
controlplane:~$ sudo crictl image ls
IMAGE TAG IMAGE ID SIZE
docker.io/library/hello-world latest bf756fb1ae65a 6.67kB
controlplane:~$ sudo crictl rmi bf756fb1ae65a
Deleted: docker.io/library/hello-world:latest
イメージのpullができたので、プロキシ設定がうまくできているようである。
kubernetesのインストール
k8sドキュメントの手順に従って、実施します。
swapの無効化
controlplane:~$ free -m
total used free shared buff/cache available
Mem: 385401 1067 383079 2 1253 382452
Swap: 8191 0 8191
controlplane:~$ sudo swapoff /swap.img
controlplane:~$ free -m
total used free shared buff/cache available
Mem: 385401 1068 383077 2 1255 382451
Swap: 0 0 0
controlplane:~$ sudo vim /etc/fstab
controlplane:~$ cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
# / was on /dev/sda2 during curtin installation
/dev/disk/by-uuid/1d472118-9c36-44a3-b159-46d4ae5c4ced / ext4 defaults 0 0
# /boot/efi was on /dev/sda1 during curtin installation
/dev/disk/by-uuid/E0B4-C69E /boot/efi vfat defaults 0 0
#/swap.img none swap sw 0 0
iptablesがnftablesバックエンドを使用しないようにする。
controlplane:~$ sudo apt-get install -y iptables arptables ebtables
（中略）
controlplane:~$ sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
controlplane:~$ sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
controlplane:~$ sudo update-alternatives --set arptables /usr/sbin/arptables-legacy
update-alternatives: using /usr/sbin/arptables-legacy to provide /usr/sbin/arptables (arptables) in manual mode
controlplane:~$ sudo update-alternatives --set ebtables /usr/sbin/ebtables-legacy
update-alternatives: using /usr/sbin/ebtables-legacy to provide /usr/sbin/ebtables (ebtables) in manual mode
Installing kubeadm, kubelet and kubectl
controlplane:~$ sudo apt-get update && sudo apt-get install -y apt-transport-https curl
（中略）
controlplane:~$ env https_proxy=http://<proxy-host>:<proxy-port>/ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
OK
controlplane:~$ cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
> deb https://apt.kubernetes.io/ kubernetes-xenial main
> EOF
deb https://apt.kubernetes.io/ kubernetes-xenial main
controlplane:~$ sudo apt-get update
（中略）
controlplane:~$ sudo apt-get install -y kubelet kubeadm kubectl
（中略）
controlplane:~$ sudo apt-mark hold kubelet kubeadm kubectl
kubelet set on hold.
kubeadm set on hold.
kubectl set on hold.
ノードにIPアドレスが複数ある場合は、あらかじめ変数ファイルへの指定が必要。
controlplane:~$ cat <<EOF | sudo tee /etc/default/kubelet
> KUBELET_EXTRA_ARGS=--node-ip=172.16.90.1
> EOF
KUBELET_EXTRA_ARGS=--node-ip=172.16.90.1
Initializing your control-plane node
controlplane:~$ sudo kubeadm init \
> --cri-socket=/run/containerd/containerd.sock \
> --apiserver-advertise-address=172.16.90.1 \
> --pod-network-cidr=192.168.0.0/16 \
> --dry-run
（中略）
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/tmp/kubeadm-init-dryrun952978795/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 172.16.90.1:6443 --token rk8u51.4wlm72el3vobkigm \
--discovery-token-ca-cert-hash sha256:9a8c3f956fd8431ec637f04557479dcf3f28d4c49c359b17211f9617f2815490
controlplane:~$ sudo kubeadm init \
> --cri-socket=/run/containerd/containerd.sock \
> --control-plane-endpoint 172.16.90.1 \
> --apiserver-advertise-address=172.16.90.1 \
> --apiserver-cert-extra-sans=xx.xx.xx.xx \
> --pod-network-cidr=192.168.0.0/16
（中略）
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 172.16.90.1:6443 --token n2nf5j.c6l9ehlwv4vcifn9 \
--discovery-token-ca-cert-hash sha256:a27b01a8428d526f5d727f13952f591003d994af33f1826006558d24f2da147c
ノードの確認
controlplane:~$ mkdir -p $HOME/.kube
controlplane:~$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
controlplane:~$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
controlplane:~$ unset KUBECONFIG
controlplane:~$ k get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
controlplane NotReady control-plane,master 47s v1.20.4 172.16.90.1 <none> Ubuntu 20.04.2 LTS 5.4.0-65-generic containerd://1.3.3-0ubuntu2.2
指定したIPアドレスがINTERNAL-IPに設定されていることが確認できた。
Calicoのインストール
controlplane:~$ kubectl apply -f https://docs.projectcalico.org/manifests/tigera-operator.yaml
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/imagesets.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/installations.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/tigerastatuses.operator.tigera.io created
namespace/tigera-operator created
podsecuritypolicy.policy/tigera-operator created
serviceaccount/tigera-operator created
clusterrole.rbac.authorization.k8s.io/tigera-operator created
clusterrolebinding.rbac.authorization.k8s.io/tigera-operator created
deployment.apps/tigera-operator created
controlplane:~$ kubectl apply -f https://docs.projectcalico.org/manifests/custom-resources.yaml
installation.operator.tigera.io/default created
controlplane:~$ kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
calico-system calico-kube-controllers-6797d7cc89-kr645 1/1 Running 0 34s
calico-system calico-node-kvqbm 1/1 Running 0 34s
calico-system calico-typha-57c94df999-glcs8 1/1 Running 0 34s
kube-system coredns-74ff55c5b-fzp4f 1/1 Running 0 16m
kube-system coredns-74ff55c5b-t2rq9 1/1 Running 0 16m
kube-system etcd-controlplane 1/1 Running 0 16m
kube-system kube-apiserver-controlplane 1/1 Running 0 16m
kube-system kube-controller-manager-controlplane 1/1 Running 0 16m
kube-system kube-proxy-qtz96 1/1 Running 0 16m
kube-system kube-scheduler-controlplane 1/1 Running 0 16m
tigera-operator tigera-operator-549bf46b5c-jrzc5 1/1 Running 0 43s
controlplane:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
controlplane Ready control-plane,master 16m v1.20.4
Calicoがいつの間にかoperatorになっていた。
とりあえずcontrol planeのインストールが完了した！
Worker nodeのインストール
Control planeインストール手順のkubeadm init実行前までの手順をWorker nodeでも実施する。
その後、kubeadm joinコマンドでworker nodeをk8sクラスタに追加する。
wworker01:~$ sudo kubeadm join 172.16.90.1:6443 --token n2nf5j.c6l9ehlwv4vcifn9 \
> --discovery-token-ca-cert-hash sha256:a27b01a8428d526f5d727f13952f591003d994af33f1826006558d24f2da147c
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
ノード一覧の確認
controlplane:~$ k get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
controlplane Ready control-plane,master 48m v1.20.4 172.16.90.1 <none> Ubuntu 20.04.2 LTS 5.4.0-65-generic containerd://1.3.3-0ubuntu2.2
worker01 Ready <none> 29m v1.20.4 172.16.90.2 <none> Ubuntu 20.04.2 LTS 5.4.0-65-generic containerd://1.3.3-0ubuntu2.2
worker02 Ready <none> 4m58s v1.20.4 172.16.90.3 <none> Ubuntu 20.04.1 LTS 5.4.0-42-generic containerd://1.3.3-0ubuntu2.2
Worker nodeが追加できたので、これでk8sクラスタの構築が完了！
参考
- https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd
- https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md#install-crictl
- https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
- https://github.com/kubernetes/kubeadm/issues/203#issuecomment-412248894
- https://docs.projectcalico.org/getting-started/kubernetes/quickstart