k8s勉強メモ(個人メモ)

kubernetesとは

https://kubernetes.io/
https://kubernetes.io/docs/home/
当初、私みたいな英語が弱い人間にはkubernetesの文章を読むの難しかったけど、最近は英語以外にも色々と追加されてありがたい

環境準備

hostマシンはMacでVMはmultipassを使って構築する
multipassはこのメモを参照

local以外にもhttps://katacoda.com/ でplaygroundを無料で試すのもできる、（ただし今は10分だけ使えなくなってる、

INSTALL

順番的には大体

1. vm環境準備 (このメモでは扱ってない
2. docker (master, workernode
3. k8s
   1. install
   2. master 設定
      1. kubeadm, kubectl, kubelet
      2. network add-on設定(weave networks)
   3. worker node設定
      1. kubeadm, kubectl, kubelet
      2. joint to master

準備作業

構成

master 1 < 必ず cpu 2core mem 2GBにする必要がある、そうじゃないと後で怒られるから
node 2

disable, off swap!!!

vmでswapoffをしないと後で怒られる

$ sudo swapoff -a && sudo sed -i '/swap/s/^/#/' /etc/fstab

docker install

ubuntuなので https://docs.docker.com/engine/install/ubuntu/ のガイドの通りインストール

install command

$ sudo apt-get remove docker docker-engine docker.io containerd runc
$ sudo apt-get update
$ sudo apt-get install \
    ca-certificates \
    curl \
    gnupg \
    lsb-release
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
$ echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
$ sudo apt-get update
$ sudo apt-get install docker-ce docker-ce-cli containerd.io

※dockerのcgroupdriveをsystemdに設定、しないとkubeletが起動できない問題がある
https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/

変更方法
https://software.fujitsu.com/jp/manual/manualfiles/m190006/j2ul2418/02z200/installation-guide/dita/setup-after-cgroup-driver-update-tool.html

$ sudo cat <<EOF | sudo tee /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"]
}
EOF

systemctl enableしておく（自動起動

sudo systemctl enable docker
sudo systemctl restart docker

docker versionで確認

ubuntu@kb-master1:~$ sudo docker version
Client: Docker Engine - Community
 Version:           20.10.14
 API version:       1.41
 Go version:        go1.16.15
 Git commit:        a224086
 Built:             Thu Mar 24 01:48:02 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.14
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.15
  Git commit:       87a90dc
  Built:            Thu Mar 24 01:45:53 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.11
  GitCommit:        3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc:
  Version:          1.0.3
  GitCommit:        v1.0.3-0-gf46b6ba
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

k8s

install方法

kubernetesのdocumentの通りinstallする
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

OP

install command

$ sudo cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

$ sudo cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

# 1. update apt package index and install packages needed to use the k8s apt repo
$ sudo sudo sysctl --system
$ sudo apt-get install -y apt-transport-https ca-certificates curl

# 2. gpg
$ sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg

# 3. add k8s apt repo
$ echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

# 4. install kublet, kubeadm, kubectl
$ sudo apt-get update
$ sudo apt-get install -y kubelet kubeadm kubectl
$ sudo apt-mark hold kubelet kubeadm kubectl

# kubeletをenableしとく <<-- 必要であれば
$ sudo systemctl enable kubelet

Cluster構成

kubeadm init (master)

• 必ずmaster-nodeで実行すること
• kubeadm実行後、発行してもらったtokenをメモっておくこと

$ sudo kubeadm init
......
.......
  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config
.........
kubeadm join 192.168.64.7:6443 --token s74273.xf7xqysb1dyvxjkl \
	--discovery-token-ca-cert-hash sha256:5a12c99481b26433cd21d6476b28e9908952f802e56526f61066f1f2a0114651

# 下記大事
$ cat <<EOF | sudo tee kb-token.txt
kubeadm join 192.168.64.7:6443 --token s74273.xf7xqysb1dyvxjkl \
	--discovery-token-ca-cert-hash sha256:5a12c99481b26433cd21d6476b28e9908952f802e56526f61066f1f2a0114651
EOF

# path追加
$ mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

# testしてみる
$ kubectl get nodes
ubuntu@kb-master1:~$ kubectl get nodes
NAME         STATUS     ROLES                  AGE     VERSION
kb-master1   NotReady   control-plane,master   6m47s   v1.23.5

まだNotReady状態、network add-on install続こう

pod network add-on install (master)

いろんなadd-onがあるみたいだけど、weave net worksをinstallしてみる
https://www.weave.works/docs/net/latest/kubernetes/kube-addon/

# install
$ kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
# 確認
$ kubectl get nodes
NAME         STATUS   ROLES                  AGE   VERSION
kb-master1   Ready    control-plane,master   14m   v1.23.5

STATUSがReadyに変わった！

kubeadm join (worker-node)

masterでkubeadm initで発行してもらったtokenを確認nodeでjoinする

$ sudo kubeadm join 192.168.64.7:6443 --token s74273.xf7xqysb1dyvxjkl \
	--discovery-token-ca-cert-hash sha256:5a12c99481b26433cd21d6476b28e9908952f802e56526f61066f1f2a0114651

最終確認(masterで)

数秒間かかったけどReadyになってるを確認

ubuntu@kb-master1:~$ kubectl get nodes -o wide
NAME         STATUS   ROLES                  AGE     VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
kb-master1   Ready    control-plane,master   24m     v1.23.5   192.168.64.7   <none>        Ubuntu 20.04.4 LTS   5.4.0-105-generic   docker://20.10.14
kb-node1     Ready    <none>                 4m31s   v1.23.5   192.168.64.4   <none>        Ubuntu 20.04.4 LTS   5.4.0-105-generic   docker://20.10.14
kb-node2     Ready    <none>                 4m8s    v1.23.5   192.168.64.5   <none>        Ubuntu 20.04.4 LTS   5.4.0-105-generic   docker://20.10.14
kb-node3     Ready    <none>                 4m6s    v1.23.5   192.168.64.6   <none>        Ubuntu 20.04.4 LTS   5.4.0-105-generic   docker://20.10.14

その他便利機能

https://kubernetes.io/docs/tasks/tools/included/optional-kubectl-configs-bash-linux/
auto completionを設定しておく

$ source <(kubectl completion bash)
$ echo 'source <(kubectl completion bash)' >>~/.bashrc
# kubeadmもしておく
$ source <(kubeadm completion bash)
$ echo 'source <(kubeadm completion bash)' >>~/.bashrc

環境設定しながら会った問題

kubletが起動できない

kubernetesをinstall後、syslogを見ると、下記のエラーでkubeletが起動できない

Apr 12 08:20:50 kb-master1 systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 4.
Apr 12 08:20:50 kb-master1 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Apr 12 08:20:50 kb-master1 systemd[1]: Started kubelet: The Kubernetes Node Agent.
Apr 12 08:20:50 kb-master1 kubelet[17383]: E0412 17:20:50.165091   17383 server.go:205] "Failed to load kubelet config file" err="failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory" path="/var/lib/kubelet/config.yaml"
Apr 12 08:20:50 kb-master1 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Apr 12 08:20:50 kb-master1 systemd[1]: kubelet.service: Failed with result 'exit-code'.

解決方法

$ sudo cat <<EOF | sudo tee /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"]
}
EOF
$ sudo systemctl restart docker

関連issue :
https://github.com/kubernetes/kubeadm/issues/1893#issuecomment-827208982