CentOS8でkubernetesを動かす。書き直し(2019/12時点)

2020/08/22
CentOS8からFireWallのバックエンドがnftableに変わったためCNIのkube-routerが正常に動かないです。
calicoに変更しました。

---

前回のこの記事でEL7向けのrpmパッケージを入れるのが気に食わなかったので書き直した

2020/02/02
「9.2.cgroup-driver設定」の誤記を修正

1.socat,iproute-tc,ipvsadm,tar,conntrack-toolsのインストール

# dnf install socat iproute-tc ipvsadm tar,conntrack-tools

2.podman,runc,CNI Pluginsのアンインストール

2.1.アンインストール

# dnf remove podman runc containernetworking-plugins

2.2.自動インストールの無効化

# echo "exclude=podman* runc* containernetworking-plugins*"  >> /etc/yum.conf

3.swap無効化

# swapoff -a
# sed -i -e '/swap/d' /etc/fstab

4.SELinux無効化

# setenforce 0
# sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

5.カーネルパラメタ設定

# cat << EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

# sysctl --system

6.firewalld,nftablesの無効化

# systemctl stop firewalld
# systemctl stop nftables

# systemctl disable firewalld
# systemctl disable nftables

7.docker-ceのインストール

7.1.dockerグループ作成

# groupadd -g 2000 docker

7.2.バイナリパッケージの展開

# cd /var/tmp
# curl -LO https://download.docker.com/linux/static/stable/x86_64/docker-19.03.5.tgz
# tar xzvf docker-19.03.5.tgz
# chown root:root docker/*
# cp docker/* /usr/bin/

7.3.systemd unit作成

# cat << EOF > /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket

[Service]
Type=notify
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP \$MAINPID
TimeoutSec=0
RestartSec=2
Restart=always

StartLimitBurst=3

StartLimitInterval=60s

LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity

TasksMax=infinity

Delegate=yes

KillMode=process

[Install]
WantedBy=multi-user.target
EOF

# cat << EOF > /usr/lib/systemd/system/docker.socket
[Unit]
Description=Docker Socket for the API
PartOf=docker.service

[Socket]
ListenStream=/var/run/docker.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker

[Install]
WantedBy=sockets.target
EOF

# cat << EOF > /usr/lib/systemd/system/containerd.service
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target

[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/bin/containerd
KillMode=process
Delegate=yes
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity

[Install]
WantedBy=multi-user.target
EOF

7.4.CNI設定用ディレクトリ作成

# mkdir -p /etc/cni/net.d/

7.5.起動

# systemctl daemon-reload
# systemctl enable containerd.service
# systemctl enable docker.socket
# systemctl enable docker.service
# systemctl restart docker

8.CNI pluginsのインストール

# CNI_VERSION="v0.7.5"
# mkdir -p /opt/cni/bin
# curl -L "https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-amd64-${CNI_VERSION}.tgz" | tar -C /opt/cni/bin -xz

9.crictlのインストール

# CRICTL_VERSION="v1.12.0"
# curl -L "https://github.com/kubernetes-incubator/cri-tools/releases/download/${CRICTL_VERSION}/crictl-${CRICTL_VERSION}-linux-amd64.tar.gz" | tar -C /usr/bin -xz

9.kubernetesのインストール

9.1.インストール

# RELEASE="$(curl -sSL https://dl.k8s.io/release/stable.txt)"

# cd /usr/bin
# curl -L --remote-name-all https://storage.googleapis.com/kubernetes-release/release/${RELEASE}/bin/linux/amd64/{kubeadm,kubelet,kubectl}
# chmod +x {kubeadm,kubelet,kubectl}

# mkdir -p /etc/systemd/system/kubelet.service.d
# curl -sSL "https://raw.githubusercontent.com/kubernetes/kubernetes/${RELEASE}/build/debs/kubelet.service" > /etc/systemd/system/kubelet.service
# curl -sSL "https://raw.githubusercontent.com/kubernetes/kubernetes/${RELEASE}/build/debs/10-kubeadm.conf" > /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

9.2.cgroup-driver設定

# cat << EOF > /etc/default/kubelet
KUBELET_EXTRA_ARGS="--cgroup-driver=cgroupfs"
EOF

9.3.system unit作成

# cat <<EOF > /etc/systemd/system/kubelet.service
[Unit]
Description=kubelet: The Kubernetes Node Agent
Documentation=http://kubernetes.io/docs/

[Service]
ExecStart=/usr/bin/kubelet
Restart=always
StartLimitInterval=0
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

# mkdir -p /etc/systemd/system/kubelet.service.d
# cat << EOF > /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
EnvironmentFile=-/etc/default/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet \$KUBELET_KUBECONFIG_ARGS \$KUBELET_CONFIG_ARGS \$KUBELET_KUBEADM_ARGS \$KUBELET_EXTRA_ARGS
EOF

9.4.起動

# systemctl daemon-reload
# systemctl enable kubelet
# systemctl restart kubelet

10.初期化

# kubeadm init --pod-network-cidr=10.244.0.0/16 --service-cidr=10.0.0.0/16

11.config設定

$ mkdir $HOME/.kube/
$ sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
$ chmod 600 $HOME/.kube/config

12.CNI(kube-routercalico)のインストール

# kubectl apply -f https://raw.githubusercontent.com/cloudnativelabs/kube-router/master/daemonset/kubeadm-kuberouter.yaml
# kubectl apply -f https://raw.githubusercontent.com/cloudnativelabs/kube-router/master/daemonset/kubeadm-kuberouter-all-features.yaml

# export KUBECONFIG=/etc/kubernetes/admin.conf
# curl -L https://docs.projectcalico.org/manifests/calico.yaml | \
sed  '/            - name: CALICO_DISABLE_FILE_LOGGING/i\            # ADD' | \
sed  '/            - name: CALICO_DISABLE_FILE_LOGGING/i\            - name: FELIX_IPTABLESBACKEND' | \
sed  '/            - name: CALICO_DISABLE_FILE_LOGGING/i\              value: Auto'  | \
sed  '/            - name: CALICO_DISABLE_FILE_LOGGING/i\            # ADD' | \
sed  '/            - name: CALICO_DISABLE_FILE_LOGGING/i\            - name: CALICO_IPV4POOL_CIDR' | \
sed  '/            - name: CALICO_DISABLE_FILE_LOGGING/i\              value: \"10.244.0.0\/16\"' | \
kubectl apply -f -

13.おまけ

13.1.マスターノードへのデプロイを有効化

# kubectl taint nodes --all node-role.kubernetes.io/master-

13.2.kubectlでの状態表示

[admin@PC000002 ~]$ kubectl get node -o wide
NAME                    STATUS   ROLES    AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION            CONTAINER-RUNTIME
pc000002.local.domain   Ready    master   20h   v1.16.3   192.168.1.239   <none>        CentOS Linux 8 (Core)   4.18.0-147.6.el8.x86_64   docker://19.3.5
[admin@PC000002 ~]$
[admin@PC000002 ~]$
[admin@PC000002 ~]$
[admin@PC000002 ~]$
[admin@PC000002 ~]$ kubectl get all -A  -o wide
NAMESPACE     NAME                                                READY   STATUS    RESTARTS   AGE   IP              NODE                    NOMINATED NODE   READINESS GATES
kube-system   pod/coredns-5644d7b6d9-f9pcc                        1/1     Running   0          20h   10.244.0.23     pc000002.local.domain   <none>           <none>
kube-system   pod/coredns-5644d7b6d9-n2db9                        1/1     Running   0          20h   10.244.0.24     pc000002.local.domain   <none>           <none>
kube-system   pod/etcd-pc000002.local.domain                      1/1     Running   0          20h   192.168.1.239   pc000002.local.domain   <none>           <none>
kube-system   pod/kube-apiserver-pc000002.local.domain            1/1     Running   0          20h   192.168.1.239   pc000002.local.domain   <none>           <none>
kube-system   pod/kube-controller-manager-pc000002.local.domain   1/1     Running   0          20h   192.168.1.239   pc000002.local.domain   <none>           <none>
kube-system   pod/kube-proxy-ldxbh                                1/1     Running   0          20h   192.168.1.239   pc000002.local.domain   <none>           <none>
kube-system   pod/kube-router-2mf22                               1/1     Running   0          20h   192.168.1.239   pc000002.local.domain   <none>           <none>
kube-system   pod/kube-scheduler-pc000002.local.domain            1/1     Running   0          20h   192.168.1.239   pc000002.local.domain   <none>           <none>

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE   SELECTOR
default       service/kubernetes   ClusterIP   10.0.0.1     <none>        443/TCP                  20h   <none>
kube-system   service/kube-dns     ClusterIP   10.0.0.10    <none>        53/UDP,53/TCP,9153/TCP   20h   k8s-app=kube-dns

NAMESPACE     NAME                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE   CONTAINERS    IMAGES                                  SELECTOR
kube-system   daemonset.apps/kube-proxy    1         1         1       1            1           beta.kubernetes.io/os=linux   20h   kube-proxy    k8s.gcr.io/kube-proxy:v1.16.3           k8s-app=kube-proxy
kube-system   daemonset.apps/kube-router   1         1         1       1            1           <none>                        20h   kube-router   docker.io/cloudnativelabs/kube-router   k8s-app=kube-router,tier=node

NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                     SELECTOR
kube-system   deployment.apps/coredns   2/2     2            2           20h   coredns      k8s.gcr.io/coredns:1.6.2   k8s-app=kube-dns

NAMESPACE     NAME                                 DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES                     SELECTOR
kube-system   replicaset.apps/coredns-5644d7b6d9   2         2         2       20h   coredns      k8s.gcr.io/coredns:1.6.2   k8s-app=kube-dns,pod-template-hash=5644d7b6d9
[admin@PC000002 ~]$

参考

Install Docker Engine - Community from binarie
Installing kubeadm - Kubernetes