これは2019/09/18時点での話です
ラズパイでkubernetesクラスターを作るという記事がググればわんさか出るこのご時世に、この記事がどれほど役に立つかは不明だが、まぁとにかく記録は残しておくことにする。
ちょいちょい地味なトラブルがあったし。
今回の内容について
タイトル通りRaspberryPiを1個つかってkubernetesを入れるだけの話。
最終的にはラズパイ3個でクラスタ構築する予定だが、現在手元にあるのはRaspberry Pi 3 Model Bが1個のみなので、これを使ってまずはシングルノード構築。
ラズパイ残り2個＋構築資材もろもろは発注済みなので2～3日以内には来るんじゃないかしら。
資材がきたら3台クラスタ構築を頑張る。
続編
- 3台クラスタ構築の手順はこちら https://qiita.com/yyojiro/items/bc8c5ef9a019a9844347
- dashboardの導入はこちら https://qiita.com/yyojiro/items/527188bb24783f9376b2
- metrics-serverの導入はこちら https://qiita.com/yyojiro/items/febfaeadabd2fe8eed08
構築手順
OSインストール
NOOBSからRaspbian Buster Liteをインストールした。
NOOBSはこの辺から拾ってきてMicroSDカードにコピーする。
余談だが、MicroSDカードもお古だったのでなんか余分なパーティションがあったりしたため、削除しまくってFAT32で初期化してからNOOBS書き込んだ。
参考 https://qiita.com/alter095/items/799c13636acbe72a3a3c
MicroSDカードをラズパイにさして電源入れればNOOBSの画面が出てくるのでRaspbian Liteを選択してインストール。（インターネットがつながる環境で実施すること）
OSのインストールが終わったら下記でログインできるはず。
- user: pi
- password: raspberry
基本設定
sshが起動していないので起こしてやる。
pi@raspberry:~$ sudo systemctl enable ssh
pi@raspberry:~$ sudo systemctl start ssh
ノード名を変えたいので下記2つの内容をいじる。
- /etc/hostname
- /etc/hosts
今回はk8s01というノード名に変更してreboot。
パッケージのインストール
sudoするのもめんどくさいからrootでガンガンいく。
root@k8s01:~# apt-get install apt-transport-https ca-certificates curl software-properties-common
root@k8s01:~# curl -fsSL https://download.docker.com/linux/$(. /etc/os-release; echo "$ID")/gpg | sudo apt-key add -
root@k8s01:~# echo "deb [arch=armhf] https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") \
$(lsb_release -cs) stable" | \
sudo tee /etc/apt/sources.list.d/docker.list
root@k8s01:~# apt-get update
root@k8s01:~# apt-get install docker-ce
はい、ここでトラブル発生。
下記のようなエラーでdockerのインストールが失敗。
dpkg: error processing package aufs-dkms (--configure):
installed aufs-dkms package post-installation script subprocess returned error exit status 10
こちらの記事を参考に下記コマンドで対処。
root@k8s01:~# rm /var/lib/dpkg/info/aufs-dkms.postinst
root@k8s01:~# rm /var/lib/dpkg/info/aufs-dkms.prerm
root@k8s01:~# dpkg --configure aufs-dkms
kubernetes用パッケージインストール。
root@k8s01:~# curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg|sudo apt-key add -
root@k8s01:~# echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kube.list
root@k8s01:~# apt-get update
root@k8s01:~# apt-get install -y kubelet kubeadm kubectl
kubernetesセットアップ
kubeadmでMasterノードを構築する。
下記の要素で構成する。
| pod間ネットワーク | flannel |
| LoadBalancer | MetalLB |
root@k8s01:~# sudo kubeadm init --pod-network-cidr=10.244.0.0/16
[init] Using Kubernetes version: v1.15.3
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 19.03.2. Latest validated version: 18.09
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR Swap]: running with swap on is not supported. Please disable swap
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
root@k8s01:~#
はい、トラブル発生。swapがあるとダメだと怒られた。
なのでswapをつぶす。
root@k8s01:~# dphys-swapfile swapoff
root@k8s01:~# dphys-swapfile uninstall
root@k8s01:~# update-rc.d dphys-swapfile remove
root@k8s01:~# systemctl stop dphys-swapfile
root@k8s01:~# systemctl disable dphys-swapfile
気を取り直して再開。
kubeadm initがうまくいくと以下のような感じのアウトプットが出る。
root@k8s01:~# sudo kubeadm init --pod-network-cidr=10.244.0.0/16
[init] Using Kubernetes version: v1.15.3
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 19.03.2. Latest validated version: 18.09
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s01 localhost] and IPs [192.168.11.10 127.0.0.1 ::1]
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s01 localhost] and IPs [192.168.11.10 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.11.10]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[apiclient] All control plane components are healthy after 110.009699 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.15" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s01 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: arfdpt.kjfnf4acxdxhiibr
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.11.10:6443 --token arfdpt.kjfnf4acxdxhiibr \
--discovery-token-ca-cert-hash sha256:f23570483e84944034e254df3c2132beabbaf9be05b30aea3413a157f716de78
root@k8s01:~#
アウトプットの仰せのままに、下記コマンド実行。
root@k8s01:~# mkdir -p $HOME/.kube
root@k8s01:~# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
root@k8s01:~# sudo chown $(id -u):$(id -g) $HOME/.kube/config
flannelインストール
root@k8s01:~# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
root@k8s01:~# kubectl get pods -n kube-system -o wide | grep flannel
kube-flannel-ds-arm-5qp2x 0/1 Init:0/1 0 33s 192.168.11.10 k8s01 <none> <none>
root@k8s01:~# kubectl describe node
Name: k8s01
Roles: master
Labels: beta.kubernetes.io/arch=arm
beta.kubernetes.io/os=linux
kubernetes.io/arch=arm
kubernetes.io/hostname=k8s01
kubernetes.io/os=linux
node-role.kubernetes.io/master=
Annotations: flannel.alpha.coreos.com/backend-data: {"VtepMAC":"8e:24:7c:fe:82:2f"}
flannel.alpha.coreos.com/backend-type: vxlan
flannel.alpha.coreos.com/kube-subnet-manager: true
flannel.alpha.coreos.com/public-ip: 192.168.11.10
kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Tue, 17 Sep 2019 11:48:46 +0100
Taints: node-role.kubernetes.io/master:NoSchedule
Unschedulable: false
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Tue, 17 Sep 2019 12:25:32 +0100 Tue, 17 Sep 2019 11:48:44 +0100 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Tue, 17 Sep 2019 12:25:32 +0100 Tue, 17 Sep 2019 11:48:44 +0100 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Tue, 17 Sep 2019 12:25:32 +0100 Tue, 17 Sep 2019 11:48:44 +0100 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Tue, 17 Sep 2019 12:25:32 +0100 Tue, 17 Sep 2019 11:54:54 +0100 KubeletReady kubelet is posting ready status. WARNING: CPU hardcapping unsupported
Addresses:
InternalIP: 192.168.11.10
Hostname: k8s01
Capacity:
cpu: 4
ephemeral-storage: 30497732Ki
memory: 948304Ki
pods: 110
Allocatable:
cpu: 4
ephemeral-storage: 28106709765
memory: 845904Ki
pods: 110
System Info:
Machine ID: a260550c5829479ea01ed1c3b9618f4f
System UUID: a260550c5829479ea01ed1c3b9618f4f
Boot ID: d12ad2d0-d18a-46fb-8f31-49db3ade09d9
Kernel Version: 4.19.57-v7+
OS Image: Raspbian GNU/Linux 10 (buster)
Operating System: linux
Architecture: arm
Container Runtime Version: docker://19.3.2
Kubelet Version: v1.15.3
Kube-Proxy Version: v1.15.3
PodCIDR: 10.244.0.0/24
Non-terminated Pods: (8 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system coredns-5c98db65d4-dkdg8 100m (2%) 0 (0%) 70Mi (8%) 170Mi (20%) 37m
kube-system coredns-5c98db65d4-nj4kr 100m (2%) 0 (0%) 70Mi (8%) 170Mi (20%) 37m
kube-system etcd-k8s01 0 (0%) 0 (0%) 0 (0%) 0 (0%) 36m
kube-system kube-apiserver-k8s01 250m (6%) 0 (0%) 0 (0%) 0 (0%) 37m
kube-system kube-controller-manager-k8s01 200m (5%) 0 (0%) 0 (0%) 0 (0%) 37m
kube-system kube-flannel-ds-arm-5qp2x 100m (2%) 100m (2%) 50Mi (6%) 50Mi (6%) 32m
kube-system kube-proxy-x8t59 0 (0%) 0 (0%) 0 (0%) 0 (0%) 37m
kube-system kube-scheduler-k8s01 100m (2%) 0 (0%) 0 (0%) 0 (0%) 37m
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 850m (21%) 100m (2%)
memory 190Mi (23%) 390Mi (47%)
ephemeral-storage 0 (0%) 0 (0%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal NodeHasSufficientPID 38m (x7 over 38m) kubelet, k8s01 Node k8s01 status is now: NodeHasSufficientPID
Normal NodeHasSufficientMemory 38m (x8 over 38m) kubelet, k8s01 Node k8s01 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 38m (x8 over 38m) kubelet, k8s01 Node k8s01 status is now: NodeHasNoDiskPressure
Normal Starting 36m kube-proxy, k8s01 Starting kube-proxy.
Normal NodeReady 31m kubelet, k8s01 Node k8s01 status is now: NodeReady
root@k8s01:~#
Masterノードでpodを動かしたい
kubectl describe nodeで下記のようになっており、このノードではpodがスケジュールされないようになっている。
Taints: node-role.kubernetes.io/master:NoSchedule
まだシングルノードなのでこのノード上でもpodを動かしたいため、下記の呪文で制限を外す。
root@k8s01:~# kubectl taint nodes --all node-role.kubernetes.io/master-
これでkubectl describe nodeで下記のようになるはず。
Taints: <none>
MetalLBインストール
下記参照。
https://metallb.universe.tf/installation/
root@k8s01:~# kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.8.1/manifests/metallb.yaml
root@k8s01:~# kubectl apply -f metallb.config.yaml
root@k8s01:~# kubectl get pod -n metallb-system
NAME READY STATUS RESTARTS AGE
controller-55d74449-zg9wd 1/1 Running 2 14m
speaker-jksdf 1/1 Running 2 14m
metallb.config.yamlの内容は下記。
metallb.config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: default
protocol: layer2
addresses:
- 192.168.11.200-192.168.11.240
address-poolsのところは適当に環境に合わせて書き換える。
動作確認
みんな大好きnginxをデプロイ。下記のようにデプロイメント用ファイル作成。
nginx.deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
replicas: 1
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: nginx-service
spec:
selector:
app: nginx
ports:
- port: 80
targetPort: 80
type: LoadBalancer
deployment作成。
root@k8s01:~# kubectl create -f nginx.deployment.yaml
deployment.apps/nginx-deployment created
service/nginx-service created
root@k8s01:~#
root@k8s01:~# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 3h21m
nginx-service LoadBalancer 10.97.23.9 192.168.11.200 80:30779/TCP 30s
nginx-testに割り振られたEXTERNAL-IPにアクセスしてみる。
root@k8s01:~# curl 192.168.11.200
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
ここまでくれば、外部からブラウザでアクセスしてみても見れるはず。。。
見れるはずだったが、最初はうまくいかなかった。
ブラウザからアクセスする前に、pingか何かでarp解決してみないとMetalLBのspeakerが反応しないっぽい。
arp解決したら外部からブラウザでアクセスしても見えるようになる。
その他トラブル
iptables -Lでiptablesの状態を確認したところ、下記のような設定が大量に作られていた。
target prot opt source destination
DROP all -- anywhere anywhere mark match 0x8000/0x8000 /* kubernetes firewall for dropping marked packets */
DROP all -- anywhere anywhere mark match 0x8000/0x8000 /* kubernetes firewall for dropping marked packets */
DROP all -- anywhere anywhere mark match 0x8000/0x8000 /* kubernetes firewall for dropping marked packets */
DROP all -- anywhere anywhere mark match 0x8000/0x8000 /* kubernetes firewall for dropping marked packets */
DROP all -- anywhere anywhere mark match 0x8000/0x8000 /* kubernetes firewall for dropping marked packets */
（すごい大量に続く）
ググった結果、どうやらiptablesのバグらしい。
https://github.com/kubernetes/kubernetes/issues/82361
iptablesを1.8.3以上にすればいいらしいが、Raspbian Busterだとまだ1.8.2しか拾えないのでワークアラウンド手順を実施。
root@k8s01:~# update-alternatives --set iptables /usr/sbin/iptables-legacy
これをやった後rebootしたらゴミが消えた。めでたしめでたし。
再構築
実は何回か再構築しているので、コマンド打つのめんどいからスクリプトにして再構築していた。
reset.sh
#!/bin/bash
# 全部消す
kubeadm reset
# 初期構築
kubeadm init --pod-network-cidr=10.244.0.0/16
cp -f /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
# flannelインストール
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
# masterでpodを動かす
kubectl taint nodes --all node-role.kubernetes.io/master-
# metallbインストール
kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.8.1/manifests/metallb.yaml
kubectl apply -f metallb.config.yaml
パッケージインストールやワークアラウンド含めてansible化しようか検討中。