TL;DR
containerdとkubeletが利用するcgroups driverの設定が合ってなかったせいで死んでいました。
デフォルト設定だと異なるのでしっかりconfigまで目を通します！！
出たエラー
ラズパイでkubeadm initを実行してK8sのコントロールnodeを作成して、kubectl get nodeでちゃんと表示されるまで確認したが突如kubectlが上手くいかない...
再起動すると少しの間kubectl get xxxが使える奇跡の時間が発生する(この時間を使ってデバッグする)
$ kubectl get pod -A
The connection to the server 192.168.10.49:6443 was refused - did you specify the right host or port?
（kubelet 再起動したら、時々復活してすぐ死ぬ奇跡の時間がある）
$ systemctl restart kubelet
$ kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-6f6b679f8f-7rkfn 0/1 Pending 0 78m
kube-system coredns-6f6b679f8f-mw8lr 0/1 Pending 0 78m
kube-system etcd-control 0/1 Running 256 (30s ago) 79m
kube-system kube-apiserver-control 0/1 Running 250 (10s ago) 79m
kube-system kube-controller-manager-control 0/1 Running 259 (10s ago) 77m
kube-system kube-proxy-nxdjv 0/1 Error 39 (38s ago) 78m
kube-system kube-scheduler-control 0/1 Running 266 (10s ago) 79m
$ kubectl get pod -A
The connection to the server 192.168.10.49:6443 was refused - did you specify the right host or port?
エラーを追う
とりあえずkubeletからkube-apiserver-control を確認
$ jounalctl -u kubelet | grep kube-apiserver
.
.
.
Unable to write event (may retry after sleeping)" err="Post \"https://192.168.10.49:6443/api/v1/namespaces/kube-system/events\": dial tcp 192.168.10.49:6443: connect: connection refused" event="&Event{ObjectMeta:{kube-apiserver-control.17ed1e413f5907c9 kube-system 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-control,UID:a215b048c264f32757b87146b6148bb2,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{kube-apiserver},},Reason:Unhealthy,Message:Readiness probe failed: Get \"https://192.168.10.49:6443/readyz\": dial tcp 192.168.10.49:6443: connect: connection refused,Source:EventSource{Component:kubelet,Host:control,},FirstTimestamp:2024-08-19 20:38:17.937274825 +0900 JST m=+51.754393154,LastTimestamp:2024-08-19 20:38:17.937274825 +0900 JST m=+51.754393154,Count:1,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:control,}"
apiserverが落ちてそうでアクセス出来てない...
kube-apiserver-controlのpodのlogを確認
$ kubectl logs kube-apiserver-control -n kube-system -f
.
.
.
W0819 11:46:40.808045 1 logging.go:55] [core] [Channel #52 SubChannel #53]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
W0819 11:46:40.857145 1 logging.go:55] [core] [Channel #28 SubChannel #29]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
W0819 11:46:40.895323 1 logging.go:55] [core] [Channel #46 SubChannel #47]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
W0819 11:46:40.904756 1 logging.go:55] [core] [Channel #118 SubChannel #119]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
W0819 11:46:40.905834 1 logging.go:55] [core] [Channel #85 SubChannel #86]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
W0819 11:46:40.915920 1 logging.go:55] [core] [Channel #169 SubChannel #170]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
W0819 11:46:40.955245 1 logging.go:55] [core] [Channel #124 SubChannel #125]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
error: http2: server sent GOAWAY and closed the connection; LastStreamID=3, ErrCode=NO_ERROR, debug=""
etcdの通信が切れてるから落ちてるっぽい
etcdのpodを確認してみる
$ kubectl logs etcd-control -n kube-system -f
.
.
.
{"level":"info","ts":"2024-08-19T12:06:35.684472Z","caller":"osutil/interrupt_unix.go:64","msg":"received signal; shutting down","signal":"terminated"}
{"level":"info","ts":"2024-08-19T12:06:35.684601Z","caller":"embed/etcd.go:377","msg":"closing etcd server","name":"control","data-dir":"/var/lib/etcd","advertise-peer-urls":["https://192.168.10.49:2380"],"advertise-client-urls":["https://192.168.10.49:2379"]}
{"level":"warn","ts":"2024-08-19T12:06:35.684930Z","caller":"embed/serve.go:212","msg":"stopping secure grpc server due to error","error":"accept tcp 127.0.0.1:2379: use of closed network connection"}
{"level":"warn","ts":"2024-08-19T12:06:35.685225Z","caller":"embed/serve.go:214","msg":"stopped secure grpc server due to error","error":"accept tcp 127.0.0.1:2379: use of closed network connection"}
{"level":"warn","ts":"2024-08-19T12:06:35.734420Z","caller":"embed/serve.go:212","msg":"stopping secure grpc server due to error","error":"accept tcp 192.168.10.49:2379: use of closed network connection"}
{"level":"warn","ts":"2024-08-19T12:06:35.734556Z","caller":"embed/serve.go:214","msg":"stopped secure grpc server due to error","error":"accept tcp 192.168.10.49:2379: use of closed network connection"}
{"level":"info","ts":"2024-08-19T12:06:35.734741Z","caller":"etcdserver/server.go:1521","msg":"skipped leadership transfer for single voting member cluster","local-member-id":"2775615e805387bb","current-leader-member-id":"2775615e805387bb"}
{"level":"info","ts":"2024-08-19T12:06:35.747342Z","caller":"embed/etcd.go:581","msg":"stopping serving peer traffic","address":"192.168.10.49:2380"}
{"level":"info","ts":"2024-08-19T12:06:35.747782Z","caller":"embed/etcd.go:586","msg":"stopped serving peer traffic","address":"192.168.10.49:2380"}
{"level":"info","ts":"2024-08-19T12:06:35.748327Z","caller":"embed/etcd.go:379","msg":"closed etcd server","name":"control","data-dir":"/var/lib/etcd","advertise-peer-urls":["https://192.168.10.49:2380"],"advertise-client-urls":["https://192.168.10.49:2379"]}
外部(きっとkubelet)からSIGTERM 来てるやん...どういうこっちゃ
整理
kubelet何しているの...
etcdにsigtermを流すlogで検索
"msg":"received signal; shutting down","signal":"terminated"で検索すると似たような質問がいっぱい
コミュニティの方で面白い記事があったので読む
In the past few days, I've been trying to deploy a Kubernetes cluster using kubeadm in Debian. After kubeadm configured the control plane, something was killing the pods randomly after a while. This post follows the story of how I tried to fix this issue.
これだ！！
記事の要約
問題が発生する原因
containerdとkubeletが利用するcgroups driverの設定が異なっているせいでした。
デフォルト設定では、containerdがcgroupfs、kubectlがsystemdと設定されているため発生していた。
(dockerを利用している場合は、systemdに設定されるから影響が出なかった)
ベストな解決策
contaienrdの設定をsystemdに設定する
#ファイル /etc/containerd/config.toml の内容
version = 2
[ plugins ]
[plugins. "io.containerd.grpc.v1.cri" ]
[plugins. "io.containerd.grpc.v1.cri" .containerd]
[plugins. "io.containerd.grpc.v1.cri" .containerd.runtimes]
[plugins. "io.containerd.grpc.v1.cri" .containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"
[plugins. "io.containerd.grpc.v1.cri" .containerd.runtimes.runc.options]
SystemdCgroup = true
問題が発生する流れ
詳細は記事を見て下さい
- containerdはcgroupfsを用いて、コンテナのcgroupを作成
- kubeletは、QoS cgroupのCPUWeightプロパティを変更するために、systemdにD-Busメッセージを送信(毎分行っている)
- systemdは、cpu.weightファイルに書き込む
- systemdが理由不明でcpusetコントローラーを削除（sysmtedが作成してないcgroupだから削除したんだと思う）
- kubeletは、Podの同期を試みてコントローラーが欠如していること気付く
- kubeletは、cgroupが「消えた」と判断してPodを終了
- kubeletはQoS cgroupを再同期し、ファイルAPIを介してcpusetを追加
- Podは再起動するが、cgroupが破壊されているので動かない
自分が経験した事象に照らし合わせてみる
再起動したらkubectl get xxxができた奇跡の時間
コンテナ作成直後からkubeletがsystemdにD-Busメッセージを送信してsystemdの処理が完了するまでは、cgroupの値は正常であっため、この奇跡の時間が生まれた。
kubeletがetcdにSIGTERMを送るあの処理
kubeletは、cgroupのコントローラーが欠如していることに気づいて、「消えた」と判断してPodを終了する為にSIGTERMを送っていた
設定を変更後
ちゃんと2時間動くことを確認
$ kubectl get pod -A
kube-system coredns-6f6b679f8f-7rkfn 0/1 Pending 0 4h4m
kube-system coredns-6f6b679f8f-mw8lr 0/1 Pending 0 4h4m
kube-system etcd-control 1/1 Running 265 (133m ago) 4h5m
kube-system kube-apiserver-control 1/1 Running 257 (135m ago) 4h5m
kube-system kube-controller-manager-control 1/1 Running 269 (134m ago) 4h3m
kube-system kube-proxy-nxdjv 1/1 Running 50 (128m ago) 4h4m
kube-system kube-scheduler-control 1/1 Running 276 (130m ago) 4h5m