少し背景
ずっと前からkubernetes podsのCPUやメモリの利用率のモニタリング方法に興味を持った時があって、ちょうど今日Kubernetesの本でkubernetes dashboardというWeb UIベースのモニタリング方法を見まして、試してみようとデプロイしてみましたが、本が古かったかもしれませんが、書かれている内容が既に存在せず、HPページの手順で行いましたが、罠だらけでした。その罠たちと戦ってきた内容をメモしたいと思います。
まず公式ページの手順を挑戦したい方
以下のリンクより試してみてください。もしかするとOSが違うと公式サイトの手順は普通に行けるかも?
問題遭遇
helm install
してreleaseを作成してあとで、早速Podsがエラーになりました
NAME READY STATUS RESTARTS AGE
pod/kubernetes-dashboard-api-57886465b-zbtcm 1/1 Running 0 19m
pod/kubernetes-dashboard-auth-59d6f87f85-66gg9 1/1 Running 0 19m
pod/kubernetes-dashboard-kong-76dff7b666-vgksj 0/1 CrashLoopBackOff 10 (75s ago) 27m
pod/kubernetes-dashboard-metrics-scraper-8d8778c4-zcwh5 1/1 Running 0 27m
pod/kubernetes-dashboard-web-7d77999479-2s6m5 1/1 Running 0 27m
皆さんも一緒にTroubleShootingしてみましょう
それぞれクリックすると展開されます
kubectl logs error
Defaulted container "proxy" out of: proxy, clear-stale-pid (init)
Error: could not prepare Kong prefix at /kong_prefix: nginx configuration is invalid (exit code 1):
nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /kong_prefix/nginx.conf:7
nginx: the configuration file /kong_prefix/nginx.conf syntax is ok
nginx: [emerg] bind() to [::1]:8444 failed (99: Cannot assign requested address)
nginx: configuration file /kong_prefix/nginx.conf test failed
Run with --v (verbose) or --vv (debug) for more details
kubectl describe pod
Name: kubernetes-dashboard-kong-76dff7b666-vgksj
Namespace: kubernetes-dashboard
Priority: 0
Service Account: kubernetes-dashboard-kong
Node: docker-desktop/192.168.65.3
Start Time: Fri, 19 Apr 2024 20:39:29 +0900
Labels: app=kubernetes-dashboard-kong
app.kubernetes.io/component=app
app.kubernetes.io/instance=kubernetes-dashboard
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=kong
app.kubernetes.io/version=3.6
helm.sh/chart=kong-2.38.0
pod-template-hash=76dff7b666
version=3.6
Annotations: kuma.io/gateway: enabled
kuma.io/service-account-token-volume: kubernetes-dashboard-kong-token
traffic.sidecar.istio.io/includeInboundPorts:
Status: Running
IP: 10.1.22.67
IPs:
IP: 10.1.22.67
Controlled By: ReplicaSet/kubernetes-dashboard-kong-76dff7b666
Init Containers:
clear-stale-pid:
Container ID: docker://3f3a67c09f710b479c7e724ea8a8f34b0eba6e9115bb6666fc06b2da9ddcf37f
Image: kong:3.6
Image ID: docker-pullable://kong@sha256:3fb1e1134180999b83745d48e24b840d34abee6a1b438f9431fbaf033c34562a
Port: <none>
Host Port: <none>
SeccompProfile: RuntimeDefault
Command:
rm
-vrf
$KONG_PREFIX/pids
State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 19 Apr 2024 20:39:42 +0900
Finished: Fri, 19 Apr 2024 20:39:42 +0900
Ready: True
Restart Count: 0
Environment:
KONG_ADMIN_ACCESS_LOG: /dev/stdout
KONG_ADMIN_ERROR_LOG: /dev/stderr
KONG_ADMIN_GUI_ACCESS_LOG: /dev/stdout
KONG_ADMIN_GUI_ERROR_LOG: /dev/stderr
KONG_ADMIN_LISTEN: 127.0.0.1:8444 http2 ssl, [::1]:8444 http2 ssl
KONG_CLUSTER_LISTEN: off
KONG_DATABASE: off
KONG_DECLARATIVE_CONFIG: /kong_dbless/kong.yml
KONG_DNS_ORDER: LAST,A,CNAME,AAAA,SRV
KONG_LUA_PACKAGE_PATH: /opt/?.lua;/opt/?/init.lua;;
KONG_NGINX_WORKER_PROCESSES: 1
KONG_PLUGINS: off
KONG_PORTAL_API_ACCESS_LOG: /dev/stdout
KONG_PORTAL_API_ERROR_LOG: /dev/stderr
KONG_PORT_MAPS: 443:8443
KONG_PREFIX: /kong_prefix/
KONG_PROXY_ACCESS_LOG: /dev/stdout
KONG_PROXY_ERROR_LOG: /dev/stderr
KONG_PROXY_LISTEN: 0.0.0.0:8443 http2 ssl, [::]:8443 http2 ssl
KONG_PROXY_STREAM_ACCESS_LOG: /dev/stdout basic
KONG_PROXY_STREAM_ERROR_LOG: /dev/stderr
KONG_ROUTER_FLAVOR: traditional
KONG_STATUS_ACCESS_LOG: off
KONG_STATUS_ERROR_LOG: /dev/stderr
KONG_STATUS_LISTEN: 0.0.0.0:8100, [::]:8100
KONG_STREAM_LISTEN: off
Mounts:
/kong_dbless/ from kong-custom-dbless-config-volume (rw)
/kong_prefix/ from kubernetes-dashboard-kong-prefix-dir (rw)
/tmp from kubernetes-dashboard-kong-tmp (rw)
Containers:
proxy:
Container ID: docker://908103eb9397495e51273249c21aecd41ebdb16485c8ef16552661c908e420b1
Image: kong:3.6
Image ID: docker-pullable://kong@sha256:3fb1e1134180999b83745d48e24b840d34abee6a1b438f9431fbaf033c34562a
Ports: 8443/TCP, 8100/TCP
Host Ports: 0/TCP, 0/TCP
SeccompProfile: RuntimeDefault
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Fri, 19 Apr 2024 20:50:27 +0900
Finished: Fri, 19 Apr 2024 20:50:28 +0900
Ready: False
Restart Count: 7
Liveness: http-get http://:status/status delay=5s timeout=5s period=10s #success=1 #failure=3
Readiness: http-get http://:status/status/ready delay=5s timeout=5s period=10s #success=1 #failure=3
Environment:
KONG_ADMIN_ACCESS_LOG: /dev/stdout
KONG_ADMIN_ERROR_LOG: /dev/stderr
KONG_ADMIN_GUI_ACCESS_LOG: /dev/stdout
KONG_ADMIN_GUI_ERROR_LOG: /dev/stderr
KONG_ADMIN_LISTEN: 127.0.0.1:8444 http2 ssl, [::1]:8444 http2 ssl
KONG_CLUSTER_LISTEN: off
KONG_DATABASE: off
KONG_DECLARATIVE_CONFIG: /kong_dbless/kong.yml
KONG_DNS_ORDER: LAST,A,CNAME,AAAA,SRV
KONG_LUA_PACKAGE_PATH: /opt/?.lua;/opt/?/init.lua;;
KONG_NGINX_WORKER_PROCESSES: 1
KONG_PLUGINS: off
KONG_PORTAL_API_ACCESS_LOG: /dev/stdout
KONG_PORTAL_API_ERROR_LOG: /dev/stderr
KONG_PORT_MAPS: 443:8443
KONG_PREFIX: /kong_prefix/
KONG_PROXY_ACCESS_LOG: /dev/stdout
KONG_PROXY_ERROR_LOG: /dev/stderr
KONG_PROXY_LISTEN: 0.0.0.0:8443 http2 ssl, [::]:8443 http2 ssl
KONG_PROXY_STREAM_ACCESS_LOG: /dev/stdout basic
KONG_PROXY_STREAM_ERROR_LOG: /dev/stderr
KONG_ROUTER_FLAVOR: traditional
KONG_STATUS_ACCESS_LOG: off
KONG_STATUS_ERROR_LOG: /dev/stderr
KONG_STATUS_LISTEN: 0.0.0.0:8100, [::]:8100
KONG_STREAM_LISTEN: off
KONG_NGINX_DAEMON: off
Mounts:
/kong_dbless/ from kong-custom-dbless-config-volume (rw)
/kong_prefix/ from kubernetes-dashboard-kong-prefix-dir (rw)
/tmp from kubernetes-dashboard-kong-tmp (rw)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kubernetes-dashboard-kong-prefix-dir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: 256Mi
kubernetes-dashboard-kong-tmp:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: 1Gi
kubernetes-dashboard-kong-token:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
kong-custom-dbless-config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: kong-dbless-config
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 15m default-scheduler Successfully assigned kubernetes-dashboard/kubernetes-dashboard-kong-76dff7b666-vgksj to docker-desktop
Normal Pulling 15m kubelet Pulling image "kong:3.6"
Normal Pulled 15m kubelet Successfully pulled image "kong:3.6" in 8.219s (12.216s including waiting)
Normal Created 15m kubelet Created container clear-stale-pid
Normal Started 15m kubelet Started container clear-stale-pid
Normal Pulled 14m (x4 over 15m) kubelet Container image "kong:3.6" already present on machine
Normal Created 14m (x4 over 15m) kubelet Created container proxy
Normal Started 14m (x4 over 15m) kubelet Started container proxy
Warning BackOff 34s (x82 over 15m) kubelet Back-off restarting failed container proxy in pod kubernetes-dashboard-kong-76dff7b666-vgksj_kubernetes-dashboard(55c8aa3a-6789-4e28-9128-c8ccf03e2fed)
kubectl get svc -n kubernetes-dashboard
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes-dashboard-api ClusterIP 10.105.177.157 <none> 8000/TCP 38m
kubernetes-dashboard-auth ClusterIP 10.111.138.36 <none> 8000/TCP 38m
kubernetes-dashboard-kong-manager NodePort 10.109.157.200 <none> 8002:31720/TCP,8445:30344/TCP 38m
kubernetes-dashboard-kong-proxy ClusterIP 10.96.140.104 <none> 443/TCP 38m
kubernetes-dashboard-metrics-scraper ClusterIP 10.99.36.212 <none> 8000/TCP 38m
kubernetes-dashboard-web ClusterIP 10.103.69.54 <none> 8000/TCP 38m
苦戦
今回の主なエラーはkongで発生しており、更にエラー文を見るとnginx: [emerg] bind() to [::1]:8444 failed (99: Cannot assign requested address)
が書かれているため、IPv6関連しそうな気がしました。
一方で、helm valuesを確認してみると、特にIP関連の属性がありませんでした。以下コード抜粋
設定のところも見てみましたが、似ているものが特にありませんでした。
## Required Kong sub-chart with DBless configuration to act as a gateway
## for our all containers.
kong:
enabled: true
## Configuration reference: https://docs.konghq.com/gateway/3.6.x/reference/configuration
env:
dns_order: LAST,A,CNAME,AAAA,SRV
plugins: 'off'
nginx_worker_processes: 1
ingressController:
enabled: false
dblessConfig:
configMap: kong-dbless-config
proxy:
type: ClusterIP
http:
enabled: false
また、GPTにも訪ねてみましたが、以下の内容をファイルにして、helm upgrade -f myvalue.yaml
みないに試ししてくださいという解決方法を提案してくれた。
env:
proxy:
KONG_ADMIN_LISTEN: "127.0.0.1:8444 http2 ssl"
KONG_PROXY_LISTEN: "0.0.0.0:8443 http2 ssl"
もちろん試してみましたが、全く効果がありませんでした。この時気付いたのは、そもそも公式のvalues.yamlにはenv.proxyなどは存在していなく、代わりにkong.envに環境変数の設定をしているみたいなので、試すことにしました。
晴れ
案の定、以下のようにカスタマイズのvaluesを修正して、helm upgrade
したらうまく起動できました!感動。
kong:
env:
admin_listen: '127.0.0.1:8444 http2 ssl'
proxy_listen: '0.0.0.0:8443 http2 ssl'
kubectl get pods --watch
結果
kubernetes-dashboard-kong-76dff7b666-vgksj 0/1 Error 15 (5m11s ago) 52m
kubernetes-dashboard-kong-76dff7b666-vgksj 0/1 CrashLoopBackOff 15 (1s ago) 52m
kubernetes-dashboard-kong-67c657f866-t82cr 0/1 Pending 0 0s
kubernetes-dashboard-kong-67c657f866-t82cr 0/1 Pending 0 0s
kubernetes-dashboard-kong-67c657f866-t82cr 0/1 Init:0/1 0 0s
kubernetes-dashboard-kong-67c657f866-t82cr 0/1 PodInitializing 0 1s
kubernetes-dashboard-kong-67c657f866-t82cr 0/1 Running 0 2s
kubernetes-dashboard-kong-67c657f866-t82cr 1/1 Running 0 10s
kubernetes-dashboard-kong-76dff7b666-vgksj 0/1 Terminating 15 (112s ago) 54m
kubernetes-dashboard-kong-76dff7b666-vgksj 0/1 Terminating 15 54m
kubernetes-dashboard-kong-76dff7b666-vgksj 0/1 Terminating 15 54m
kubernetes-dashboard-kong-76dff7b666-vgksj 0/1 Terminating 15 54m
kubernetes-dashboard-kong-76dff7b666-vgksj 0/1 Terminating 15 54m
kubernetes-dashboard-kong-76dff7b666-vgksj 0/1 Terminating 15 54m
あとは簡単でした。まずsample userを作成して、以下のコマンドでport forwardをした上で、https://localhost:8443/#/login
でログインすれば画面が出てきます。
kubectl -n kubernetes-dashboard port-forward svc/kubernetes-dashboard-kong-proxy 8443:443
最後に
本には数ページの内容でしたが、実際にやってみると手順が間違っていたり、古かったりなどがよくありますので、TroubleShootingをしながら最後に解けるのが面白く、何がどこにあるかもより理解深めた気がします。