少し背景

ずっと前からkubernetes podsのCPUやメモリの利用率のモニタリング方法に興味を持った時があって、ちょうど今日Kubernetesの本でkubernetes dashboardというWeb UIベースのモニタリング方法を見まして、試してみようとデプロイしてみましたが、本が古かったかもしれませんが、書かれている内容が既に存在せず、HPページの手順で行いましたが、罠だらけでした。その罠たちと戦ってきた内容をメモしたいと思います。

まず公式ページの手順を挑戦したい方

以下のリンクより試してみてください。もしかするとOSが違うと公式サイトの手順は普通に行けるかも？

問題遭遇

helm installしてreleaseを作成してあとで、早速Podsがエラーになりました

NAME                                                      READY   STATUS             RESTARTS       AGE
pod/kubernetes-dashboard-api-57886465b-zbtcm              1/1     Running            0              19m
pod/kubernetes-dashboard-auth-59d6f87f85-66gg9            1/1     Running            0              19m
pod/kubernetes-dashboard-kong-76dff7b666-vgksj            0/1     CrashLoopBackOff   10 (75s ago)   27m
pod/kubernetes-dashboard-metrics-scraper-8d8778c4-zcwh5   1/1     Running            0              27m
pod/kubernetes-dashboard-web-7d77999479-2s6m5             1/1     Running            0              27m

皆さんも一緒にTroubleShootingしてみましょう

それぞれクリックすると展開されます

kubectl logs error

Defaulted container "proxy" out of: proxy, clear-stale-pid (init)
Error: could not prepare Kong prefix at /kong_prefix: nginx configuration is invalid (exit code 1):
nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /kong_prefix/nginx.conf:7
nginx: the configuration file /kong_prefix/nginx.conf syntax is ok
nginx: [emerg] bind() to [::1]:8444 failed (99: Cannot assign requested address)
nginx: configuration file /kong_prefix/nginx.conf test failed


  Run with --v (verbose) or --vv (debug) for more details

kubectl describe pod

Name:             kubernetes-dashboard-kong-76dff7b666-vgksj
Namespace:        kubernetes-dashboard
Priority:         0
Service Account:  kubernetes-dashboard-kong
Node:             docker-desktop/192.168.65.3
Start Time:       Fri, 19 Apr 2024 20:39:29 +0900
Labels:           app=kubernetes-dashboard-kong
                  app.kubernetes.io/component=app
                  app.kubernetes.io/instance=kubernetes-dashboard
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=kong
                  app.kubernetes.io/version=3.6
                  helm.sh/chart=kong-2.38.0
                  pod-template-hash=76dff7b666
                  version=3.6
Annotations:      kuma.io/gateway: enabled
                  kuma.io/service-account-token-volume: kubernetes-dashboard-kong-token
                  traffic.sidecar.istio.io/includeInboundPorts:
Status:           Running
IP:               10.1.22.67
IPs:
  IP:           10.1.22.67
Controlled By:  ReplicaSet/kubernetes-dashboard-kong-76dff7b666
Init Containers:
  clear-stale-pid:
    Container ID:    docker://3f3a67c09f710b479c7e724ea8a8f34b0eba6e9115bb6666fc06b2da9ddcf37f
    Image:           kong:3.6
    Image ID:        docker-pullable://kong@sha256:3fb1e1134180999b83745d48e24b840d34abee6a1b438f9431fbaf033c34562a
    Port:            <none>
    Host Port:       <none>
    SeccompProfile:  RuntimeDefault
    Command:
      rm
      -vrf
      $KONG_PREFIX/pids
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 19 Apr 2024 20:39:42 +0900
      Finished:     Fri, 19 Apr 2024 20:39:42 +0900
    Ready:          True
    Restart Count:  0
    Environment:
      KONG_ADMIN_ACCESS_LOG:         /dev/stdout
      KONG_ADMIN_ERROR_LOG:          /dev/stderr
      KONG_ADMIN_GUI_ACCESS_LOG:     /dev/stdout
      KONG_ADMIN_GUI_ERROR_LOG:      /dev/stderr
      KONG_ADMIN_LISTEN:             127.0.0.1:8444 http2 ssl, [::1]:8444 http2 ssl
      KONG_CLUSTER_LISTEN:           off
      KONG_DATABASE:                 off
      KONG_DECLARATIVE_CONFIG:       /kong_dbless/kong.yml
      KONG_DNS_ORDER:                LAST,A,CNAME,AAAA,SRV
      KONG_LUA_PACKAGE_PATH:         /opt/?.lua;/opt/?/init.lua;;
      KONG_NGINX_WORKER_PROCESSES:   1
      KONG_PLUGINS:                  off
      KONG_PORTAL_API_ACCESS_LOG:    /dev/stdout
      KONG_PORTAL_API_ERROR_LOG:     /dev/stderr
      KONG_PORT_MAPS:                443:8443
      KONG_PREFIX:                   /kong_prefix/
      KONG_PROXY_ACCESS_LOG:         /dev/stdout
      KONG_PROXY_ERROR_LOG:          /dev/stderr
      KONG_PROXY_LISTEN:             0.0.0.0:8443 http2 ssl, [::]:8443 http2 ssl
      KONG_PROXY_STREAM_ACCESS_LOG:  /dev/stdout basic
      KONG_PROXY_STREAM_ERROR_LOG:   /dev/stderr
      KONG_ROUTER_FLAVOR:            traditional
      KONG_STATUS_ACCESS_LOG:        off
      KONG_STATUS_ERROR_LOG:         /dev/stderr
      KONG_STATUS_LISTEN:            0.0.0.0:8100, [::]:8100
      KONG_STREAM_LISTEN:            off
    Mounts:
      /kong_dbless/ from kong-custom-dbless-config-volume (rw)
      /kong_prefix/ from kubernetes-dashboard-kong-prefix-dir (rw)
      /tmp from kubernetes-dashboard-kong-tmp (rw)
Containers:
  proxy:
    Container ID:    docker://908103eb9397495e51273249c21aecd41ebdb16485c8ef16552661c908e420b1
    Image:           kong:3.6
    Image ID:        docker-pullable://kong@sha256:3fb1e1134180999b83745d48e24b840d34abee6a1b438f9431fbaf033c34562a
    Ports:           8443/TCP, 8100/TCP
    Host Ports:      0/TCP, 0/TCP
    SeccompProfile:  RuntimeDefault
    State:           Waiting
      Reason:        CrashLoopBackOff
    Last State:      Terminated
      Reason:        Error
      Exit Code:     1
      Started:       Fri, 19 Apr 2024 20:50:27 +0900
      Finished:      Fri, 19 Apr 2024 20:50:28 +0900
    Ready:           False
    Restart Count:   7
    Liveness:        http-get http://:status/status delay=5s timeout=5s period=10s #success=1 #failure=3
    Readiness:       http-get http://:status/status/ready delay=5s timeout=5s period=10s #success=1 #failure=3
    Environment:
      KONG_ADMIN_ACCESS_LOG:         /dev/stdout
      KONG_ADMIN_ERROR_LOG:          /dev/stderr
      KONG_ADMIN_GUI_ACCESS_LOG:     /dev/stdout
      KONG_ADMIN_GUI_ERROR_LOG:      /dev/stderr
      KONG_ADMIN_LISTEN:             127.0.0.1:8444 http2 ssl, [::1]:8444 http2 ssl
      KONG_CLUSTER_LISTEN:           off
      KONG_DATABASE:                 off
      KONG_DECLARATIVE_CONFIG:       /kong_dbless/kong.yml
      KONG_DNS_ORDER:                LAST,A,CNAME,AAAA,SRV
      KONG_LUA_PACKAGE_PATH:         /opt/?.lua;/opt/?/init.lua;;
      KONG_NGINX_WORKER_PROCESSES:   1
      KONG_PLUGINS:                  off
      KONG_PORTAL_API_ACCESS_LOG:    /dev/stdout
      KONG_PORTAL_API_ERROR_LOG:     /dev/stderr
      KONG_PORT_MAPS:                443:8443
      KONG_PREFIX:                   /kong_prefix/
      KONG_PROXY_ACCESS_LOG:         /dev/stdout
      KONG_PROXY_ERROR_LOG:          /dev/stderr
      KONG_PROXY_LISTEN:             0.0.0.0:8443 http2 ssl, [::]:8443 http2 ssl
      KONG_PROXY_STREAM_ACCESS_LOG:  /dev/stdout basic
      KONG_PROXY_STREAM_ERROR_LOG:   /dev/stderr
      KONG_ROUTER_FLAVOR:            traditional
      KONG_STATUS_ACCESS_LOG:        off
      KONG_STATUS_ERROR_LOG:         /dev/stderr
      KONG_STATUS_LISTEN:            0.0.0.0:8100, [::]:8100
      KONG_STREAM_LISTEN:            off
      KONG_NGINX_DAEMON:             off
    Mounts:
      /kong_dbless/ from kong-custom-dbless-config-volume (rw)
      /kong_prefix/ from kubernetes-dashboard-kong-prefix-dir (rw)
      /tmp from kubernetes-dashboard-kong-tmp (rw)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  kubernetes-dashboard-kong-prefix-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  256Mi
  kubernetes-dashboard-kong-tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  1Gi
  kubernetes-dashboard-kong-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
  kong-custom-dbless-config-volume:
    Type:        ConfigMap (a volume populated by a ConfigMap)
    Name:        kong-dbless-config
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  15m                 default-scheduler  Successfully assigned kubernetes-dashboard/kubernetes-dashboard-kong-76dff7b666-vgksj to docker-desktop
  Normal   Pulling    15m                 kubelet            Pulling image "kong:3.6"
  Normal   Pulled     15m                 kubelet            Successfully pulled image "kong:3.6" in 8.219s (12.216s including waiting)
  Normal   Created    15m                 kubelet            Created container clear-stale-pid
  Normal   Started    15m                 kubelet            Started container clear-stale-pid
  Normal   Pulled     14m (x4 over 15m)   kubelet            Container image "kong:3.6" already present on machine
  Normal   Created    14m (x4 over 15m)   kubelet            Created container proxy
  Normal   Started    14m (x4 over 15m)   kubelet            Started container proxy
  Warning  BackOff    34s (x82 over 15m)  kubelet            Back-off restarting failed container proxy in pod kubernetes-dashboard-kong-76dff7b666-vgksj_kubernetes-dashboard(55c8aa3a-6789-4e28-9128-c8ccf03e2fed)

kubectl get svc -n kubernetes-dashboard

NAME                                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                         AGE
kubernetes-dashboard-api               ClusterIP   10.105.177.157   <none>        8000/TCP                        38m
kubernetes-dashboard-auth              ClusterIP   10.111.138.36    <none>        8000/TCP                        38m
kubernetes-dashboard-kong-manager      NodePort    10.109.157.200   <none>        8002:31720/TCP,8445:30344/TCP   38m
kubernetes-dashboard-kong-proxy        ClusterIP   10.96.140.104    <none>        443/TCP                         38m
kubernetes-dashboard-metrics-scraper   ClusterIP   10.99.36.212     <none>        8000/TCP                        38m
kubernetes-dashboard-web               ClusterIP   10.103.69.54     <none>        8000/TCP                        38m

苦戦

今回の主なエラーはkongで発生しており、更にエラー文を見るとnginx: [emerg] bind() to [::1]:8444 failed (99: Cannot assign requested address)が書かれているため、IPv6関連しそうな気がしました。

一方で、helm valuesを確認してみると、特にIP関連の属性がありませんでした。以下コード抜粋

設定のところも見てみましたが、似ているものが特にありませんでした。

## Required Kong sub-chart with DBless configuration to act as a gateway
## for our all containers.
kong:
  enabled: true
  ## Configuration reference: https://docs.konghq.com/gateway/3.6.x/reference/configuration
  env:
    dns_order: LAST,A,CNAME,AAAA,SRV
    plugins: 'off'
    nginx_worker_processes: 1
  ingressController:
    enabled: false
  dblessConfig:
    configMap: kong-dbless-config
  proxy:
    type: ClusterIP
    http:
      enabled: false

また、GPTにも訪ねてみましたが、以下の内容をファイルにして、helm upgrade -f myvalue.yamlみないに試ししてくださいという解決方法を提案してくれた。

env:
  proxy:
    KONG_ADMIN_LISTEN: "127.0.0.1:8444 http2 ssl"
    KONG_PROXY_LISTEN: "0.0.0.0:8443 http2 ssl"

もちろん試してみましたが、全く効果がありませんでした。この時気付いたのは、そもそも公式のvalues.yamlにはenv.proxyなどは存在していなく、代わりにkong.envに環境変数の設定をしているみたいなので、試すことにしました。

晴れ

案の定、以下のようにカスタマイズのvaluesを修正して、helm upgradeしたらうまく起動できました！感動。

kong:
  env:
    admin_listen: '127.0.0.1:8444 http2 ssl'
    proxy_listen: '0.0.0.0:8443 http2 ssl'

kubectl get pods --watch 結果

kubernetes-dashboard-kong-76dff7b666-vgksj            0/1     Error              15 (5m11s ago)   52m
kubernetes-dashboard-kong-76dff7b666-vgksj            0/1     CrashLoopBackOff   15 (1s ago)      52m
kubernetes-dashboard-kong-67c657f866-t82cr            0/1     Pending            0                0s
kubernetes-dashboard-kong-67c657f866-t82cr            0/1     Pending            0                0s
kubernetes-dashboard-kong-67c657f866-t82cr            0/1     Init:0/1           0                0s
kubernetes-dashboard-kong-67c657f866-t82cr            0/1     PodInitializing    0                1s
kubernetes-dashboard-kong-67c657f866-t82cr            0/1     Running            0                2s
kubernetes-dashboard-kong-67c657f866-t82cr            1/1     Running            0                10s
kubernetes-dashboard-kong-76dff7b666-vgksj            0/1     Terminating        15 (112s ago)    54m
kubernetes-dashboard-kong-76dff7b666-vgksj            0/1     Terminating        15               54m
kubernetes-dashboard-kong-76dff7b666-vgksj            0/1     Terminating        15               54m
kubernetes-dashboard-kong-76dff7b666-vgksj            0/1     Terminating        15               54m
kubernetes-dashboard-kong-76dff7b666-vgksj            0/1     Terminating        15               54m
kubernetes-dashboard-kong-76dff7b666-vgksj            0/1     Terminating        15               54m

あとは簡単でした。まずsample userを作成して、以下のコマンドでport forwardをした上で、https://localhost:8443/#/loginでログインすれば画面が出てきます。

kubectl -n kubernetes-dashboard port-forward svc/kubernetes-dashboard-kong-proxy 8443:443

最後に

本には数ページの内容でしたが、実際にやってみると手順が間違っていたり、古かったりなどがよくありますので、TroubleShootingをしながら最後に解けるのが面白く、何がどこにあるかもより理解深めた気がします。