この記事は Kubernetes v1.25.4 と cAdvisor v0.46.0 の仕様に基づいて執筆しています。

kubelet から提供される cAdvisor メトリクス

公式ドキュメントに記載されているとおり、Kubernetes では kubelet バイナリに cAdvisor が統合されており、cAdvisor によって提供されるコンテナ関連のメトリクスを /metrics/cadvisor エンドポイントから収集できるようになっています。しかし、デフォルトで全てのメトリクスが収集できるわけではありません。

cAdvisor から提供される各メトリクスは option parameter でカテゴライズされており、cAdvisor バイナリやライブラリを使用する際に公開したいメトリクスのカテゴリーを指定する仕様になっています。カテゴリーの一覧はこちらから参照できます。

cAdvisor では「カテゴリー」という文言は出てきませんが、わかりやすさを考慮して以降もカテゴリーと呼称しています。

この前提を踏まえて kubelet の実装を見てみると、以下のカテゴリーのメトリクスのみが提供される仕様であることがわかります。

cpu
memory
cpuLoad
diskIO
network
app
process
oom_event

他カテゴリーのメトリクスを収集する方法

cAdvisor を単体で使用する際は --disable_metrics フラグと --enable_metrics フラグで option parameter を指定することで収集したいメトリクスを制御できますが、kubelet は cAdvisor のオプションを任意に設定できる仕様になっていません。

正確に言うと、cAdvisor ライブラリは設定の一部を CLI フラグを直接参照する実装になっているので
¹ 、ドキュメントには記載されていませんが、kubelet を介して cAdvisor フラグを渡すことは可能になっています。

$ kubelet --help | grep 'DEPRECATED: This is a cadvisor flag'
      --application-metrics-count-limit int                      Max number of application metrics to store (per container) (default 100) (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
      --boot-id-file string                                      Comma-separated list of files to check for boot-id. Use the first one that exists. (default "/proc/sys/kernel/random/boot_id") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
      --container-hints string                                   location of the container hints file (default "/etc/cadvisor/container_hints.json") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
      --containerd string                                        containerd endpoint (default "/run/containerd/containerd.sock") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
      --containerd-namespace string                              containerd namespace (default "k8s.io") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
      --docker string                                            docker endpoint (default "unix:///var/run/docker.sock") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
      --docker-env-metadata-whitelist env_metadata_whitelist     DEPRECATED: this flag will be removed, please use env_metadata_whitelist. A comma-separated list of environment variable keys matched with specified prefix that needs to be collected for docker containers (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
      --docker-only                                              Only report docker containers in addition to root stats (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
      --docker-tls                                               use TLS to connect to docker (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
      --docker-tls-ca string                                     path to trusted CA (default "ca.pem") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
      --docker-tls-cert string                                   path to client certificate (default "cert.pem") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
      --docker-tls-key string                                    path to private key (default "key.pem") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
      --enable-load-reader                                       Whether to enable cpu load reader (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
      --event-storage-age-limit string                           Max length of time for which to store events (per type). Value is a comma separated list of key values, where the keys are event types (e.g.: creation, oom) or "default" and the value is a duration. Default is applied to all non-specified event types (default "default=0") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
      --event-storage-event-limit string                         Max number of events to store (per type). Value is a comma separated list of key values, where the keys are event types (e.g.: creation, oom) or "default" and the value is an integer. Default is applied to all non-specified event types (default "default=0") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
      --global-housekeeping-interval duration                    Interval between global housekeepings (default 1m0s) (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
      --log-cadvisor-usage                                       Whether to log the usage of the cAdvisor container (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
      --machine-id-file string                                   Comma-separated list of files to check for machine-id. Use the first one that exists. (default "/etc/machine-id,/var/lib/dbus/machine-id") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
      --storage-driver-buffer-duration duration                  Writes in the storage driver will be buffered for this duration, and committed to the non memory backends as a single transaction (default 1m0s) (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
      --storage-driver-db string                                 database name (default "cadvisor") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
      --storage-driver-host string                               database host:port (default "localhost:8086") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
      --storage-driver-password string                           database password (default "root") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
      --storage-driver-secure                                    use secure connection with database (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
      --storage-driver-table string                              table name (default "stats") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
      --storage-driver-user string                               database username (default "root") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)

この仕様はバグで Kubernetes v1.10 から非推奨なフラグにされて廃止が予定されているので、使うことは避けた方が良いでしょう。詳細は Kubernetes 1.10: 既知の問題 (Known Issues) と廃止予定の機能 (Deprecations) をご参照ください。

結論ですが、kubelet に実装されていないカテゴリーのメトリクスを収集したい場合は、cAdvisor を DaemonSet で起動する対応が必要となります。詳細は https://github.com/kubernetes/kubernetes/issues/60279#issuecomment-419459174 をご参照ください。

cpuLoad カテゴリーのメトリクスを収集する際の注意点

以下の cpuLoad カテゴリーのメトリクスは、先述の通り kubelet で提供される実装になっています。

container_cpu_load_average_10s
container_tasks_state

しかし、cAdvisor ライブラリでは --enable_load_reader フラグを指定しないと上記のメトリクスは収集できない仕様なので、メトリクス自体は公開されるが値が常に0という状態になります。

それらのメトリクスが収集できないにも関わらず kubelet に実装されているのかは謎ですが、おそらく非推奨といえど --enable_load_reader フラグ (kubelet に渡す際は --enable-load-reader のように kebab case であることに注意) を指定することは可能ではあるので、一応残しているといった雰囲気でしょうか。正確な意図を知りたく Kubernetes の Community Forums に質問を投げたので、興味がある方はウォッチしてもらえればと思います。

こちらも kubelet とは別で DaemonSet で cAdvisor を稼働させることが解決方法となりますが、他コンテナのロードアベレージを取得するために、ホストのネットワークで稼働させる必要があることに注意してください。詳細は https://github.com/google/cadvisor/issues/2529 をご参照ください。

さいごに

この記事では、kubelet から提供される cAdvisor メトリクスの仕様をまとめました。関連した話ですが、cAdvisor を廃止してメトリクス収集を CRI 実装に移行する KEP も出ているので、興味がある方はチェックしてみると良いかもしれません。

kubelet では、このライブラリの仕様に従って https://github.com/kubernetes/kubernetes/blob/v1.25.4/pkg/kubelet/cadvisor/cadvisor_linux.go#L63-L80 のようなハックがなされています。この仕様は google/cadvisor#1175 や google/cadvisor#2886 で課題として挙げられているので、今後改善されるかもしれません。 ↩