はじめに
ROSA（Red Hat OpenShift Service on AWS）の構成を検討する中で、デフォルトで構成されたPV未使用のPrometheusのPodがどれぐらいInfra/Workerノード上のディスクを消費しているのかが気になり確認した際の内容を記載する。
本記事に記載する内容は特にPrometheusに特化した話ではなく、OpenShiftのPodがWorkerノード上のディスクをどの程度使用しているかを確認する方法についての話となる。
環境
- ROSA ver.4.10
- STS＆PrivateLinkモード
- Workerノード数：3台
- 複数台のテスト用アプリPodが稼働する環境
確認手順
手順の概要は以下のとおり。
- 確認対象のPodが稼働するWorkerノードの特定
- Infra/Workerノードに
oc debug nodeでアクセス -
crictlコマンドでPodの情報の確認 -
duコマンドでディスク使用量の確認
※Podに直接アクセスして確認する方法もあるが、、ここではあえて上記のようにノード上で確認する方法をとってみた。
確認対象のPodが稼働するWorkerノードの特定
oc get pod -o wideコマンドにて、OpenShift-Monitoring Project上のPodを確認する。
$ oc get pod -n openshift-monitoring -o wide
NAME READY STATUS RESTARTS AGE IP NODE
NOMINATED NODE READINESS GATES
alertmanager-main-0 6/6 Running 0 62d 10.128.4.7 ip-10-0-135-107.ec2.internal <none> <none>
alertmanager-main-1 6/6 Running 0 62d 10.131.2.8 ip-10-0-182-241.ec2.internal <none> <none>
cluster-monitoring-operator-99fb67bf4-784lp 2/2 Running 0 62d 10.128.0.39 ip-10-0-221-180.ec2.internal <none> <none>
configure-alertmanager-operator-56ff578d77-sktsq 1/1 Running 0 9d 10.131.2.88 ip-10-0-182-241.ec2.internal <none> <none>
configure-alertmanager-operator-registry-6knhb 1/1 Running 0 56d 10.131.1.67 ip-10-0-215-48.ec2.internal <none> <none>
grafana-65dc6dd678-r89dp 3/3 Running 0 62d 10.128.4.9 ip-10-0-135-107.ec2.internal <none> <none>
kube-state-metrics-66bf46c69d-lfxkb 3/3 Running 0 62d 10.128.4.20 ip-10-0-135-107.ec2.internal <none> <none>
node-exporter-45hsx 2/2 Running 2 62d 10.0.182.241 ip-10-0-182-241.ec2.internal <none> <none>
node-exporter-5hz6v 2/2 Running 2 62d 10.0.190.186 ip-10-0-190-186.ec2.internal <none> <none>
node-exporter-864s2 2/2 Running 2 62d 10.0.153.196 ip-10-0-153-196.ec2.internal <none> <none>
node-exporter-bmxkn 2/2 Running 2 62d 10.0.221.180 ip-10-0-221-180.ec2.internal <none> <none>
node-exporter-bq7n5 2/2 Running 2 62d 10.0.135.140 ip-10-0-135-140.ec2.internal <none> <none>
node-exporter-jr47c 2/2 Running 2 62d 10.0.135.107 ip-10-0-135-107.ec2.internal <none> <none>
node-exporter-p5xwt 2/2 Running 2 62d 10.0.199.127 ip-10-0-199-127.ec2.internal <none> <none>
node-exporter-szrfv 2/2 Running 0 49d 10.0.171.183 ip-10-0-171-183.ec2.internal <none> <none>
node-exporter-x6g8w 2/2 Running 2 62d 10.0.215.48 ip-10-0-215-48.ec2.internal <none> <none>
openshift-state-metrics-5d5454f44d-mbk8w 3/3 Running 0 62d 10.128.4.10 ip-10-0-135-107.ec2.internal <none> <none>
osd-rebalance-infra-nodes-27828240-rnjjn 0/1 Completed 0 40m 10.128.4.136 ip-10-0-135-107.ec2.internal <none> <none>
osd-rebalance-infra-nodes-27828255-hrxd9 0/1 Completed 0 25m 10.128.4.140 ip-10-0-135-107.ec2.internal <none> <none>
osd-rebalance-infra-nodes-27828270-wfw5s 0/1 Completed 0 10m 10.128.4.143 ip-10-0-135-107.ec2.internal <none> <none>
prometheus-adapter-7bd5fcb654-h6x2h 1/1 Running 0 2d14h 10.128.4.205 ip-10-0-135-107.ec2.internal <none> <none>
prometheus-adapter-7bd5fcb654-j5pqs 1/1 Running 0 2d14h 10.130.2.38 ip-10-0-199-127.ec2.internal <none> <none>
prometheus-k8s-0 6/6 Running 0 62d 10.131.2.7 ip-10-0-182-241.ec2.internal <none> <none>
prometheus-k8s-1 6/6 Running 0 62d 10.130.2.7 ip-10-0-199-127.ec2.internal <none> <none>
prometheus-operator-68d866488f-pf7dw 2/2 Running 0 62d 10.128.4.11 ip-10-0-135-107.ec2.internal <none> <none>
sre-dns-latency-exporter-6qr68 1/1 Running 3 139d 10.131.2.5 ip-10-0-182-241.ec2.internal <none> <none>
sre-dns-latency-exporter-762sp 1/1 Running 0 49d 10.129.4.5 ip-10-0-171-183.ec2.internal <none> <none>
sre-dns-latency-exporter-8qvvs 1/1 Running 2 139d 10.129.0.9 ip-10-0-153-196.ec2.internal <none> <none>
sre-dns-latency-exporter-cmmr5 1/1 Running 2 139d 10.128.4.5 ip-10-0-135-107.ec2.internal <none> <none>
sre-dns-latency-exporter-f7n8c 1/1 Running 1 139d 10.130.2.4 ip-10-0-199-127.ec2.internal <none> <none>
sre-dns-latency-exporter-fqg68 1/1 Running 2 139d 10.128.2.2 ip-10-0-135-140.ec2.internal <none> <none>
sre-dns-latency-exporter-htjnf 1/1 Running 2 139d 10.131.0.5 ip-10-0-215-48.ec2.internal <none> <none>
sre-dns-latency-exporter-kflb7 1/1 Running 2 139d 10.128.0.2 ip-10-0-221-180.ec2.internal <none> <none>
sre-dns-latency-exporter-phqxg 1/1 Running 2 139d 10.130.0.12 ip-10-0-190-186.ec2.internal <none> <none>
telemeter-client-774cd546df-q6j4b 3/3 Running 0 62d 10.128.4.19 ip-10-0-135-107.ec2.internal <none> <none>
thanos-querier-55dd66b46-fvclz 6/6 Running 0 62d 10.128.4.17 ip-10-0-135-107.ec2.internal <none> <none>
thanos-querier-55dd66b46-gkpsr 6/6 Running 0 62d 10.130.2.17 ip-10-0-199-127.ec2.internal <none> <none>
token-refresher-7f8c567555-qwwn2 1/1 Running 0 62d 10.130.2.18 ip-10-0-199-127.ec2.internal <none> <none>
Pod数が多いのでPrometheusだけgrepする。
$ oc get pod -n openshift-monitoring -o wide | grep prome
prometheus-adapter-7bd5fcb654-h6x2h 1/1 Running 0 2d14h 10.128.4.205 ip-10-0-135-107.ec2.internal <none> <none>
prometheus-adapter-7bd5fcb654-j5pqs 1/1 Running 0 2d14h 10.130.2.38 ip-10-0-199-127.ec2.internal <none> <none>
prometheus-k8s-0 6/6 Running 0 62d 10.131.2.7 ip-10-0-182-241.ec2.internal <none> <none>
prometheus-k8s-1 6/6 Running 0 62d 10.130.2.7 ip-10-0-199-127.ec2.internal <none> <none>
prometheus-operator-68d866488f-pf7dw 2/2 Running 0 62d 10.128.4.11 ip-10-0-135-107.ec2.internal <none> <none>
→prometheusは以下のノード上で稼働している。
- prometheus-k8s-0: ip-10-0-182-241.ec2.internal
- prometheus-k8s-1: ip-10-0-199-127.ec2.internal
※今回はprometheusのpodを確認対象とした。
（参考）
上記ノードはRoleとしてはInfra/Workerラベルが付与されている。
$ oc get no -o wide
（infra,workerロールのノードのみ抜粋）
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
ip-10-0-135-107.ec2.internal Ready infra,worker 141d v1.23.5+012e945 10.0.135.107 <none> Red Hat Enterprise Linux CoreOS 410.84.202208161501-0 (Ootpa) 4.18.0-305.57.1.el8_4.x86_64 cri-o://1.23.3-15.rhaos4.10.git6af791c.el8
ip-10-0-182-241.ec2.internal Ready infra,worker 141d v1.23.5+012e945 10.0.182.241 <none> Red Hat Enterprise Linux CoreOS 410.84.202208161501-0 (Ootpa) 4.18.0-305.57.1.el8_4.x86_64 cri-o://1.23.3-15.rhaos4.10.git6af791c.el8ip-
ip-10-0-199-127.ec2.internal Ready infra,worker 141d v1.23.5+012e945 10.0.199.127 <none> Red Hat Enterprise Linux CoreOS 410.84.202208161501-0 (Ootpa) 4.18.0-305.57.1.el8_4.x86_64 cri-o://1.23.3-15.rhaos4.10.git6af791c.el8
Infra/Workerノードにoc debug nodeでアクセス
prometheus podが稼働するWorkerノードにoc debug nodeコマンドでアクセスする。
$ oc debug node/ip-10-0-182-241.ec2.internal
Starting pod/ip-10-0-182-241ec2internal-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.182.241
If you don't see a command prompt, try pressing enter.
sh-4.4#
sh-4.4# chroot /host
sh-4.4#
コマンド実行するため、chroot /hostを実行しておく。
crictlコマンドでPodの情報の確認
稼働するpodの情報を確認するため、crictlコマンドを実行する。
sh-4.4# crictl ps
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID
250070a9b431a a1ce65e6ae985f352a5e339a1392420ac4279eef182d2e20c191b3ba84d04fc7 3 minutes ago Running container-00 0 88e93156d908a
13a02acb6bb30 quay.io/app-sre/splunk-forwarder@sha256:afc413cd2504b586b6f28c16355db243c8bfdef536cc80e44f61e03c01e12235 21 hours ago Running splunk-uf 0 9fe134fcbf6d6
531afe579fa6a dcfb77e8999b71b39fa062d4f274a0784943a017991cf5d5a4058c08ed9f7554 8 days ago Running webhook 0 a4c9e48f9316c
535cb9dadc7ae quay.io/app-sre/addon-operator-manager@sha256:41a7cf65835e4229e4468593a64ab0cdc4dac5a265aab88a379216c5eef688d6 8 days ago Running manager 0 0b4637b4ee99f
439339a0a779c 19fe8540f8737eec13b6ef65a0acfe4135415299551fed824afaefcb3067f7ad 8 days ago Running metrics-relay-server 0 0b4637b4ee99f
626378edf135d quay.io/app-sre/configure-alertmanager-operator@sha256:6c86f39326fb76657d4ed64a5bc0470374574be30507cb67d3ffb1ac78e6f851 9 days ago Running configure-alertmanager-operator 0 13fcca973bbe5
02c392154d4fb quay.io/app-sre/ocm-agent@sha256:51bafd25d611662abd8937ff3914b9a4ae5ef9361719e38947a56fd38199014d 12 days ago Running ocm-agent 0 78d512939824c
370947ae91b06 quay.io/app-sre/ocm-agent-operator@sha256:d697996b5051ee1fdf174a7d23fdd0f0fe612876e8b0d8e94a0c92fee72ea011 13 days ago Running ocm-agent-operator 0 d4e6016222c39
fbd22d001eef5 quay.io/app-sre/custom-domains-operator@sha256:59158acb6a90927f89d85ffaf448e4831b8e8b352d4539b597b97e5f5ff92f6b 4 weeks ago Running custom-domains-operator 0 c4f5c43d5d5eb
9208b357e7965 quay.io/app-sre/must-gather-operator@sha256:2e9c61f5bb3d7f95805130843b0368853d38934756e892954cdf3e251941ac43 6 weeks ago Running must-gather-operator 0 27b5ebcdd77e4
54c912fdc02c5 quay.io/app-sre/managed-node-metadata-operator@sha256:81cb5d1fcc8261858727876036c86fcf45b5b3cf9f5129450019c6f107cd11da 7 weeks ago Running managed-node-metadata-operator 0 aa02b2b6f8f3e
4d971e953ad86 410e15456e199925ac531d55aa982db94829abd83b825a1f3c8ef8ad367ec3a8 2 months ago Running prom-label-proxy 0 32e6aba11b9b3
340cf915b365c b0eb8d3033872241913cc97cb4ef9e9b8e52e3584b8c3a3dfbc38332f17501aa 2 months ago Running kube-rbac-proxy-metric 0 32e6aba11b9b3
6fb5f39097c94 b0eb8d3033872241913cc97cb4ef9e9b8e52e3584b8c3a3dfbc38332f17501aa 2 months ago Running kube-rbac-proxy 0 32e6aba11b9b3
c4ab5a439405d 4a35485847278231eaefeeba227df3b848fcd88251bf387cc75814683c6e1cc9 2 months ago Running alertmanager-proxy 0 32e6aba11b9b3
44aba6706a156 d7e69e3e6c77da1eb69a016accb5755bd259cdf6b08bbe2505482bdaf191c1a7 2 months ago Running config-reloader 0 32e6aba11b9b3
43a4c4e9f1aa1 5b7a0f5af01de4e8ccd7d80557c92f25452b8ae54423ebbd93cd6b8cc11bf086 2 months ago Running alertmanager 0 32e6aba11b9b3
0a26c550f6b59 b0eb8d3033872241913cc97cb4ef9e9b8e52e3584b8c3a3dfbc38332f17501aa 2 months ago Running kube-rbac-proxy-thanos 0 fde2304412307
bc28291be733a b0eb8d3033872241913cc97cb4ef9e9b8e52e3584b8c3a3dfbc38332f17501aa 2 months ago Running kube-rbac-proxy 0 fde2304412307
9d94467cb5722 4a35485847278231eaefeeba227df3b848fcd88251bf387cc75814683c6e1cc9 2 months ago Running prometheus-proxy 0 fde2304412307
4359fb8b597b6 1cff5886ffcf498a37eb3868762403fb2b0ca2feb4a14c98471fc256b680619e 2 months ago Running thanos-sidecar 0 fde2304412307
01b339ba5ea1d d7e69e3e6c77da1eb69a016accb5755bd259cdf6b08bbe2505482bdaf191c1a7 2 months ago Running config-reloader 0 fde2304412307
27cccaa137e39 d3017e38d93fb39bcad5ae9ee8521f5829c774c389ae7e65e44a00388d67cf1e 2 months ago Running prometheus 0 fde2304412307
9789aad219d40 b08d2cd9618b330d905aeafe0bb8187a768fbe94c17fe9916d088dbf59acefc3 2 months ago Running kube-multus-additional-cni-plugins 1 923c608753588
d05f57994cae1 46bea5c4f6e46d4112dbbf837186ba68c231c9edd3bb22ccfbf76ad32a329ffd 2 months ago Running main 3 6beefa759e932
86c189c7529f6 53ba5113526775eab777eed2bfdd9d6148ae563a3330f609394dcca565283f7b 2 months ago Running serve-healthcheck-canary 1 989fc22ba34bb
fd0a17b4d8ecb 77a2af086aeb1c31914686118f3f20e44f836a251d89f256c5620cda914445a9 2 months ago Running network-check-target-container 1 0139053cefa0d
5ed1426014c35 b0eb8d3033872241913cc97cb4ef9e9b8e52e3584b8c3a3dfbc38332f17501aa 2 months ago Running kube-rbac-proxy 1 f57ba7ed09785
1032c983cbbe0 3e0c06861c106b17273bd4c553ca4be6f34ca3cc226f89b2092aa9d7062a735d 2 months ago Running network-metrics-daemon 1 f57ba7ed09785
c8dc4f438ce2e b0eb8d3033872241913cc97cb4ef9e9b8e52e3584b8c3a3dfbc38332f17501aa 2 months ago Running kube-rbac-proxy 1 f74f3801db28d
f6453301898b4 fa34f535a918d4d6d07e8b105b9209dfd8b6c9152cee340e10f55934d6b07690 2 months ago Running node-exporter 1 f74f3801db28d
56010ff9bb875 75ae6c537332567afdc034809cf030f0cfea3cb7ecbc2557be821406abf5adb4 2 months ago Running csi-liveness-probe 1 fc7868d48269e
360e48e827df6 b0eb8d3033872241913cc97cb4ef9e9b8e52e3584b8c3a3dfbc38332f17501aa 2 months ago Running kube-rbac-proxy 1 aa3699225a6dd
61b3fe4869316 4a35485847278231eaefeeba227df3b848fcd88251bf387cc75814683c6e1cc9 2 months ago Running oauth-proxy 1 6c0bdff805442
5ca4224aa30f6 97448add286660fbbf4e14abb55c4a728c07162727cf7f7753c014c6540ba9b7 2 months ago Running csi-node-driver-registrar 1 fc7868d48269e
4cc36ee09e831 f44d6c1a2dad7deef542a030cce029d3904da2d69a2a7203094e8e7a15d30883 2 months ago Running dns-node-resolver 1 3a86f4d95389b
7d70ca3301053 bded42f93595dfe68ce77cac9a401479c0e60180b82947576276e95319fd185c 2 months ago Running sdn 1 aa3699225a6dd
8a8fb4dd0b34f 9e085bf5fd76546272ae0de97e127cceb9fa19518128928f4ea1ff19794a420d 2 months ago Running csi-driver 1 fc7868d48269e
b46475b65fe1b 3e34a83ad92bf15167ec7b450520a61bb5028985ac19ce0bf16aa89d84b58c0f 2 months ago Running machine-config-daemon 1 6c0bdff805442
ab100fa37c49f 8cc9e7e28492bd39a5a0efde92b8a11cbbeb7066e907cb25dab5d7d1ad346477 2 months ago Running node-ca 1 dfb3974a993f5
8a7b1ea93c0c3 b08d2cd9618b330d905aeafe0bb8187a768fbe94c17fe9916d088dbf59acefc3 2 months ago Running kube-multus 1 628ede44e698b
3918dd5445192 57d6bfa9aa0faa2b850635e158e56230035e1910361d5412bb939ec7fe409527 2 months ago Running tuned 1 04b08f03e5865
sh-4.4#
pod数が多いので、grep。
sh-4.4# crictl ps | grep prometheus
9d94467cb5722 4a35485847278231eaefeeba227df3b848fcd88251bf387cc75814683c6e1cc9 2 months ago Running prometheus-proxy 0 fde2304412307
27cccaa137e39 d3017e38d93fb39bcad5ae9ee8521f5829c774c389ae7e65e44a00388d67cf1e 2 months ago Running prometheus 0 fde2304412307
sh-4.4#
prometheusのpodの詳細情報を確認するため、crictl inspectを実行する。
（長いですが、全量載せておきます）
sh-4.4# crictl inspect 27cccaa137e39
{
"status": {
"id": "27cccaa137e395427c3ddaef728528ddfd3e1d83cd2710a9d15fa1202e93e084",
"metadata": {
"attempt": 0,
"name": "prometheus"
},
"state": "CONTAINER_RUNNING",
"createdAt": "2022-09-27T08:46:55.100751755Z",
"startedAt": "2022-09-27T08:46:55.126901515Z",
"finishedAt": "0001-01-01T00:00:00Z",
"exitCode": 0,
"image": {
"annotations": {},
"image": "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:00d156818e2e4264c3268c9ae41de9b561e2811363b64515ca64865c992ec8ea"
},
"imageRef": "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:00d156818e2e4264c3268c9ae41de9b561e2811363b64515ca64865c992ec8ea",
"reason": "",
"message": "",
"labels": {
"io.kubernetes.container.name": "prometheus",
"io.kubernetes.pod.name": "prometheus-k8s-0",
"io.kubernetes.pod.namespace": "openshift-monitoring",
"io.kubernetes.pod.uid": "a4477163-737d-457c-8698-8688fdd00003"
},
"annotations": {
"io.kubernetes.container.hash": "7ba9b53",
"io.kubernetes.container.restartCount": "0",
"io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
"io.kubernetes.container.terminationMessagePolicy": "FallbackToLogsOnError",
"io.kubernetes.pod.terminationGracePeriod": "600"
},
"mounts": [
{
"containerPath": "/prometheus",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volume-subpaths/pvc-b17ae945-02d2-43e2-a2d2-23891203b1f3/prometheus/3",
"propagation": "PROPAGATION_PRIVATE",
"readonly": false,
"selinuxRelabel": true
},
{
"containerPath": "/dev/termination-log",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/containers/prometheus/5ab56fc4",
"propagation": "PROPAGATION_PRIVATE",
"readonly": false,
"selinuxRelabel": true
},
{
"containerPath": "/etc/hosts",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/etc-hosts",
"propagation": "PROPAGATION_PRIVATE",
"readonly": false,
"selinuxRelabel": true
},
{
"containerPath": "/etc/prometheus/config_out",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~empty-dir/config-out",
"propagation": "PROPAGATION_PRIVATE",
"readonly": true,
"selinuxRelabel": true
},
{
"containerPath": "/etc/prometheus/certs",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~projected/tls-assets",
"propagation": "PROPAGATION_PRIVATE",
"readonly": true,
"selinuxRelabel": true
},
{
"containerPath": "/etc/prometheus/secrets/prometheus-k8s-thanos-sidecar-tls",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls",
"propagation": "PROPAGATION_PRIVATE",
"readonly": true,
"selinuxRelabel": true
},
{
"containerPath": "/etc/prometheus/secrets/kube-etcd-client-certs",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-kube-etcd-client-certs",
"propagation": "PROPAGATION_PRIVATE",
"readonly": true,
"selinuxRelabel": true
},
{
"containerPath": "/etc/prometheus/secrets/prometheus-k8s-tls",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls",
"propagation": "PROPAGATION_PRIVATE",
"readonly": true,
"selinuxRelabel": true
},
{
"containerPath": "/etc/prometheus/secrets/prometheus-k8s-proxy",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-proxy",
"propagation": "PROPAGATION_PRIVATE",
"readonly": true,
"selinuxRelabel": true
},
{
"containerPath": "/etc/prometheus/web_config/web-config.yaml",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volume-subpaths/web-config/prometheus/5",
"propagation": "PROPAGATION_PRIVATE",
"readonly": true,
"selinuxRelabel": true
},
{
"containerPath": "/etc/prometheus/secrets/kube-rbac-proxy",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-kube-rbac-proxy",
"propagation": "PROPAGATION_PRIVATE",
"readonly": true,
"selinuxRelabel": true
},
{
"containerPath": "/etc/prometheus/secrets/metrics-client-certs",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-metrics-client-certs",
"propagation": "PROPAGATION_PRIVATE",
"readonly": true,
"selinuxRelabel": true
},
{
"containerPath": "/etc/prometheus/configmaps/serving-certs-ca-bundle",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-serving-certs-ca-bundle",
"propagation": "PROPAGATION_PRIVATE",
"readonly": true,
"selinuxRelabel": true
},
{
"containerPath": "/etc/prometheus/configmaps/kubelet-serving-ca-bundle",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-kubelet-serving-ca-bundle",
"propagation": "PROPAGATION_PRIVATE",
"readonly": true,
"selinuxRelabel": true
},
{
"containerPath": "/etc/prometheus/configmaps/metrics-client-ca",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-metrics-client-ca",
"propagation": "PROPAGATION_PRIVATE",
"readonly": true,
"selinuxRelabel": true
},
{
"containerPath": "/etc/prometheus/rules/prometheus-k8s-rulefiles-0",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/prometheus-k8s-rulefiles-0",
"propagation": "PROPAGATION_PRIVATE",
"readonly": true,
"selinuxRelabel": true
},
{
"containerPath": "/var/run/secrets/kubernetes.io/serviceaccount",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~projected/kube-api-access-jfzqj",
"propagation": "PROPAGATION_PRIVATE",
"readonly": true,
"selinuxRelabel": true
},
{
"containerPath": "/etc/pki/ca-trust/extracted/pem/",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/prometheus-trusted-ca-bundle",
"propagation": "PROPAGATION_PRIVATE",
"readonly": true,
"selinuxRelabel": true
}
],
"logPath": "/var/log/pods/openshift-monitoring_prometheus-k8s-0_a4477163-737d-457c-8698-8688fdd00003/prometheus/0.log"
},
"info": {
"sandboxID": "fde2304412307a44d51118bcf4d3ef4089fd71e5aa12418ca6716a6395194d7d",
"pid": 4693,
"runtimeSpec": {
"ociVersion": "1.0.2-dev",
"process": {
"user": {
"uid": 65534,
"gid": 65534,
"additionalGids": [
65534
]
},
"args": [
"/bin/prometheus",
"--web.console.templates=/etc/prometheus/consoles",
"--web.console.libraries=/etc/prometheus/console_libraries",
"--config.file=/etc/prometheus/config_out/prometheus.env.yaml",
"--storage.tsdb.path=/prometheus",
"--storage.tsdb.retention.time=11d",
"--web.enable-lifecycle",
"--web.external-url=https://prometheus-k8s-openshift-monitoring.apps.aws-rosa-xxx-xxx.xxx.xxx.openshiftapps.com/",
"--web.route-prefix=/",
"--web.listen-address=127.0.0.1:9090",
"--web.config.file=/etc/prometheus/web_config/web-config.yaml"
],
"env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"TERM=xterm",
"HOSTNAME=prometheus-k8s-0",
"NSS_SDB_USE_CACHE=no",
"GRAFANA_PORT_3000_TCP_ADDR=172.30.251.5",
"GRAFANA_PORT_3002_TCP_ADDR=172.30.251.5",
"THANOS_QUERIER_PORT_9093_TCP_PROTO=tcp",
"TOKEN_REFRESHER_SERVICE_PORT_HTTP=80",
"KUBERNETES_PORT_443_TCP_ADDR=172.30.0.1",
"CONFIGURE_ALERTMANAGER_OPERATOR_SERVICE_PORT=8080",
"CONFIGURE_ALERTMANAGER_OPERATOR_PORT_8080_TCP_PROTO=tcp",
"ALERTMANAGER_MAIN_SERVICE_HOST=172.30.199.91",
"TOKEN_REFRESHER_SERVICE_PORT=80",
"KUBERNETES_SERVICE_PORT_HTTPS=443",
"PROMETHEUS_K8S_SERVICE_PORT_WEB=9091",
"THANOS_QUERIER_PORT_9093_TCP_PORT=9093",
"PROMETHEUS_K8S_PORT_9091_TCP=tcp://172.30.214.226:9091",
"ALERTMANAGER_MAIN_PORT_9092_TCP_PROTO=tcp",
"GRAFANA_PORT_3000_TCP_PROTO=tcp",
"GRAFANA_PORT_3002_TCP_PORT=3002",
"PROMETHEUS_K8S_PORT=tcp://172.30.214.226:9091",
"THANOS_QUERIER_PORT_9093_TCP_ADDR=172.30.64.108",
"TOKEN_REFRESHER_PORT_80_TCP=tcp://172.30.227.107:80",
"ALERTMANAGER_MAIN_PORT_9097_TCP_ADDR=172.30.199.91",
"THANOS_QUERIER_PORT_9094_TCP_PROTO=tcp",
"CONFIGURE_ALERTMANAGER_OPERATOR_PORT_8080_TCP=tcp://172.30.32.44:8080",
"GRAFANA_PORT=tcp://172.30.251.5:3000",
"SRE_DNS_LATENCY_EXPORTER_PORT_80_TCP=tcp://172.30.68.79:80",
"PROMETHEUS_K8S_PORT_9092_TCP_ADDR=172.30.214.226",
"CONFIGURE_ALERTMANAGER_OPERATOR_REGISTRY_PORT_50051_TCP_PORT=50051",
"ALERTMANAGER_MAIN_PORT_9097_TCP_PORT=9097",
"KUBERNETES_PORT_443_TCP_PORT=443",
"PROMETHEUS_K8S_SERVICE_PORT_METRICS=9092",
"THANOS_QUERIER_PORT_9091_TCP_PORT=9091",
"THANOS_QUERIER_PORT_9092_TCP=tcp://172.30.64.108:9092",
"THANOS_QUERIER_PORT_9094_TCP_PORT=9094",
"ALERTMANAGER_MAIN_PORT_9094_TCP_PROTO=tcp",
"THANOS_QUERIER_SERVICE_PORT_WEB=9091",
"CONFIGURE_ALERTMANAGER_OPERATOR_PORT=tcp://172.30.32.44:8080",
"PROMETHEUS_K8S_PORT_9091_TCP_ADDR=172.30.214.226",
"ALERTMANAGER_MAIN_PORT_9094_TCP=tcp://172.30.199.91:9094",
"ALERTMANAGER_MAIN_PORT_9092_TCP_ADDR=172.30.199.91",
"PROMETHEUS_ADAPTER_PORT_443_TCP=tcp://172.30.48.128:443",
"GRAFANA_PORT_3000_TCP_PORT=3000",
"GRAFANA_PORT_3000_TCP=tcp://172.30.251.5:3000",
"CONFIGURE_ALERTMANAGER_OPERATOR_REGISTRY_SERVICE_PORT_GRPC=50051",
"CONFIGURE_ALERTMANAGER_OPERATOR_REGISTRY_PORT_50051_TCP_PROTO=tcp",
"ALERTMANAGER_MAIN_SERVICE_PORT=9094",
"PROMETHEUS_ADAPTER_SERVICE_PORT=443",
"SRE_DNS_LATENCY_EXPORTER_PORT_80_TCP_ADDR=172.30.68.79",
"THANOS_QUERIER_SERVICE_PORT_TENANCY=9092",
"THANOS_QUERIER_PORT=tcp://172.30.64.108:9091",
"THANOS_QUERIER_PORT_9091_TCP=tcp://172.30.64.108:9091",
"PROMETHEUS_ADAPTER_PORT_443_TCP_PORT=443",
"THANOS_QUERIER_PORT_9091_TCP_ADDR=172.30.64.108",
"THANOS_QUERIER_PORT_9092_TCP_PROTO=tcp",
"CONFIGURE_ALERTMANAGER_OPERATOR_PORT_8080_TCP_PORT=8080",
"ALERTMANAGER_MAIN_PORT_9094_TCP_ADDR=172.30.199.91",
"TOKEN_REFRESHER_PORT=tcp://172.30.227.107:80",
"GRAFANA_SERVICE_PORT_HTTPS=3000",
"SRE_DNS_LATENCY_EXPORTER_SERVICE_HOST=172.30.68.79",
"PROMETHEUS_K8S_PORT_9091_TCP_PORT=9091",
"THANOS_QUERIER_PORT_9094_TCP_ADDR=172.30.64.108",
"PROMETHEUS_ADAPTER_SERVICE_HOST=172.30.48.128",
"THANOS_QUERIER_SERVICE_PORT=9091",
"THANOS_QUERIER_PORT_9092_TCP_ADDR=172.30.64.108",
"THANOS_QUERIER_PORT_9093_TCP=tcp://172.30.64.108:9093",
"TOKEN_REFRESHER_PORT_80_TCP_PORT=80",
"GRAFANA_PORT_3002_TCP_PROTO=tcp",
"PROMETHEUS_K8S_PORT_9092_TCP_PROTO=tcp",
"CONFIGURE_ALERTMANAGER_OPERATOR_REGISTRY_PORT_50051_TCP_ADDR=172.30.187.125",
"ALERTMANAGER_MAIN_SERVICE_PORT_WEB=9094",
"CONFIGURE_ALERTMANAGER_OPERATOR_REGISTRY_PORT=tcp://172.30.187.125:50051",
"ALERTMANAGER_MAIN_PORT_9097_TCP=tcp://172.30.199.91:9097",
"KUBERNETES_SERVICE_PORT=443",
"GRAFANA_SERVICE_PORT=3000",
"SRE_DNS_LATENCY_EXPORTER_SERVICE_PORT_HTTP_MAIN=80",
"PROMETHEUS_K8S_PORT_9092_TCP=tcp://172.30.214.226:9092",
"PROMETHEUS_K8S_PORT_9091_TCP_PROTO=tcp",
"ALERTMANAGER_MAIN_SERVICE_PORT_METRICS=9097",
"ALERTMANAGER_MAIN_PORT_9097_TCP_PROTO=tcp",
"THANOS_QUERIER_SERVICE_PORT_METRICS=9094",
"CONFIGURE_ALERTMANAGER_OPERATOR_SERVICE_PORT_METRICS=8080",
"GRAFANA_SERVICE_HOST=172.30.251.5",
"GRAFANA_SERVICE_PORT_METRICS=3002",
"PROMETHEUS_K8S_SERVICE_PORT=9091",
"CONFIGURE_ALERTMANAGER_OPERATOR_REGISTRY_SERVICE_PORT=50051",
"THANOS_QUERIER_SERVICE_HOST=172.30.64.108",
"CONFIGURE_ALERTMANAGER_OPERATOR_SERVICE_HOST=172.30.32.44",
"GRAFANA_PORT_3002_TCP=tcp://172.30.251.5:3002",
"SRE_DNS_LATENCY_EXPORTER_SERVICE_PORT=80",
"ALERTMANAGER_MAIN_PORT_9092_TCP=tcp://172.30.199.91:9092",
"ALERTMANAGER_MAIN_PORT_9092_TCP_PORT=9092",
"THANOS_QUERIER_PORT_9092_TCP_PORT=9092",
"CONFIGURE_ALERTMANAGER_OPERATOR_PORT_8080_TCP_ADDR=172.30.32.44",
"THANOS_QUERIER_SERVICE_PORT_TENANCY_RULES=9093",
"TOKEN_REFRESHER_SERVICE_HOST=172.30.227.107",
"SRE_DNS_LATENCY_EXPORTER_PORT_80_TCP_PORT=80",
"PROMETHEUS_ADAPTER_SERVICE_PORT_HTTPS=443",
"THANOS_QUERIER_PORT_9094_TCP=tcp://172.30.64.108:9094",
"TOKEN_REFRESHER_PORT_80_TCP_PROTO=tcp",
"SRE_DNS_LATENCY_EXPORTER_PORT_80_TCP_PROTO=tcp",
"ALERTMANAGER_MAIN_SERVICE_PORT_TENANCY=9092",
"ALERTMANAGER_MAIN_PORT_9094_TCP_PORT=9094",
"PROMETHEUS_ADAPTER_PORT_443_TCP_ADDR=172.30.48.128",
"THANOS_QUERIER_PORT_9091_TCP_PROTO=tcp",
"KUBERNETES_PORT_443_TCP=tcp://172.30.0.1:443",
"KUBERNETES_PORT_443_TCP_PROTO=tcp",
"ALERTMANAGER_MAIN_PORT=tcp://172.30.199.91:9094",
"KUBERNETES_SERVICE_HOST=172.30.0.1",
"KUBERNETES_PORT=tcp://172.30.0.1:443",
"CONFIGURE_ALERTMANAGER_OPERATOR_REGISTRY_PORT_50051_TCP=tcp://172.30.187.125:50051",
"PROMETHEUS_ADAPTER_PORT=tcp://172.30.48.128:443",
"PROMETHEUS_ADAPTER_PORT_443_TCP_PROTO=tcp",
"CONFIGURE_ALERTMANAGER_OPERATOR_REGISTRY_SERVICE_HOST=172.30.187.125",
"TOKEN_REFRESHER_PORT_80_TCP_ADDR=172.30.227.107",
"SRE_DNS_LATENCY_EXPORTER_PORT=tcp://172.30.68.79:80",
"PROMETHEUS_K8S_SERVICE_HOST=172.30.214.226",
"PROMETHEUS_K8S_PORT_9092_TCP_PORT=9092",
"__doozer=merge",
"BUILD_RELEASE=202208150436.p0.g3c6cd55.assembly.stream",
"BUILD_VERSION=v4.10.0",
"OS_GIT_MAJOR=4",
"OS_GIT_MINOR=10",
"OS_GIT_PATCH=0",
"OS_GIT_TREE_STATE=clean",
"OS_GIT_VERSION=4.10.0-202208150436.p0.g3c6cd55.assembly.stream-3c6cd55",
"SOURCE_GIT_TREE_STATE=clean",
"OS_GIT_COMMIT=3c6cd55",
"SOURCE_DATE_EPOCH=1640014589",
"SOURCE_GIT_COMMIT=3c6cd55e3a5e5332e810274ad27fddf662d4f506",
"SOURCE_GIT_TAG=golang-github-prometheus-prometheus-4.0.0-0.102.0-4305-g3c6cd55e3",
"SOURCE_GIT_URL=https://github.com/openshift/prometheus",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"container=oci"
],
"cwd": "/prometheus",
"capabilities": {
"bounding": [
"CAP_CHOWN",
"CAP_DAC_OVERRIDE",
"CAP_FSETID",
"CAP_FOWNER",
"CAP_NET_BIND_SERVICE",
"CAP_SETPCAP"
],
"effective": [
"CAP_CHOWN",
"CAP_DAC_OVERRIDE",
"CAP_FSETID",
"CAP_FOWNER",
"CAP_NET_BIND_SERVICE",
"CAP_SETPCAP"
],
"permitted": [
"CAP_CHOWN",
"CAP_DAC_OVERRIDE",
"CAP_FSETID",
"CAP_FOWNER",
"CAP_NET_BIND_SERVICE",
"CAP_SETPCAP"
]
},
"oomScoreAdj": 968,
"selinuxLabel": "system_u:system_r:container_t:s0:c21,c0"
},
"root": {
"path": "/var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged"
},
"hostname": "prometheus-k8s-0",
"mounts": [
{
"destination": "/proc",
"type": "proc",
"source": "proc",
"options": [
"nosuid",
"noexec",
"nodev"
]
},
{
"destination": "/dev",
"type": "tmpfs",
"source": "tmpfs",
"options": [
"nosuid",
"strictatime",
"mode=755",
"size=65536k"
]
},
{
"destination": "/dev/pts",
"type": "devpts",
"source": "devpts",
"options": [
"nosuid",
"noexec",
"newinstance",
"ptmxmode=0666",
"mode=0620",
"gid=5"
]
},
{
"destination": "/dev/mqueue",
"type": "mqueue",
"source": "mqueue",
"options": [
"nosuid",
"noexec",
"nodev"
]
},
{
"destination": "/sys",
"type": "sysfs",
"source": "sysfs",
"options": [
"nosuid",
"noexec",
"nodev",
"ro"
]
},
{
"destination": "/sys/fs/cgroup",
"type": "cgroup",
"source": "cgroup",
"options": [
"nosuid",
"noexec",
"nodev",
"relatime",
"ro"
]
},
{
"destination": "/dev/shm",
"type": "bind",
"source": "/run/containers/storage/overlay-containers/fde2304412307a44d51118bcf4d3ef4089fd71e5aa12418ca6716a6395194d7d/userdata/shm",
"options": [
"rw",
"bind"
]
},
{
"destination": "/etc/resolv.conf",
"type": "bind",
"source": "/run/containers/storage/overlay-containers/fde2304412307a44d51118bcf4d3ef4089fd71e5aa12418ca6716a6395194d7d/userdata/resolv.conf",
"options": [
"rw",
"bind",
"nodev",
"nosuid",
"noexec"
]
},
{
"destination": "/etc/hostname",
"type": "bind",
"source": "/run/containers/storage/overlay-containers/fde2304412307a44d51118bcf4d3ef4089fd71e5aa12418ca6716a6395194d7d/userdata/hostname",
"options": [
"rw",
"bind"
]
},
{
"destination": "/run/.containerenv",
"type": "bind",
"source": "/run/containers/storage/overlay-containers/fde2304412307a44d51118bcf4d3ef4089fd71e5aa12418ca6716a6395194d7d/userdata/.containerenv",
"options": [
"rw",
"bind"
]
},
{
"destination": "/prometheus",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volume-subpaths/pvc-b17ae945-02d2-43e2-a2d2-23891203b1f3/prometheus/3",
"options": [
"rw",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/dev/termination-log",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/containers/prometheus/5ab56fc4",
"options": [
"rw",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/etc/hosts",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/etc-hosts",
"options": [
"rw",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/run/secrets",
"type": "bind",
"source": "/run/containers/storage/overlay-containers/27cccaa137e395427c3ddaef728528ddfd3e1d83cd2710a9d15fa1202e93e084/userdata/run/secrets",
"options": [
"bind",
"rprivate",
"bind"
]
},
{
"destination": "/etc/prometheus/config_out",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~empty-dir/config-out",
"options": [
"ro",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/etc/prometheus/certs",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~projected/tls-assets",
"options": [
"ro",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/etc/prometheus/secrets/metrics-client-certs",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-metrics-client-certs",
"options": [
"ro",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/etc/prometheus/secrets/prometheus-k8s-tls",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls",
"options": [
"ro",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/etc/prometheus/secrets/prometheus-k8s-proxy",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-proxy",
"options": [
"ro",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/etc/prometheus/secrets/kube-etcd-client-certs",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-kube-etcd-client-certs",
"options": [
"ro",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/etc/prometheus/secrets/kube-rbac-proxy",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-kube-rbac-proxy",
"options": [
"ro",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/etc/prometheus/secrets/prometheus-k8s-thanos-sidecar-tls",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls",
"options": [
"ro",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/etc/prometheus/configmaps/serving-certs-ca-bundle",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-serving-certs-ca-bundle",
"options": [
"ro",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/etc/prometheus/configmaps/kubelet-serving-ca-bundle",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-kubelet-serving-ca-bundle",
"options": [
"ro",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/etc/prometheus/configmaps/metrics-client-ca",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-metrics-client-ca",
"options": [
"ro",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/etc/prometheus/rules/prometheus-k8s-rulefiles-0",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/prometheus-k8s-rulefiles-0",
"options": [
"ro",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/etc/prometheus/web_config/web-config.yaml",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volume-subpaths/web-config/prometheus/5",
"options": [
"ro",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/var/run/secrets/kubernetes.io/serviceaccount",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~projected/kube-api-access-jfzqj",
"options": [
"ro",
"rbind",
"rprivate",
"bind"
]
},
{
"destination": "/etc/pki/ca-trust/extracted/pem/",
"type": "bind",
"source": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/prometheus-trusted-ca-bundle",
"options": [
"ro",
"rbind",
"rprivate",
"bind"
]
}
],
"annotations": {
"openshift.io/scc": "nonroot",
"io.kubernetes.cri-o.Name": "k8s_prometheus_prometheus-k8s-0_openshift-monitoring_a4477163-737d-457c-8698-8688fdd00003_0",
"io.kubernetes.cri-o.SandboxName": "k8s_prometheus-k8s-0_openshift-monitoring_a4477163-737d-457c-8698-8688fdd00003_0",
"kubectl.kubernetes.io/default-container": "prometheus",
"io.kubernetes.cri-o.Stdin": "false",
"io.kubernetes.cri-o.ResolvPath": "/run/containers/storage/overlay-containers/fde2304412307a44d51118bcf4d3ef4089fd71e5aa12418ca6716a6395194d7d/userdata/resolv.conf",
"io.kubernetes.cri-o.Volumes": "[{\"container_path\":\"/prometheus\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volume-subpaths/pvc-b17ae945-02d2-43e2-a2d2-23891203b1f3/prometheus/3\",\"readonly\":false,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/dev/termination-log\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/containers/prometheus/5ab56fc4\",\"readonly\":false,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/hosts\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/etc-hosts\",\"readonly\":false,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/config_out\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~empty-dir/config-out\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/certs\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~projected/tls-assets\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/secrets/prometheus-k8s-thanos-sidecar-tls\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/secrets/kube-etcd-client-certs\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-kube-etcd-client-certs\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/secrets/prometheus-k8s-tls\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/secrets/prometheus-k8s-proxy\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-proxy\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/web_config/web-config.yaml\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volume-subpaths/web-config/prometheus/5\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/secrets/kube-rbac-proxy\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-kube-rbac-proxy\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/secrets/metrics-client-certs\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-metrics-client-certs\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/configmaps/serving-certs-ca-bundle\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-serving-certs-ca-bundle\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/configmaps/kubelet-serving-ca-bundle\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-kubelet-serving-ca-bundle\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/configmaps/metrics-client-ca\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-metrics-client-ca\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/prometheus/rules/prometheus-k8s-rulefiles-0\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/prometheus-k8s-rulefiles-0\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/var/run/secrets/kubernetes.io/serviceaccount\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~projected/kube-api-access-jfzqj\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true},{\"container_path\":\"/etc/pki/ca-trust/extracted/pem/\",\"host_path\":\"/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/prometheus-trusted-ca-bundle\",\"readonly\":true,\"propagation\":0,\"selinux_relabel\":true}]",
"io.kubernetes.container.hash": "7ba9b53",
"io.kubernetes.pod.name": "prometheus-k8s-0",
"io.kubernetes.cri-o.IP.0": "10.131.2.7",
"io.kubernetes.cri-o.SeccompProfilePath": "",
"io.kubernetes.cri-o.Metadata": "{\"name\":\"prometheus\"}",
"io.kubernetes.cri-o.Labels": "{\"io.kubernetes.container.name\":\"prometheus\",\"io.kubernetes.pod.name\":\"prometheus-k8s-0\",\"io.kubernetes.pod.namespace\":\"openshift-monitoring\",\"io.kubernetes.pod.uid\":\"a4477163-737d-457c-8698-8688fdd00003\"}",
"io.kubernetes.pod.uid": "a4477163-737d-457c-8698-8688fdd00003",
"kubernetes.io/config.seen": "2022-09-27T08:46:47.929904784Z",
"io.kubernetes.cri-o.ImageName": "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:00d156818e2e4264c3268c9ae41de9b561e2811363b64515ca64865c992ec8ea",
"io.kubernetes.cri-o.ContainerType": "container",
"io.kubernetes.cri-o.StdinOnce": "false",
"io.kubernetes.cri-o.Created": "2022-09-27T08:46:55.0148206Z",
"io.kubernetes.container.terminationMessagePolicy": "FallbackToLogsOnError",
"kubernetes.io/config.source": "api",
"io.kubernetes.cri-o.ImageRef": "d3017e38d93fb39bcad5ae9ee8521f5829c774c389ae7e65e44a00388d67cf1e",
"io.kubernetes.container.restartCount": "0",
"org.systemd.property.DefaultDependencies": "true",
"io.kubernetes.container.name": "prometheus",
"io.kubernetes.cri-o.LogPath": "/var/log/pods/openshift-monitoring_prometheus-k8s-0_a4477163-737d-457c-8698-8688fdd00003/prometheus/0.log",
"io.kubernetes.cri-o.TTY": "false",
"io.container.manager": "cri-o",
"io.kubernetes.cri-o.MountPoint": "/var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged",
"io.kubernetes.cri-o.Annotations": "{\"io.kubernetes.container.hash\":\"7ba9b53\",\"io.kubernetes.container.restartCount\":\"0\",\"io.kubernetes.container.terminationMessagePath\":\"/dev/termination-log\",\"io.kubernetes.container.terminationMessagePolicy\":\"FallbackToLogsOnError\",\"io.kubernetes.pod.terminationGracePeriod\":\"600\"}",
"io.kubernetes.pod.terminationGracePeriod": "600",
"io.kubernetes.pod.namespace": "openshift-monitoring",
"io.kubernetes.cri-o.SandboxID": "fde2304412307a44d51118bcf4d3ef4089fd71e5aa12418ca6716a6395194d7d",
"io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
"io.kubernetes.cri-o.Image": "d3017e38d93fb39bcad5ae9ee8521f5829c774c389ae7e65e44a00388d67cf1e",
"io.kubernetes.cri-o.ContainerID": "27cccaa137e395427c3ddaef728528ddfd3e1d83cd2710a9d15fa1202e93e084",
"org.systemd.property.TimeoutStopUSec": "uint64 600000000",
"org.systemd.property.CollectMode": "'inactive-or-failed'",
"org.systemd.property.After": "['crio.service']"
},
"linux": {
"resources": {
"devices": [
{
"allow": false,
"access": "rwm"
}
],
"cpu": {
"shares": 71,
"quota": 0,
"period": 100000
},
"pids": {
"limit": 4096
},
"hugepageLimits": [
{
"pageSize": "2MB",
"limit": 0
},
{
"pageSize": "1GB",
"limit": 0
}
]
},
"cgroupsPath": "kubepods-burstable-poda4477163_737d_457c_8698_8688fdd00003.slice:crio:27cccaa137e395427c3ddaef728528ddfd3e1d83cd2710a9d15fa1202e93e084",
"namespaces": [
{
"type": "pid"
},
{
"type": "network",
"path": "/var/run/netns/5b03e11b-6a0a-4027-b037-cb8aec0048c7"
},
{
"type": "ipc",
"path": "/var/run/ipcns/5b03e11b-6a0a-4027-b037-cb8aec0048c7"
},
{
"type": "uts",
"path": "/var/run/utsns/5b03e11b-6a0a-4027-b037-cb8aec0048c7"
},
{
"type": "mount"
}
],
"maskedPaths": [
"/proc/acpi",
"/proc/kcore",
"/proc/keys",
"/proc/latency_stats",
"/proc/timer_list",
"/proc/timer_stats",
"/proc/sched_debug",
"/proc/scsi",
"/sys/firmware"
],
"readonlyPaths": [
"/proc/asound",
"/proc/bus",
"/proc/fs",
"/proc/irq",
"/proc/sys",
"/proc/sysrq-trigger"
],
"mountLabel": "system_u:object_r:container_file_t:s0:c21,c0"
}
},
"privileged": false
}
}
sh-4.4#
→ Workerノード上のディレクトリパスを確認したいので、Pathでgrepする。
sh-4.4# crictl inspect 27cccaa137e39 | grep Path
"io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
"containerPath": "/prometheus",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volume-subpaths/pvc-b17ae945-02d2-43e2-a2d2-23891203b1f3/prometheus/3",
"containerPath": "/dev/termination-log",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/containers/prometheus/5ab56fc4",
"containerPath": "/etc/hosts",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/etc-hosts",
"containerPath": "/etc/prometheus/config_out",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~empty-dir/config-out",
"containerPath": "/etc/prometheus/certs",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~projected/tls-assets",
"containerPath": "/etc/prometheus/secrets/prometheus-k8s-thanos-sidecar-tls",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls",
"containerPath": "/etc/prometheus/secrets/kube-etcd-client-certs",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-kube-etcd-client-certs",
"containerPath": "/etc/prometheus/secrets/prometheus-k8s-tls",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls",
"containerPath": "/etc/prometheus/secrets/prometheus-k8s-proxy",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-prometheus-k8s-proxy",
"containerPath": "/etc/prometheus/web_config/web-config.yaml",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volume-subpaths/web-config/prometheus/5",
"containerPath": "/etc/prometheus/secrets/kube-rbac-proxy",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-kube-rbac-proxy",
"containerPath": "/etc/prometheus/secrets/metrics-client-certs",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~secret/secret-metrics-client-certs",
"containerPath": "/etc/prometheus/configmaps/serving-certs-ca-bundle",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-serving-certs-ca-bundle",
"containerPath": "/etc/prometheus/configmaps/kubelet-serving-ca-bundle",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-kubelet-serving-ca-bundle",
"containerPath": "/etc/prometheus/configmaps/metrics-client-ca",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/configmap-metrics-client-ca",
"containerPath": "/etc/prometheus/rules/prometheus-k8s-rulefiles-0",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/prometheus-k8s-rulefiles-0",
"containerPath": "/var/run/secrets/kubernetes.io/serviceaccount",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~projected/kube-api-access-jfzqj",
"containerPath": "/etc/pki/ca-trust/extracted/pem/",
"hostPath": "/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes/kubernetes.io~configmap/prometheus-trusted-ca-bundle",
"logPath": "/var/log/pods/openshift-monitoring_prometheus-k8s-0_a4477163-737d-457c-8698-8688fdd00003/prometheus/0.log"
"io.kubernetes.cri-o.Annotations": "{\"io.kubernetes.container.hash\":\"7ba9b53\",\"io.kubernetes.container.restartCount\":\"0\",\"io.kubernetes.container.terminationMessagePath\":\"/dev/termination-log\",\"io.kubernetes.container.terminationMessagePolicy\":\"FallbackToLogsOnError\",\"io.kubernetes.pod.terminationGracePeriod\":\"600\"}",
"io.kubernetes.cri-o.LogPath": "/var/log/pods/openshift-monitoring_prometheus-k8s-0_a4477163-737d-457c-8698-8688fdd00003/prometheus/0.log",
"io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
"io.kubernetes.cri-o.ResolvPath": "/run/containers/storage/overlay-containers/fde2304412307a44d51118bcf4d3ef4089fd71e5aa12418ca6716a6395194d7d/userdata/resolv.conf",
"io.kubernetes.cri-o.SeccompProfilePath": "",
"cgroupsPath": "kubepods-burstable-poda4477163_737d_457c_8698_8688fdd00003.slice:crio:27cccaa137e395427c3ddaef728528ddfd3e1d83cd2710a9d15fa1202e93e084",
"maskedPaths": [
"readonlyPaths": [
sh-4.4#
→ホスト（Workerノード）の/var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/ディレクトリを利用していることが分かる。
"root": {
"path": "/var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged"
},
他に、上記の部分を確認すると、/var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/mergedディレクトリを利用していることが分かる。
ディスク使用量の確認
Infra/Workerノード上の当該ディレクトリのサイズを確認する。
sh-4.4# du -sh /var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/*
0 /var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/containers
4.0K /var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/etc-hosts
4.0K /var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/plugins
14G /var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volume-subpaths
1.1M /var/lib/kubelet/pods/a4477163-737d-457c-8698-8688fdd00003/volumes
sh-4.4#
→ 約14GB 使用。
sh-4.4# du -sh /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/*
0 /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/bin
0 /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/boot
0 /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/dev
2.3M /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/etc
0 /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/home
0 /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/lib
0 /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/lib64
0 /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/lost+found
0 /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/media
0 /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/mnt
0 /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/opt
0 /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/proc
0 /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/prometheus
996K /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/root
0 /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/run
0 /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/sbin
0 /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/srv
0 /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/sys
8.0K /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/tmp
370M /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/usr
24M /var/lib/containers/storage/overlay/f41404e5d2a5c9bfef486091284a6192640ec6dbe4683230d7e4b9d117c58c84/merged/var
sh-4.4#
→ 約370MB 使用。
合計して、Prometheus 1podが約15GB程度使用していることが確認できた。
※参考：本ROSA環境は検証用の簡単なアプリケーションが数個稼働している程度の環境であり、Prometheusのretantionなどの設定はデフォルトのまま。
まとめ
Podのディスク使用量を確認するうえで、Podが使用するInfra/Workerノードのディレクトリ情報が必要になり、そのためにcrictlコマンドを利用し確認を行った。
Podのディスク使用量の確認方法については、他にもスマートなやり方がありそうだが、、探せなかったので、ひとまず本記事の方法で確認を行った。
参考
本記事ではストレージの利用に関しての説明は省略したが、OpenShiftのPodが利用するストレージについては以下マニュアルを参照のこと。
Understanding ephemeral storage
おまけ
ROSAのInfra/Workerノードはデフォルトで300GBのEBSが利用される。
現状では下記の量だけディスクが使用されており、今後他のPodが大量にディスクを消費するようなことがあるならPVの利用も検討すべきか。
sh-4.4# df -h /var/lib
Filesystem Size Used Avail Use% Mounted on
/dev/nvme0n1p4 300G 19G 281G 7% /var