本記事はOpenshift 4.6のMaster/Worker Nodeのリソース消費量について実際のコマンド出力と共に簡易的にまとめたものです。
OpenshiftクラスターをデプロイしOpenshift image registryをManagedに設定した位の状態です。
サイジングする上でCoreOSやシステム系Podが予約・消費しているリソース量について考慮漏れになることがあるため、参考程度に載せておきます。
構成
- vSphere UPIデプロイパターン
- Master 3 Node/Worker 2 Node構成
[root@prov ~]# oc get nodes
NAME STATUS ROLES AGE VERSION
master1 Ready master 45d v1.19.14+fcff70a
master2 Ready master 45d v1.19.14+fcff70a
master3 Ready master 45d v1.19.14+fcff70a
worker1 Ready app,worker 45d v1.19.14+fcff70a
worker2 Ready app,worker 45d v1.19.14+fcff70a
[root@prov ~]# oc version
Client Version: 4.7.34
Server Version: 4.6.49
Kubernetes Version: v1.19.14+fcff70a
[root@prov ~]#
-
各Master Nodeへのリソース搭載量
- CPU：8
- MEM：8GB
- Disk：100GB
-
各Worker Nodeへのリソース搭載量
- CPU：8
- MEM：8GB
- Disk：100GB
※上記リソース量は適当に割り当てており、管理するノード数やデプロイするOperatorやアプリケーションPod数により大きく変わるため、Red Hatのサイトで適切なリソース量を計算すること
oc describeコマンド抜粋
- Master Node
[root@prov ~]# oc describe node master1
Name: master1
Roles: master
Capacity:
cpu: 8
ephemeral-storage: 104305644Ki
hugepages-2Mi: 0
memory: 8160732Ki
pods: 250
Allocatable:
cpu: 7500m
ephemeral-storage: 95054339528
hugepages-2Mi: 0
memory: 7009756Ki
pods: 250
ProviderID: vsphere://423d9531-0c8e-1964-5d5d-8f35fc223822
Non-terminated Pods: (31 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
openshift-apiserver apiserver-5648856c64-zlx95 110m (1%) 0 (0%) 250Mi (3%) 0 (0%) 45d
openshift-authentication oauth-openshift-7565c7b94d-2kt7n 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 45d
openshift-cluster-node-tuning-operator tuned-nlfq5 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 45d
openshift-cluster-samples-operator cluster-samples-operator-7dbc7958f-6ltt8 20m (0%) 0 (0%) 0 (0%) 0 (0%) 45d
openshift-cluster-storage-operator csi-snapshot-controller-5458888996-4zlbd 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 45d
openshift-console downloads-9776476c4-pd6xm 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 45d
openshift-controller-manager controller-manager-qdzgz 100m (1%) 0 (0%) 100Mi (1%) 0 (0%) 47h
openshift-dns dns-default-4tbbp 65m (0%) 0 (0%) 110Mi (1%) 512Mi (7%) 45d
openshift-etcd etcd-master1 430m (5%) 0 (0%) 860Mi (12%) 0 (0%) 45d
openshift-etcd etcd-quorum-guard-78d56fdc5b-lnc77 10m (0%) 0 (0%) 5Mi (0%) 0 (0%) 45d
openshift-image-registry node-ca-j6cv5 10m (0%) 0 (0%) 10Mi (0%) 0 (0%) 45d
openshift-kube-apiserver-operator kube-apiserver-operator-6cc8455974-nldsv 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 45d
openshift-kube-apiserver kube-apiserver-master1 340m (4%) 0 (0%) 1224Mi (17%) 0 (0%) 47h
openshift-kube-controller-manager kube-controller-manager-master1 100m (1%) 0 (0%) 500Mi (7%) 0 (0%) 45d
openshift-kube-scheduler openshift-kube-scheduler-master1 25m (0%) 0 (0%) 150Mi (2%) 0 (0%) 45d
openshift-machine-api cluster-autoscaler-operator-69876758dd-6bhfw 30m (0%) 0 (0%) 70Mi (1%) 0 (0%) 45d
openshift-machine-api machine-api-controllers-654d4df8c9-q8plc 70m (0%) 0 (0%) 140Mi (2%) 0 (0%) 45d
openshift-machine-api machine-api-operator-5ffdfc86c-fk5ct 20m (0%) 0 (0%) 70Mi (1%) 0 (0%) 45d
openshift-machine-config-operator machine-config-controller-656f69c9bb-842xs 20m (0%) 0 (0%) 50Mi (0%) 0 (0%) 45d
openshift-machine-config-operator machine-config-daemon-qh79g 40m (0%) 0 (0%) 100Mi (1%) 0 (0%) 45d
openshift-machine-config-operator machine-config-server-cz6zv 20m (0%) 0 (0%) 50Mi (0%) 0 (0%) 45d
openshift-monitoring node-exporter-hlgvc 9m (0%) 0 (0%) 210Mi (3%) 0 (0%) 45d
openshift-multus multus-5mb6t 10m (0%) 0 (0%) 150Mi (2%) 0 (0%) 45d
openshift-multus multus-admission-controller-kn29z 20m (0%) 0 (0%) 20Mi (0%) 0 (0%) 45d
openshift-multus network-metrics-daemon-c8vgz 20m (0%) 0 (0%) 120Mi (1%) 0 (0%) 45d
openshift-oauth-apiserver apiserver-698fdcc744-rqnh5 150m (2%) 0 (0%) 200Mi (2%) 0 (0%) 45d
openshift-operator-lifecycle-manager packageserver-75b9bddfdd-blt8s 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 45d
openshift-sdn ovs-fvnpm 100m (1%) 0 (0%) 400Mi (5%) 0 (0%) 45d
openshift-sdn sdn-controller-x2csg 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 45d
openshift-sdn sdn-d5jvw 110m (1%) 0 (0%) 220Mi (3%) 0 (0%) 45d
openshift-service-ca service-ca-5ff99fbc4c-zs9w9 10m (0%) 0 (0%) 120Mi (1%) 0 (0%) 45d
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 1909m (25%) 0 (0%)
memory 5479Mi (80%) 512Mi (7%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
- Worker Node
[root@prov ~]# oc describe node worker1
Name: worker1
Roles: app,worker
Capacity:
cpu: 8
ephemeral-storage: 104305644Ki
hugepages-2Mi: 0
memory: 10225132Ki
pods: 250
Allocatable:
cpu: 7500m
ephemeral-storage: 95054339528
hugepages-2Mi: 0
memory: 9074156Ki
pods: 250
ProviderID: vsphere://423da5a4-0e4e-35f5-07a5-e2bf8e003d88
Non-terminated Pods: (21 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
openshift-cluster-node-tuning-operator tuned-7b2mg 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 45d
openshift-dns dns-default-xftxt 65m (0%) 0 (0%) 110Mi (1%) 512Mi (5%) 45d
openshift-image-registry image-registry-549fc8689d-j9cmz 100m (1%) 0 (0%) 256Mi (2%) 0 (0%) 28d
openshift-image-registry node-ca-rfxtk 10m (0%) 0 (0%) 10Mi (0%) 0 (0%) 45d
openshift-ingress router-default-76fd9bbc66-ldqld 100m (1%) 0 (0%) 256Mi (2%) 0 (0%) 28d
openshift-kube-storage-version-migrator migrator-68bc7c89f-9wx22 100m (1%) 0 (0%) 200Mi (2%) 0 (0%) 28d
openshift-machine-config-operator machine-config-daemon-ts248 40m (0%) 0 (0%) 100Mi (1%) 0 (0%) 45d
openshift-monitoring alertmanager-main-0 8m (0%) 0 (0%) 270Mi (3%) 0 (0%) 47h
openshift-monitoring grafana-575fbc5464-n4dz9 5m (0%) 0 (0%) 120Mi (1%) 0 (0%) 28d
openshift-monitoring kube-state-metrics-69b4fbb79-n4s5s 4m (0%) 0 (0%) 120Mi (1%) 0 (0%) 28d
openshift-monitoring node-exporter-rs75q 9m (0%) 0 (0%) 210Mi (2%) 0 (0%) 45d
openshift-monitoring openshift-state-metrics-7fb987f599-4vgxr 3m (0%) 0 (0%) 190Mi (2%) 0 (0%) 28d
openshift-monitoring prometheus-adapter-84dc49f498-8p24m 1m (0%) 0 (0%) 25Mi (0%) 0 (0%) 47h
openshift-monitoring prometheus-k8s-1 75m (1%) 0 (0%) 1194Mi (13%) 0 (0%) 47h
openshift-monitoring telemeter-client-97449bd78-rp7b4 3m (0%) 0 (0%) 20Mi (0%) 0 (0%) 28d
openshift-monitoring thanos-querier-69875575db-fsc9d 9m (0%) 0 (0%) 92Mi (1%) 0 (0%) 28d
openshift-monitoring thanos-querier-69875575db-hmw7g 9m (0%) 0 (0%) 92Mi (1%) 0 (0%) 28d
openshift-multus multus-hmwxt 10m (0%) 0 (0%) 150Mi (1%) 0 (0%) 45d
openshift-multus network-metrics-daemon-z6tsz 20m (0%) 0 (0%) 120Mi (1%) 0 (0%) 45d
openshift-sdn ovs-lbfnl 100m (1%) 0 (0%) 400Mi (4%) 0 (0%) 45d
openshift-sdn sdn-qxvxk 110m (1%) 0 (0%) 220Mi (2%) 0 (0%) 45d
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 791m (10%) 0 (0%)
memory 4205Mi (47%) 512Mi (5%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
[root@prov ~]#
リソース考慮点
- 各ノード、500mのCPUリソースが予約されている。（恐らくCoreOS分）
- Openshiftは 1Core = 1000m で計算
- Allocatable にて 7500m となっている。(8000m - 500m = 7500m)
- 各ノード、約1GBのMEMリソースが予約されている。（恐らくCoreOS分）
- リソースクォータ値について
- 上記出力結果より、CPUクォータに Master Nodeは1909m、Worker Nodeは791mが設定されている。
- 上記出力結果より、MEMクォータに Master Nodeは5479Mi、Worker Nodeは4205Miが設定されている。
- デプロイするOperatorによりリソース消費量は大きく変わるため余裕を持ったサイジングが必要
- Openshift Logging(Elasticsearch/Fluentd/Kibana) Operatorをデプロイすると更にリソース予約・消費量が上昇する。
- ログ流量によってリソース消費量が大きく異なるため要注意
- Prometheusについては収集対象の数により異なるためアプリケーションコンテナが多くなる環境は要注意
- Compliance、File Integrity Operatorなどは全てのNodeでPodが起動するため要注意（インフラノードを用意してもスキャン用Podが起動する）
- Openshift Logging(Elasticsearch/Fluentd/Kibana) Operatorをデプロイすると更にリソース予約・消費量が上昇する。
Openshiftには様々なOperatorが存在しており、どれだけのリソースが消費されるかは不明であるため、本番環境への利用を検討されている場合はPoCを実施するのが無難である。
以上