Kubernetes における稼働統計情報の取得には、Prometheus Metrics を使用する事が一般的です。

現在の Prometheus Metrics の情報は Grafana UI などを利用して簡単に参照することができますが、ある特定期間の稼働統計情報を参照したい時に不便な場合があります。

ここでは、Red Hat OpenShift Container Platform (OCP) 4.6 の Cluster Monitoring で、Prometheus Metrics から Linux vmstat コマンドのようなノード単位の CPU 使用率を取得する例をご紹介します。

Linux vmstat コマンド

Linux vmstat コマンドは、以下のような情報が取得できます。

$ vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 14748572   3236 659080    0    0    35     4   31   31  0  1 99  0  0

ここでは、vmstat コマンドで出力される us, sy, id, wa に相当する情報を Prometheus Metrics から取得してみます。

$ man vmstat
 :
FIELD DESCRIPTION FOR VM MODE
   CPU
       These are percentages of total CPU time.
       us: Time spent running non-kernel code.  (user time, including nice time)
       sy: Time spent running kernel code.  (system time)
       id: Time spent idle.  Prior to Linux 2.5.41, this includes IO-wait time.
       wa: Time spent waiting for IO.  Prior to Linux 2.5.41, included in idle.
       st: Time stolen from a virtual machine.  Prior to Linux 2.6.11, unknown.
 :

事前に oc login を実行済みの想定で、以下の要領で実施します。

使用する Metric の選択

Prometheus が取得している情報 ( scraped data ) から使用する Metric を選択します。
OCP Cluster Monitoring の Prometheus が取得している情報は、以下の要領で確認することができます。

$ TOKEN=$(oc whoami -t)
$ ROUTE=$(oc get route prometheus-k8s -n openshift-monitoring -o jsonpath='{.spec.host}')
$ curl -v -ks -G -L -H "Authorization: Bearer ${TOKEN}" "https://${ROUTE}/api/v1/metadata" > METADATA.json

ここでは、最も一般的な node_cpu_seconds_total を使用します。

$ jq -r '.data | with_entries(select(.key == "node_cpu_seconds_total"))' METADATA.json
{
  "node_cpu_seconds_total": [
    {
      "type": "counter",
      "help": "Seconds the cpus spent in each mode.",
      "unit": ""
    }
  ]
}

CPU 使用率の取得

ある特定期間の稼働統計情報の取得を想定して、Query Range API ( query_range ) を使用します。
ここでは、60秒毎の全 CPU の使用率平均を、日本時間の 2021/10/03 00:00:00 から 23:59:59 まで60秒単位で取得してみます。

$ TOKEN=$(oc whoami -t)
$ ROUTE=$(oc get route prometheus-k8s -n openshift-monitoring -o jsonpath='{.spec.host}')
$ START=$(date --date="2021-10-03 00:00:00+09:00" '+%s')
$ END=$(date --date="2021-10-03 23:59:59+09:00" '+%s')
$ curl -v -ks -G -L -H "Authorization: Bearer ${TOKEN}" "https://${ROUTE}/api/v1/query_range" \
--data-urlencode "query=avg without(cpu) (rate(node_cpu_seconds_total[60s]))" \
--data-urlencode "start=${START}" \
--data-urlencode "end=${END}" \
--data-urlencode "step=60" \
> CPU.json

HTTP Status Code が 200 で、JSON の .status が以下のようになっていれば、正常に取得されています。

$ jq -r '.status' CPU.json
success

取得した CPU 使用率 ( CPU.json ) の整形

取得した CPU 使用率の視認性を高めるために整形します。
ここでは、一般的に広く利用されている Microsoft Excel で使用可能な CSV に変換してみます。
CSV への変換は様々な実装が考えられますが、ここでは jq コマンドと awk コマンドを使用してみます。

JSON to TMP

jq コマンドで、CPU.json を一時ファイル ( CPU.tmp ) に変換します。

$ jq -r '.data.result[] |
.values[] += [(.metric.instance + " " + .metric.mode)] |
.metric.mode as $M | select($M == "user" or $M == "system" or $M == "iowait" or $M == "idle") |
.values[] | @csv' CPU.json > CPU.tmp

$ head CPU.tmp
1633186800,"0.3987222222300867","infra01 idle"
1633186860,"0.30188888889840904","infra01 idle"
1633186920,"0.5176840654579138","infra01 idle"
1633186980,"0.4833888888876471","infra01 idle"
1633187040,"0.4952777777694993","infra01 idle"
1633187100,"0.5275000000062088","infra01 idle"
1633187160,"0.4790342651359072","infra01 idle"
1633187220,"0.5096666666575604","infra01 idle"
1633187280,"0.49066666666832226","infra01 idle"
1633187340,"0.5019665355506373","infra01 idle"

TMP to CSV

awk コマンドで、CPU.tmp を CSV ファイル ( CPU.csv ) に変換します。

$ SEP="        " # Tab Character
$ TS="$(awk -F "," '{print $1}' CPU.tmp | sort -n | uniq | tr "\n" "$SEP")"
$ ES="$(awk -F "," '{print $3}' CPU.tmp | sort    | uniq | tr "\n" "$SEP")"
$ awk -v FS="," -v SEP="${SEP}" -v TS="${TS}" -v ES="${ES}" '
BEGIN {
OA[1][1]=""
split(TS, TA, SEP)
split(ES, EA, SEP)
}
{
for (i=1; i<=length(TA); i++){
        if ($1 == TA[i]){
                for(j=1; j<=length(EA); j++){ if($3 == EA[j]) { break } }
                OA[i][j]=$2
       }
}
}
END {
printf "%s,", "TIME"
for (i=1; i<=length(EA); i++){
        if (i == length(EA)){ S="\n" }else{ S="," }
        printf "%s%s", EA[i], S
}
for (i=1; i<=length(TA); i++){
        printf "\"%s\",", strftime("%y/%m/%d %H:%M:%S %Z", TA[i])
        for (j=1; j<=length(OA[i]); j++){
                if (j == length(OA[i])){ S="\n" }else{ S="," }
                printf "%s%s", OA[i][j], S
        }
}
}
' CPU.tmp > CPU.csv

$ head CPU.csv 
TIME,"infra01 idle","infra01 iowait","infra01 system","infra01 user","infra02 idle","infra02 iowait","infra02 system","infra02 user","infra03 idle","infra03 iowait","infra03 system","infra03 user","master01 idle","master01 iowait","master01 system","master01 user","master02 idle","master02 iowait","master02 system","master02 user","master03 idle","master03 iowait","master03 system","master03 user","worker01 idle","worker01 iowait","worker01 system","worker01 user","worker02 idle","worker02 iowait","worker02 system","worker02 user"
"21/10/03 00:00:00 JST","0.3987222222300867","0.0008333333333212067","0.08261111111108523","0.490222222223464","0.5200555555522441","0.0005555555555575765","0.04905555555532272","0.4120555555551416","0.7921666666658388","0.0004444444444440402","0.0662222222218083","0.11905555555519337","0.6671666666658388","0.0003888888888923247","0.06416666666692536","0.22533333333299702","0.8691111111133877","0.00044444444444656663","0.03433333333313284","0.07788888888866899","0.8824444444487906","0.0004999999999982821","0.03211111111110464","0.06711111111152503","0.921444444461829","0","0.0281111111107748","0.03888888889018239","0.9005555555638339","0.00011111111111353642","0.043444444443513114","0.04299999999994826"
"21/10/03 00:01:00 JST","0.30188888889840904","0.0013888888888888887","0.07783333333273831","0.5917777777804683","0.34643341410692885","0.004945873252269644","0.06990908484779448","0.5568275279535547","0.7985555555257531","0.0009444444444347431","0.06016666666708058","0.11938888889013065","0.7134205964717821","0.0025556691408496436","0.05789146184265802","0.18584159296007516","0.8725891031912415","0.003944093858323908","0.031330548395290744","0.0742156252776755","0.8868171918758154","0.0039446197608784634","0.027501222276448606","0.06550291124070655","0.9221111111135946","0.0002222222222220201","0.027555555555348596","0.03933333333245374","0.7872222222180829","0.0006666666666660603","0.06077777777746733","0.13211111111126633"
"21/10/03 00:02:00 JST","0.5176840654579138","0.0007224791036869701","0.06107727191885274","0.4015316556998044","0.5281111111243565","0.0005000000000109139","0.045722222221471986","0.4098333333334368","0.8037222222218082","0.00038888888889737747","0.06111111111175785","0.11461111110919672","0.7263888888888888","0.000444444444442777","0.05755555555547795","0.17916666666666667","0.8828333333341611","0.0005000000000008084","0.031722222222338635","0.06722222222209287","0.8913333333304359","0.0005555555555550502","0.028722222222293363","0.0634444444443539","0.9228888888739877","0.00011111111111101006","0.027444444444780752","0.038555555555245105","0.9017777777794335","0","0.04222222222273962","0.04388888888837149"
"21/10/03 00:03:00 JST","0.4833888888876471","0.0007222222222177759","0.061055555555503814","0.43138888888837146","0.5196111111073858","0.0005000000000008084","0.04583333333333334","0.41738888888826803","0.7861111111111111","0.000388888888887272","0.06411111111131806","0.1273333333342129","0.6887777777802613","0.00033333333333429333","0.06394444444449618","0.2056666666667701","0.8672777777744665","0.0005000000000008084","0.03327777777772604","0.08055555555555555","0.88966666665963","0.0005555555555550503","0.02944444444421808","0.06344444444435389","0.9211111110945542","0.00011111111111101006","0.027555555555348592","0.03988888888982021","0.8808888888814382","0.00011111111111353642","0.04433333333322985","0.06044444444382356"
"21/10/03 00:04:00 JST","0.4952777777694993","0.0008333333333414177","0.058222222222118744","0.4263888888888889","0.5219444444506532","0.0006111111111143448","0.04877777777793299","0.41172222222408483","0.7915555555476911","0.00027777777778384107","0.06449999999992238","0.12183333333426466","0.7075555555625923","0.0002777777777800515","0.0605000000000776","0.19288888888873368","0.875833333325055","0.0006111111111118184","0.032500000000032336","0.07088888888861725","0.8841666666635623","0.0005000000000033347","0.03172222222217695","0.06511111111127926","0.921847917877767","0.00022221234611774818","0.027665437090934615","0.03955379760832256","0.9019999999966886","0.00011111111110848367","0.042222222221446115","0.04311111111116285"
"21/10/03 00:05:00 JST","0.5275000000062088","0.0007777777777644385","0.05744444444426335","0.39566666666521794","0.5479444444489976","0.0005555555555575766","0.043166666666770145","0.39377777777715683","0.8363888888930281","0.00044444444443393464","0.05683333333387659","0.0888333333332816","0.7460000000066227","0.00027777777777752516","0.053444444444741945","0.16722222222209288","0.8853888888895096","0.0003333333333330301","0.029333333333488552","0.06727777777802355","0.9012777777802613","0.0003333333333330301","0.02677777777781658","0.05727777777776484","0.9241111111102832","0.00022222222222454646","0.02722222222170482","0.03844444444403052","0.9011111111069718","0","0.041777777777881255","0.04322222222237744"
"21/10/03 00:06:00 JST","0.4790342651359072","0.0006110839518275969","0.06105284209585511","0.43731389716126445","0.5289444444494115","0.0006111111111042393","0.04483333333304876","0.4102777777782952","0.7868333333378865","0.00033333333334060924","0.06505555555599535","0.12544444444356487","0.7128333333414048","0.000333333333331767","0.05872222222161427","0.1900555555553486","0.8783333333364376","0.0005000000000008084","0.03216666666671194","0.06933333333322984","0.8877222222172552","0.0005555555555525238","0.029499999999987064","0.06611111111091708","0.9258888888897168","0","0.025888888888423226","0.037777777778036474","0.8784444444295432","0.00011111111110848367","0.045555555555296855","0.06077777777746733"
"21/10/03 00:07:00 JST","0.5096666666575604","0.001055555555558385","0.0599444444446514","0.4101666666670805","0.5377222222172551","0.0005555555555575766","0.04461111111191308","0.40133333333198806","0.8203333333238131","0.00038888888888727194","0.05938888888857844","0.10088888888971674","0.7105555555627991","0.00033333333333429333","0.058944444444690214","0.19400000000023282","0.8642222222199456","0.00044444444444151387","0.03427777777784892","0.08083333333326866","0.8942222222220153","0.00044444444444404017","0.029000000000006465","0.06072222222250679","0.9253333333258826","0.00011111111111101006","0.02644444444449618","0.037999999999172154","0.857333333314293","0","0.05233333333421292","0.07522222222372268"
"21/10/03 00:08:00 JST","0.49066666666832226","0.0006666666666711131","0.06177777777839866","0.4258888888897167","0.5351111111024188","0.0005000000000008085","0.048722222222972456","0.40133333333716203","0.8003888888905446","0.0003333333333406092","0.05966666666661492","0.11811111111245637","0.6932777777779847","0.0002777777777787883","0.06127777777728625","0.20516666666630448","0.875","0.00038888888889232476","0.032611111111085236","0.07238888888839734","0.8914444444442374","0.0004444444444465666","0.029222222221950582","0.06322222222192471","0.9244444444361659","0.00011111111111101006","0.026888888888707798","0.038333333332815925","0.8721111111135946","0","0.049555555556435135","0.06355555555524511"

これで、作成した CSV を Microsoft Excel で使用することができます。
以下は、Microsoft Excel で CPU 使用率を積み上げグラフにしてみたものです。