OpenShift - Get Multi-Container Pod logs

oc logs

OpenShift で Pod の状況を確認する際に oc logs コマンドを使用することがあります。ここでは、Multi-Container Pod (複数の Container が稼働する Pod）のログ取得方法をご紹介します。

ここでは、5個（Init Container を含めると6個）の Container が稼働する kube-apiserver を例として使用してみます。

$ oc project -q
openshift-kube-apiserver

$ oc get pod kube-apiserver-rawoid-12bde
NAME                          READY   STATUS    RESTARTS   AGE
kube-apiserver-rawoid-12bde   5/5     Running   0          64m

$ oc -o json get pod kube-apiserver-rawoid-12bde  | jq -r '.spec |.initContainers[].name, .containers[].name'
setup
kube-apiserver
kube-apiserver-cert-syncer
kube-apiserver-cert-regeneration-controller
kube-apiserver-insecure-readyz
kube-apiserver-check-endpoints

--all-containers option

oc logs --all-containers を使用することで、Pod 内の Container のログを一括で取得することが出来ます。--prefix を併用することで Container Name 等の情報を付与します。

$ oc logs --prefix --all-containers kube-apiserver-rawoid-12bde > kube-apiserver-rawoid-12bde.LOG

$ head -n 5 kube-apiserver-rawoid-12bde.LOG
[pod/kube-apiserver-rawoid-12bde/setup] Fixing audit permissions ...
[pod/kube-apiserver-rawoid-12bde/setup] Acquiring exclusive lock /var/log/kube-apiserver/.lock ...
[pod/kube-apiserver-rawoid-12bde/setup] flock: getting lock took 0.000003 seconds
[pod/kube-apiserver-rawoid-12bde/kube-apiserver] flock: getting lock took 0.000004 seconds
[pod/kube-apiserver-rawoid-12bde/kube-apiserver] Copying system trust bundle ...

$ awk '{print $1}' kube-apiserver-rawoid-12bde.LOG | sort | uniq -c
  20395 [pod/kube-apiserver-rawoid-12bde/kube-apiserver]
     11 [pod/kube-apiserver-rawoid-12bde/kube-apiserver-cert-regeneration-controller]
     58 [pod/kube-apiserver-rawoid-12bde/kube-apiserver-cert-syncer]
     54 [pod/kube-apiserver-rawoid-12bde/kube-apiserver-check-endpoints]
      1 [pod/kube-apiserver-rawoid-12bde/kube-apiserver-insecure-readyz]
      3 [pod/kube-apiserver-rawoid-12bde/setup]

bash script

oc logs --all-containers ではログが1つのファイルに混在する事になります。別々のファイルに収集したい場合は、以下のような bash script で実装することが出来ます。

$ POD="kube-apiserver-rawoid-12bde"
$ for C in $(oc -o json get pod $POD | jq -r '.spec |.initContainers[].name, .containers[].name')
do
  oc logs --prefix -c $C $POD > $POD"_"$C".LOG"
done

$ ls -lt
total 7844
-rw-r--r-- 1 root root   18281 Oct 17 01:57 kube-apiserver-rawoid-12bde_kube-apiserver-check-endpoints.LOG
-rw-r--r-- 1 root root     171 Oct 17 01:57 kube-apiserver-rawoid-12bde_kube-apiserver-insecure-readyz.LOG
-rw-r--r-- 1 root root   13077 Oct 17 01:57 kube-apiserver-rawoid-12bde_kube-apiserver-cert-regeneration-controller.LOG
-rw-r--r-- 1 root root   51531 Oct 17 01:57 kube-apiserver-rawoid-12bde_kube-apiserver-cert-syncer.LOG
-rw-r--r-- 1 root root 7931297 Oct 17 01:57 kube-apiserver-rawoid-12bde_kube-apiserver.LOG
-rw-r--r-- 1 root root     355 Oct 17 01:57 kube-apiserver-rawoid-12bde_setup.LOG

$ head -n 1 *.LOG
==> kube-apiserver-rawoid-12bde_kube-apiserver-cert-regeneration-controller.LOG <==
[pod/kube-apiserver-rawoid-12bde/kube-apiserver-cert-regeneration-controller] W1017 00:23:59.863590       1 cmd.go:245] Using insecure, self-signed certificates

==> kube-apiserver-rawoid-12bde_kube-apiserver-cert-syncer.LOG <==
[pod/kube-apiserver-rawoid-12bde/kube-apiserver-cert-syncer] I1017 00:23:59.758428       1 base_controller.go:67] Waiting for caches to sync for CertSyncController

==> kube-apiserver-rawoid-12bde_kube-apiserver-check-endpoints.LOG <==
[pod/kube-apiserver-rawoid-12bde/kube-apiserver-check-endpoints] W1017 00:24:00.087320       1 cmd.go:245] Using insecure, self-signed certificates

==> kube-apiserver-rawoid-12bde_kube-apiserver-insecure-readyz.LOG <==
[pod/kube-apiserver-rawoid-12bde/kube-apiserver-insecure-readyz] I1017 00:23:59.989533       1 readyz.go:111] Listening on 0.0.0.0:6080

==> kube-apiserver-rawoid-12bde_kube-apiserver.LOG <==
[pod/kube-apiserver-rawoid-12bde/kube-apiserver] flock: getting lock took 0.000004 seconds

==> kube-apiserver-rawoid-12bde_setup.LOG <==
[pod/kube-apiserver-rawoid-12bde/setup] Fixing audit permissions ...