目的
- kube-bench を理解する
手段
KillerCoda で手を動かす
環境
KillerCoda
やってみる
基本形は以下の通りです。
$ kube-bench run -s master
# -sオプションでコマンド実行対象を指定
特定の CIS を確認するコマンドは以下の通りです。
$ kube-bench run -s master --check 1.1.19
[INFO] 1 Control Plane Security Configuration
[INFO] 1.1 Control Plane Node Configuration Files
[PASS] 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated)
== Summary master ==
1 checks PASS
0 checks FAIL
0 checks WARN
0 checks INFO
== Summary total ==
1 checks PASS
0 checks FAIL
0 checks WARN
0 checks INFO
# 1.1.19 は PASS している
FAIL している場合は以下の通りです。
$ kube-bench run -s master --check 1.1.12
[INFO] 1 Control Plane Security Configuration
[INFO] 1.1 Control Plane Node Configuration Files
[FAIL] 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated)
== Remediations master ==
1.1.12 On the etcd server node, get the etcd data directory, passed as an argument --data-dir,
from the command 'ps -ef | grep etcd'.
Run the below command (based on the etcd data directory found above).
For example, chown etcd:etcd /var/lib/etcd
== Summary master ==
0 checks PASS
1 checks FAIL
0 checks WARN
0 checks INFO
== Summary total ==
0 checks PASS
1 checks FAIL
0 checks WARN
0 checks INFO
# Remediations master の説明に沿って対処すれば PASS できます
kube-bench 実行対象は以下の通りです。
controlplane:~$ kube-bench run -s kubelet
The specified --targets "kubelet" are not configured for the CIS Benchmark cis-1.10\n Valid targets [master node controlplane etcd policies]
合わせてコマンドヘルプも確認します。
controlplane:~$ kube-bench run -h |more
Run tests. If no arguments are specified, runs tests from all files
Usage:
kube-bench run [flags]
Flags:
-h, --help help for run
-s, --targets strings Specify targets of the benchmark to run. These names need to match the filenames in the cfg/<version> directory.
For example, to run the tests specified in master.yaml and etcd.yaml, specify --targets=master,etcd
If no targets are specified, run tests from all files in the cfg/<version> directory.
cfg/<version> directory はどこを差しているでしょうか？確認します。
controlplane:~$ find / -name master.yaml
/etc/kube-bench/cfg/gke-1.6.0/master.yaml
--- snip ---
kubelet はスキャン対象外なのでしょうか？kube-bench 自体のコンフィグを読んでみます。
controlplane:/etc/kube-bench/cfg$ grep kubelet config.yaml
- kubelet
kubelet:
- "hyperkube kubelet"
- "kubelet"
- kubelet
kubelet:
- "/etc/systemd/system/kubelet.service.d/10-kubeadm.conf"
- "/etc/systemd/system/kubelet.service"
- "/lib/systemd/system/kubelet.service"
- "/etc/systemd/system/snap.kubelet.daemon.service"
- "/etc/systemd/system/snap.microk8s.daemon-kubelet.service"
- "hyperkube kubelet"
- "kubelet"
- "/etc/kubernetes/kubelet.conf"
--- snip ---
/etc/kubernetes/kubelet.conf はスキャン対象ですね。つまり kube-bench run -s master を指定すれば、自動的に kubelet もスキャンされるはずです。確認します。
controlplane:/etc/kube-bench/cfg$ kube-bench run -s master |grep -A5 -B5 kubelet
[PASS] 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Manual)
[INFO] 1.2 API Server
[WARN] 1.2.1 Ensure that the --anonymous-auth argument is set to false (Manual)
[PASS] 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated)
[WARN] 1.2.3 Ensure that the --DenyServiceExternalIPs is set (Manual)
[PASS] 1.2.4 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated)
[FAIL] 1.2.5 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated)
[PASS] 1.2.6 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated)
[PASS] 1.2.7 Ensure that the --authorization-mode argument includes Node (Automated)
[PASS] 1.2.8 Ensure that the --authorization-mode argument includes RBAC (Automated)
[WARN] 1.2.9 Ensure that the admission control plugin EventRateLimit is set (Manual)
[PASS] 1.2.10 Ensure that the admission control plugin AlwaysAdmit is not set (Automated)
--
1.2.3 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml
on the control plane node and add the `DenyServiceExternalIPs` plugin
to the enabled admission plugins, as such --enable-admission-plugin=DenyServiceExternalIPs.
1.2.5 Follow the Kubernetes documentation and setup the TLS connection between
the apiserver and kubelets. Then, edit the API server pod specification file
/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the
--kubelet-certificate-authority parameter to the path to the cert file for the certificate authority.
--kubelet-certificate-authority=<ca-string>
1.2.9 Follow the Kubernetes documentation and set the desired limits in a configuration file.
Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml
and set the below parameters.
--enable-admission-plugins=...,EventRateLimit,...
ドンピシャで kubelet 側の問題を発見できていませんが、環境依存だと推測します。
あとがき
サービス問題ですね。