Kubernetes Auditingで誰がなにをしたのかをみてみる

はじめに

本番環境やそれに相当するようなセンシティブな環境では、監査要件として

『誰がどのようなコマンドを打ったか』

を収集することがよくある。
自分の場合、大体は（内部向け説明がしやすい）従来の運用方法を踏襲するような形で上記要件をクリアする（してしまう）が、KubernetesのAuditingを使うことで似たような情報を収集することができる。
（「どのようなコマンドか」ではなく「どのようなリクエストか」を収集）

[https://kubernetes.io/docs/tasks/debug-application-cluster/audit/:embed:cite]
どのような出力がされるのか、試しにみてみる。

環境

• minikube v1.9.2

疎通

[https://minikube.sigs.k8s.io/docs/tutorials/audit-policy/:title]

AuditPolicyの作成

minikube stop
mkdir -p ~/.minikube/files/etc/ssl/certs
cat <<EOF > ~/.minikube/files/etc/ssl/certs/audit-policy.yaml
# Log all requests at the Metadata level.
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata
EOF

minikubeの起動

minikube start \
  --extra-config=apiserver.audit-policy-file=/etc/ssl/certs/audit-policy.yaml \
  --extra-config=apiserver.audit-log-path=-

Audit logの確認

kubectl logs kube-apiserver-minikube -n kube-system | grep audit.k8s.io/v1

出力の例

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Metadata",
  "auditID": "e8b44217-4018-4cac-b25a-978349d428e0",
  "stage": "RequestReceived",
  "requestURI": "/apis/batch/v1beta1/cronjobs?limit=500",
  "verb": "list",
  "user": {
    "username": "system:serviceaccount:kube-system:cronjob-controller",
    "uid": "5eb6b4b2-a6e3-4551-800f-1d1aeb638e0f",
    "groups": [
      "system:serviceaccounts",
      "system:serviceaccounts:kube-system",
      "system:authenticated"
    ]
  },
  "sourceIPs": [
    "192.168.99.100"
  ],
  "userAgent": "kube-controller-manager/v1.18.0 (linux/amd64) kubernetes/9e99141/system:serviceaccount:kube-system:cronjob-controller",
  "objectRef": {
    "resource": "cronjobs",
    "apiGroup": "batch",
    "apiVersion": "v1beta1"
  },
  "requestReceivedTimestamp": "2020-04-22T01:56:52.416692Z",
  "stageTimestamp": "2020-04-22T01:56:52.416692Z"
}

grepの条件をところをminikube-userに変えてみる。

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Metadata",
  "auditID": "105f9973-cbca-49ed-9d6b-41e232225738",
  "stage": "ResponseStarted",
  "requestURI": "/api/v1/namespaces/kube-system/pods/kube-apiserver-minikube/log?follow=true",
  "verb": "get",
  "user": {
    "username": "minikube-user",
    "groups": [
      "system:masters",
      "system:authenticated"
    ]
  },
  "sourceIPs": [
    "192.168.99.1"
  ],
  "userAgent": "kubectl/v1.15.3 (linux/amd64) kubernetes/2d3c76f",
  "objectRef": {
    "resource": "pods",
    "namespace": "kube-system",
    "name": "kube-apiserver-minikube",
    "apiVersion": "v1",
    "subresource": "log"
  },
  "responseStatus": {
    "metadata": {},
    "code": 200
  },
  "requestReceivedTimestamp": "2020-04-22T02:00:11.412922Z",
  "stageTimestamp": "2020-04-22T02:00:11.424775Z",
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": ""
  }
}

kubectl logsを打っているので、その情報が出力されていることがわかる。
出力する情報を増やしたいので、minikube-userのみRequestレベルのAudit Logを出力するようaudit-policy.yamlを修正し、Minikubeを再起動。

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Request
  users: ["minikube-user"]

試しにkubectl createしてみる

kubectl create deployment --image=nginx nginx-app

上記コマンドを実行すると、Audit logは下記のようになり、

{
    "kind": "Event",
    "apiVersion": "audit.k8s.io/v1",
    "level": "Request",
    "auditID": "2e935f75-6162-479e-a4b6-1bd311a4ff86",
    "stage": "ResponseComplete",
    "requestURI": "/apis/apps/v1/namespaces/default/deployments",
    "verb": "create",
    "user": {
        "username": "minikube-user",
        "groups": [
            "system:masters",
            "system:authenticated"
        ]
    },
    "sourceIPs": [
        "192.168.99.1"
    ],
    "userAgent": "kubectl/v1.15.3 (linux/amd64) kubernetes/2d3c76f",
    "objectRef": {
        "resource": "deployments",
        "namespace": "default",
        "name": "nginx-app",
        "apiGroup": "apps",
        "apiVersion": "v1"
    },
    "responseStatus": {
        "metadata": {},
        "code": 201
    },
    "requestObject": {
        "kind": "Deployment",
        "apiVersion": "apps/v1",
        "metadata": {
            "name": "nginx-app",
            "creationTimestamp": null,
            "labels": {
                "app": "nginx-app"
            }
        },
        "spec": {
            "replicas": 1,
            "selector": {
                "matchLabels": {
                    "app": "nginx-app"
                }
            },
            "template": {
                "metadata": {
                    "creationTimestamp": null,
                    "labels": {
                        "app": "nginx-app"
                    }
                },
                "spec": {
                    "containers": [
                        {
                            "name": "nginx",
                            "image": "nginx",
                            "resources": {},
                            "terminationMessagePath": "/dev/termination-log",
                            "terminationMessagePolicy": "File",
                            "imagePullPolicy": "Always"
                        }
                    ],
                    "restartPolicy": "Always",
                    "terminationGracePeriodSeconds": 30,
                    "dnsPolicy": "ClusterFirst",
                    "securityContext": {},
                    "schedulerName": "default-scheduler"
                }
            },
            "strategy": {
                "type": "RollingUpdate",
                "rollingUpdate": {
                    "maxUnavailable": "25%",
                    "maxSurge": "25%"
                }
            },
            "revisionHistoryLimit": 10,
            "progressDeadlineSeconds": 600
        },
        "status": {}
    },
    "requestReceivedTimestamp": "2020-04-22T02:05:52.184886Z",
    "stageTimestamp": "2020-04-22T02:05:52.203978Z",
    "annotations": {
        "authorization.k8s.io/decision": "allow",
        "authorization.k8s.io/reason": ""
    }
}

• .user.username（minikube-user）が
• .requestObject（Deployment)を.verb（create）した

ことがわかる。

上記ではAudit LevelをRequestにしたが、他にもあるのでいろいろ試してみると良いと思う。

Level	内容
None	don’t log events that match this rule.
Metadata	log request metadata (requesting user, timestamp, resource, verb, etc.) but not request or response body.
Request	log event metadata and request body but not response body. This does not apply for non-resource requests.
RequestResponse	log event metadata, request and response bodies. This does not apply for non-resource requests.