1. はじめに
VaultのAgent Sidecar Injectorの動作確認を行った際の手順を記載したもの
2. Vault Agent Sidecar Injector
- https://developer.hashicorp.com/vault/docs/platform/k8s/injector
- https://www.hashicorp.com/blog/kubernetes-vault-integration-via-sidecar-agent-injector-vs-csi-provider
2.1. 動作概要
- App Pod起動時に、initContainerやsidecarとして、Agent containerが起動する
- Agent containerが、app用のService Accountを使ってVaultに認証し、tokenを取得する
- Agent containerが、tokenを使ってVaultからsecretを取得する
- Agent containerが取得した Secret情報は、App Container内のshared memory volume(共有ボリューム, /vault/secrets)にファイルとして格納される
- App Container内のアプリが、共有ボリューム内のファイルを参照し、Secret情報を使用する
2.2. 動作に必要なリソース
- App Pod - Secretを利用する側
- Namespace
- Service Account
- Deployment(Pod)
- Vault - Secretを格納する側
- Secret
- Policy
- Auth Method
3. 動作確認の流れ
以下の流れで動作確認を実施した。
- k8s clusterの構築
- App Podの事前準備(Namespace, Service Account)
- Vaultの構築
- Vaultの設定(Secret, Policy, Auth Method)
- App Podのデプロイ
4. 手順詳細
動作確認にk8s(minikube)を使用したため、環境がない場合は下記を参考に構築しておく。
4.1. k8s clusterの構築
検証用のk8s clusterを構築する。
minikubeを起動する。
コマンド
$ minikube start --memory='4g' --cpus=4
$ alias kubectl="minikube kubectl --"
$ kubectl get pod,svc -A
実行結果を展開する
実行結果
ubuntu@ubuntu:~$ minikube start --memory='4g' --cpus=4
😄 minikube v1.33.0 on Ubuntu 24.04 (vbox/amd64)
✨ Automatically selected the docker driver
📌 Using Docker driver with root privileges
👍 Starting "minikube" primary control-plane node in "minikube" cluster
🚜 Pulling base image v0.0.43 ...
🔥 Creating docker container (CPUs=4, Memory=4096MB) ...
🐳 Preparing Kubernetes v1.30.0 on Docker 26.0.1 ...
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
🔗 Configuring bridge CNI (Container Networking Interface) ...
🔎 Verifying Kubernetes components...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟 Enabled addons: default-storageclass, storage-provisioner
💡 kubectl not found. If you need it, try: 'minikube kubectl -- get pods -A'
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ alias kubectl="minikube kubectl --"
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ kubectl get pod,svc -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system pod/coredns-7db6d8ff4d-mccl5 1/1 Running 0 35s
kube-system pod/etcd-minikube 1/1 Running 0 48s
kube-system pod/kube-apiserver-minikube 1/1 Running 0 49s
kube-system pod/kube-controller-manager-minikube 1/1 Running 0 53s
kube-system pod/kube-proxy-hfr9j 1/1 Running 0 35s
kube-system pod/kube-scheduler-minikube 1/1 Running 0 49s
kube-system pod/storage-provisioner 1/1 Running 1 (3s ago) 45s
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 50s
kube-system service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 50s
ubuntu@ubuntu:~$
4.2. App Podの事前準備
Secretを参照するApp Podが使用するリソースを作成する。
Vaultの設定でこのパラメータが必要となるため事前に作成したが、
パラメータが決まっていれば、実際にリソースがなくてもVault上で設定は可能。
NamespaceとService Accountを作成する。
コマンド
$ kubectl create ns app-ns
$ kubectl create sa app-sa -n app-ns
実行結果を展開する
実行結果
ubuntu@ubuntu:~$ kubectl create ns app-ns
namespace/app-ns created
ubuntu@ubuntu:~$ kubectl create sa app-sa -n app-ns
serviceaccount/app-sa created
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ kubectl get ns
NAME STATUS AGE
app-ns Active 56m
default Active 57m
kube-node-lease Active 57m
kube-public Active 57m
kube-system Active 57m
vault Active 52m
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ kubectl get sa -n app-ns
NAME SECRETS AGE
app-sa 0 56m
default 0 56m
ubuntu@ubuntu:~$
4.3. Vault構築
Secretを管理するVaultを構築する。
VaultはApp Podとは別のNamespaceを使用する。
4.3.1. Vaultインストール
Helmからインストールする。
コマンド
$ helm repo add hashicorp https://helm.releases.hashicorp.com
$ helm repo update
$ helm search repo hashicorp/vault
$ helm install vault hashicorp/vault -n vault --create-namespace
実行結果を展開する
実行結果
ubuntu@ubuntu:~$ helm repo add hashicorp https://helm.releases.hashicorp.com
"hashicorp" has been added to your repositories
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "hashicorp" chart repository
Update Complete. ⎈Happy Helming!⎈
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ helm search repo hashicorp/vault
NAME CHART VERSION APP VERSION DESCRIPTION
hashicorp/vault 0.28.0 1.16.1 Official HashiCorp Vault Chart
hashicorp/vault-secrets-operator 0.6.0 0.6.0 Official Vault Secrets Operator Chart
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ helm install vault hashicorp/vault -n vault --create-namespace
NAME: vault
LAST DEPLOYED: Sun May 5 07:28:48 2024
NAMESPACE: vault
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing HashiCorp Vault!
Now that you have deployed Vault, you should look over the docs on using
Vault with Kubernetes available here:
https://developer.hashicorp.com/vault/docs
Your release is named vault. To learn more about the release, try:
$ helm status vault
$ helm get manifest vault
ubuntu@ubuntu:~$
4.3.2. Vault初期設定
Vaultインストール時点では、下記の状態(READYが0/1)で、初期設定が必要となるため実施する。
実行結果
ubuntu@ubuntu:~$ kubectl get pod -n vault
NAME READY STATUS RESTARTS AGE
pod/vault-0 0/1 Running 0 111s
pod/vault-agent-injector-ff58f5d77-5ggf9 1/1 Running 0 111s
ubuntu@ubuntu:~$
下記コマンドを実行する。
コマンド
$ kubectl exec -it vault-0 -n vault -- vault operator init
$ kubectl exec -it vault-0 -n vault -- vault operator unseal <Unseal Key 1>
$ kubectl exec -it vault-0 -n vault -- vault operator unseal <Unseal Key 2>
$ kubectl exec -it vault-0 -n vault -- vault operator unseal <Unseal Key 3>
vault operator initコマンド実行後に出力されるInitial Root Tokenは後で使用するため控えておく
実行結果を展開する
実行結果
ubuntu@ubuntu:~$ kubectl exec -it vault-0 -n vault -- vault operator init
Unseal Key 1: y1DN06xWAGLZcNPi6T4Ny6qlI2yaMxvZfGodHejuggom
Unseal Key 2: +T0IZltQR1iQOsgN7wSFPaSS9MyQreF+fhzgGpfuiLXs
Unseal Key 3: ESyeZ/iW4S8aws4bxnWQQZIXtEnN/jAvlbBI1rIu0L27
Unseal Key 4: efiHDkOqnrJFkYmqwEqd2JuwE8bWhEugZIXjaHupOFNQ
Unseal Key 5: 7+Noh+DMTpFOKQFEOJ3fdBh4XzqPL82wS+HVeXEA7+ov
Initial Root Token: hvs.322DPaVyEtWIExvLNgUyrbQf
Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.
Vault does not store the generated root key. Without at least 3 keys to
reconstruct the root key, Vault will remain permanently sealed!
It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ kubectl exec -it vault-0 -n vault -- vault operator unseal y1DN06xWAGLZcNPi6T4Ny6qlI2yaMxvZfGodHejuggom
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 1/3
Unseal Nonce e13effe0-9c38-2ad1-1d0c-d030d4e6ccf3
Version 1.16.1
Build Date 2024-04-03T12:35:53Z
Storage Type file
HA Enabled false
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ kubectl exec -it vault-0 -n vault -- vault operator unseal +T0IZltQR1iQOsgN7wSFPaSS9MyQreF+fhzgGpfuiLXs
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 2/3
Unseal Nonce e13effe0-9c38-2ad1-1d0c-d030d4e6ccf3
Version 1.16.1
Build Date 2024-04-03T12:35:53Z
Storage Type file
HA Enabled false
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ kubectl exec -it vault-0 -n vault -- vault operator unseal ESyeZ/iW4S8aws4bxnWQQZIXtEnN/jAvlbBI1rIu0L27
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.16.1
Build Date 2024-04-03T12:35:53Z
Storage Type file
Cluster Name vault-cluster-24d8d5c0
Cluster ID c7e6ab71-be02-d662-1bfe-2b2b20867e34
HA Enabled false
ubuntu@ubuntu:~$
初期設定後、Vaultが起動していること(READYが1/1)を確認する。
実行結果
ubuntu@ubuntu:~$ kubectl get pod -n vault
NAME READY STATUS RESTARTS AGE
pod/vault-0 1/1 Running 0 3m38s
pod/vault-agent-injector-ff58f5d77-5ggf9 1/1 Running 0 3m38s
ubuntu@ubuntu:~$
4.4. Vault設定
App Podが参照するSecret情報や必要となるVaultの設定を行う。
4.4.1. vault login
下記コマンドを実行し、vaultにログインする。
ログイン時にtokenの入力が求められるため、vault operator init実行時に表示されたInitial Root Tokenを使用してログインする。
コマンド
$ kubectl exec -it vault-0 -n vault -- /bin/sh
(vault)$ vault login
実行結果を展開する
実行結果
ubuntu@ubuntu:~$ kubectl exec -it vault-0 -n vault -- /bin/sh
/ $ vault login
Token (will be hidden):
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
Key Value
--- -----
token hvs.322DPaVyEtWIExvLNgUyrbQf
token_accessor whbdSAeZlpAg40MKnePQtsdl
token_duration ∞
token_renewable false
token_policies ["root"]
identity_policies []
policies ["root"]
/ $
4.4.2. Secret登録
App Pod内で使用するSecretを登録する。
コマンド
(vault)$ vault secrets enable -path=app-secret kv-v2
(vault)$ vault kv put app-secret/kv-secret SECRET_KEY1="aaaa" SECRET_KEY2="bbbb"
(vault)$ vault kv list app-secret/
(vault)$ vault kv get app-secret/kv-secret
実行結果を展開する
実行結果
/ $ vault secrets enable -path=app-secret kv-v2
Success! Enabled the kv-v2 secrets engine at: app-secret/
/ $ vault kv put app-secret/kv-secret SECRET_KEY1="aaaa" SECRET_KEY2="bbbb"
====== Secret Path ======
app-secret/data/kv-secret
======= Metadata =======
Key Value
--- -----
created_time 2024-05-05T07:51:10.054349939Z
custom_metadata <nil>
deletion_time n/a
destroyed false
version 1
/ $
/ $
/ $ vault kv list app-secret/
Keys
----
kv-secret
/ $
/ $ vault kv get app-secret/kv-secret
====== Secret Path ======
app-secret/data/kv-secret
======= Metadata =======
Key Value
--- -----
created_time 2024-05-05T07:51:10.054349939Z
custom_metadata <nil>
deletion_time n/a
destroyed false
version 1
======= Data =======
Key Value
--- -----
SECRET_KEY1 aaaa
SECRET_KEY2 bbbb
/ $
4.4.3. Policy設定
SecretへのPolicyを設定する。
コマンド
(vault)$ vault policy write app-secret-all-read-policy - <<EOF
path "app-secret/data/kv-secret" {
capabilities = ["read"]
}
EOF
(vault)$ vault policy list
(vault)$ vault policy read app-secret-all-read-policy
実行結果を展開する
実行結果
/ $ vault policy write app-secret-all-read-policy - <<EOF
> path "app-secret/data/kv-secret" {
> capabilities = ["read"]
> }
> EOF
Success! Uploaded policy: app-secret-all-read-policy
/ $
/ $ vault policy list
app-secret-all-read-policy
default
root
/ $
/ $ vault policy read app-secret-all-read-policy
path "app-secret/data/kv-secret" {
capabilities = ["read"]
}
/ $
4.4.4. Auth Method設定
app podがSecretを取得するための認証方法を設定し、ポリシーと紐づける。
コマンド
(vault)$ vault auth enable --path=kubernetes kubernetes
(vault)$ vault write auth/kubernetes/config kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"
(vault)$ vault auth list
(vault)$ vault read auth/kubernetes/config
(vault)$
(vault)$ vault write auth/kubernetes/role/app-role \
bound_service_account_names=app-sa \
bound_service_account_namespaces=app-ns \
policies=app-secret-all-read-policy \
ttl=24h
(vault)$ vault list auth/kubernetes/role/
(vault)$ vault read auth/kubernetes/role/app-role
実行結果を展開する
実行結果
/ $ vault auth enable --path=kubernetes kubernetes
Success! Enabled kubernetes auth method at: kubernetes/
/ $
/ $ vault write auth/kubernetes/config kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"
Success! Data written to: auth/kubernetes/config
/ $
/ $ vault auth list
Path Type Accessor Description Version
---- ---- -------- ----------- -------
kubernetes/ kubernetes auth_kubernetes_b514c948 n/a n/a
token/ token auth_token_3da24388 token based credentials n/a
/ $
/ $ vault read auth/kubernetes/config
Key Value
--- -----
disable_iss_validation true
disable_local_ca_jwt false
issuer n/a
kubernetes_ca_cert n/a
kubernetes_host https://10.96.0.1:443
pem_keys []
token_reviewer_jwt_set false
use_annotations_as_alias_metadata false
/ $
/ $ vault write auth/kubernetes/role/app-role \
> bound_service_account_names=app-sa \
> bound_service_account_namespaces=app-ns \
> policies=app-secret-all-read-policy \
> ttl=24h
Success! Data written to: auth/kubernetes/role/app-role
/ $
/ $ vault list auth/kubernetes/role/
Keys
----
app-role
/ $
/ $ vault read auth/kubernetes/role/app-role
Key Value
--- -----
alias_name_source serviceaccount_uid
bound_service_account_names [app-sa]
bound_service_account_namespace_selector n/a
bound_service_account_namespaces [app-ns]
policies [app-secret-all-read-policy]
token_bound_cidrs []
token_explicit_max_ttl 0s
token_max_ttl 0s
token_no_default_policy false
token_num_uses 0
token_period 0s
token_policies [app-secret-all-read-policy]
token_ttl 24h
token_type default
ttl 24h
/ $
4.4.5. Vault Podからログアウト
Vaultの設定は完了したので、Vault Podからexitする。
実行結果
/ $ exit
ubuntu@ubuntu:~$
4.5. App Podのデプロイ
App Podをデプロイし、VaultからSecretが取得できていることを確認する。
App Pod用のmanifestを作成し、applyする。
apply後、App Podが作成されるので、共有ボリューム(/vault/secrets)に、Secret用のファイルが生成されていることを確認する。
実行結果
ubuntu@ubuntu:~$ vi app-deployment.yaml
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ kubectl apply -f app-deployment.yaml
deployment.apps/app-deployment created
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ kubectl get pod -n app-ns
NAME READY STATUS RESTARTS AGE
app-deployment-5bbbdd5cb9-q6h8b 2/2 Running 0 11s
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ export POD_NAME=$(kubectl get pod -n app-ns |grep app-deploy |awk '{print $1}')
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ kubectl exec -ti -n app-ns $POD_NAME -c app -- cat /vault/secrets/kv-secret.txt
{ SECRET_KEY1: "aaaa", SECRET_KEY2: "bbbb" }
ubuntu@ubuntu:~$
app-deployment.yamlを展開する
app-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: app-deployment
namespace: app-ns
labels:
app: app-label
spec:
replicas: 1
selector:
matchLabels:
app: app-label
template:
metadata:
labels:
app: app-label
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/role: "app-role"
vault.hashicorp.com/agent-inject-secret-kv-secret.txt: "app-secret/data/kv-secret"
vault.hashicorp.com/agent-inject-template-kv-secret.txt: |
{{- with secret "app-secret/data/kv-secret" -}}
{ SECRET_KEY1: "{{ .Data.data.SECRET_KEY1 }}", SECRET_KEY2: "{{ .Data.data.SECRET_KEY2 }}" }
{{- end -}}
spec:
serviceAccountName: app-sa
containers:
- name: app
image: ubuntu:24.04
command: ["/bin/sh", "-c", "while :; do sleep 10; done"]
5. Vault Agent Sidecar Injectorの挙動確認
5.1. Vault Injectorが作成するContainer
Vault Injector(annotation)によってApp Pod内に追加されるコンテナーは、vault-agent-initとvault-agentの2つ。
実行結果を展開する
実行結果
ubuntu@ubuntu:~$ kubectl describe pod -n app-ns $(kubectl get pod -n app-ns |grep app-deploy |awk '{print $1}')
Name: app-deployment-5bbbdd5cb9-q6h8b
Namespace: app-ns
Priority: 0
Service Account: app-sa
Node: minikube/192.168.49.2
Start Time: Sun, 05 May 2024 09:03:58 +0000
Labels: app=app-label
pod-template-hash=5bbbdd5cb9
Annotations: vault.hashicorp.com/agent-inject: true
vault.hashicorp.com/agent-inject-secret-kv-secret.txt: app-secret/data/kv-secret
vault.hashicorp.com/agent-inject-status: injected
vault.hashicorp.com/role: app-role
Status: Running
IP: 10.244.0.12
IPs:
IP: 10.244.0.12
Controlled By: ReplicaSet/app-deployment-5bbbdd5cb9
Init Containers:
vault-agent-init:
Container ID: docker://b25cd0453ababcd37aad895e83d77b143659902cc438ad23ac643fd25c163837
Image: hashicorp/vault:1.16.1
Image ID: docker-pullable://hashicorp/vault@sha256:5efe43c659d4dc53539a589a95d5ffe77fffb761b407766e0408b0b18c30cad8
Port: <none>
Host Port: <none>
Command:
/bin/sh
-ec
Args:
echo ${VAULT_CONFIG?} | base64 -d > /home/vault/config.json && vault agent -config=/home/vault/config.json
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sun, 05 May 2024 09:04:00 +0000
Finished: Sun, 05 May 2024 09:04:00 +0000
Ready: True
Restart Count: 0
Limits:
cpu: 500m
memory: 128Mi
Requests:
cpu: 250m
memory: 64Mi
Environment:
NAMESPACE: app-ns (v1:metadata.namespace)
HOST_IP: (v1:status.hostIP)
POD_IP: (v1:status.podIP)
VAULT_LOG_LEVEL: info
VAULT_LOG_FORMAT: standard
VAULT_CONFIG: eyJhdXRvX2F1dGgiOnsibWV0aG9kIjp7InR5cGUiOiJrdWJlcm5ldGVzIiwibW91bnRfcGF0aCI6ImF1dGgva3ViZXJuZXRlcyIsImNvbmZpZyI6eyJyb2xlIjoiYXBwLXJvbGUiLCJ0b2tlbl9wYXRoIjoiL3Zhci9ydW4vc2VjcmV0cy9rdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3Rva2VuIn19LCJzaW5rIjpbeyJ0eXBlIjoiZmlsZSIsImNvbmZpZyI6eyJwYXRoIjoiL2hvbWUvdmF1bHQvLnZhdWx0LXRva2VuIn19XX0sImV4aXRfYWZ0ZXJfYXV0aCI6dHJ1ZSwicGlkX2ZpbGUiOiIvaG9tZS92YXVsdC8ucGlkIiwidmF1bHQiOnsiYWRkcmVzcyI6Imh0dHA6Ly92YXVsdC52YXVsdC5zdmM6ODIwMCJ9LCJ0ZW1wbGF0ZSI6W3siZGVzdGluYXRpb24iOiIvdmF1bHQvc2VjcmV0cy9rdi1zZWNyZXQudHh0IiwiY29udGVudHMiOiJ7eyB3aXRoIHNlY3JldCBcImFwcC1zZWNyZXQvZGF0YS9rdi1zZWNyZXRcIiB9fXt7IHJhbmdlICRrLCAkdiA6PSAuRGF0YSB9fXt7ICRrIH19OiB7eyAkdiB9fVxue3sgZW5kIH19e3sgZW5kIH19IiwibGVmdF9kZWxpbWl0ZXIiOiJ7eyIsInJpZ2h0X2RlbGltaXRlciI6In19In1dLCJ0ZW1wbGF0ZV9jb25maWciOnsiZXhpdF9vbl9yZXRyeV9mYWlsdXJlIjp0cnVlfX0=
Mounts:
/home/vault from home-init (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lhvrv (ro)
/vault/secrets from vault-secrets (rw)
Containers:
app:
Container ID: docker://eb08cadde7325f1db8679a57226e4aa802f103db5f43c2d3472f7e19041e3b9b
Image: ubuntu:24.04
Image ID: docker-pullable://ubuntu@sha256:3f85b7caad41a95462cf5b787d8a04604c8262cdcdf9a472b8c52ef83375fe15
Port: <none>
Host Port: <none>
Command:
/bin/sh
-c
while :; do sleep 10; done
State: Running
Started: Sun, 05 May 2024 09:04:02 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lhvrv (ro)
/vault/secrets from vault-secrets (rw)
vault-agent:
Container ID: docker://05045e4fa3bd273fe7a1c770ab3c866d6eb970f6205501caee11a68d0ca9c746
Image: hashicorp/vault:1.16.1
Image ID: docker-pullable://hashicorp/vault@sha256:5efe43c659d4dc53539a589a95d5ffe77fffb761b407766e0408b0b18c30cad8
Port: <none>
Host Port: <none>
Command:
/bin/sh
-ec
Args:
echo ${VAULT_CONFIG?} | base64 -d > /home/vault/config.json && vault agent -config=/home/vault/config.json
State: Running
Started: Sun, 05 May 2024 09:04:02 +0000
Ready: True
Restart Count: 0
Limits:
cpu: 500m
memory: 128Mi
Requests:
cpu: 250m
memory: 64Mi
Environment:
NAMESPACE: app-ns (v1:metadata.namespace)
HOST_IP: (v1:status.hostIP)
POD_IP: (v1:status.podIP)
VAULT_LOG_LEVEL: info
VAULT_LOG_FORMAT: standard
VAULT_CONFIG: eyJhdXRvX2F1dGgiOnsibWV0aG9kIjp7InR5cGUiOiJrdWJlcm5ldGVzIiwibW91bnRfcGF0aCI6ImF1dGgva3ViZXJuZXRlcyIsImNvbmZpZyI6eyJyb2xlIjoiYXBwLXJvbGUiLCJ0b2tlbl9wYXRoIjoiL3Zhci9ydW4vc2VjcmV0cy9rdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3Rva2VuIn19LCJzaW5rIjpbeyJ0eXBlIjoiZmlsZSIsImNvbmZpZyI6eyJwYXRoIjoiL2hvbWUvdmF1bHQvLnZhdWx0LXRva2VuIn19XX0sImV4aXRfYWZ0ZXJfYXV0aCI6ZmFsc2UsInBpZF9maWxlIjoiL2hvbWUvdmF1bHQvLnBpZCIsInZhdWx0Ijp7ImFkZHJlc3MiOiJodHRwOi8vdmF1bHQudmF1bHQuc3ZjOjgyMDAifSwidGVtcGxhdGUiOlt7ImRlc3RpbmF0aW9uIjoiL3ZhdWx0L3NlY3JldHMva3Ytc2VjcmV0LnR4dCIsImNvbnRlbnRzIjoie3sgd2l0aCBzZWNyZXQgXCJhcHAtc2VjcmV0L2RhdGEva3Ytc2VjcmV0XCIgfX17eyByYW5nZSAkaywgJHYgOj0gLkRhdGEgfX17eyAkayB9fToge3sgJHYgfX1cbnt7IGVuZCB9fXt7IGVuZCB9fSIsImxlZnRfZGVsaW1pdGVyIjoie3siLCJyaWdodF9kZWxpbWl0ZXIiOiJ9fSJ9XSwidGVtcGxhdGVfY29uZmlnIjp7ImV4aXRfb25fcmV0cnlfZmFpbHVyZSI6dHJ1ZX19
Mounts:
/home/vault from home-sidecar (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lhvrv (ro)
/vault/secrets from vault-secrets (rw)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kube-api-access-lhvrv:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
home-init:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
home-sidecar:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
vault-secrets:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 39s default-scheduler Successfully assigned app-ns/app-deployment-5bbbdd5cb9-q6h8b to minikube
Normal Pulled 38s kubelet Container image "hashicorp/vault:1.16.1" already present on machine
Normal Created 37s kubelet Created container vault-agent-init
Normal Started 37s kubelet Started container vault-agent-init
Normal Pulled 36s kubelet Container image "ubuntu:24.04" already present on machine
Normal Created 36s kubelet Created container app
Normal Started 35s kubelet Started container app
Normal Pulled 35s kubelet Container image "hashicorp/vault:1.16.1" already present on machine
Normal Created 35s kubelet Created container vault-agent
Normal Started 35s kubelet Started container vault-agent
ubuntu@ubuntu:~$
5.2. agent-inject-templateの役割
Annoationsにあるagent-inject-templateをコメントアウトした場合、data以外にmetadataなどの情報も一緒に格納された。
app-deployment.yaml
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/role: "app-role"
vault.hashicorp.com/agent-inject-secret-kv-secret.txt: "app-secret/data/kv-secret"
# コメントアウトする
#vault.hashicorp.com/agent-inject-template-kv-secret.txt: |
# {{- with secret "app-secret/data/kv-secret" -}}
# { SECRET_KEY1: "{{ .Data.data.SECRET_KEY1 }}", SECRET_KEY2: "{{ .Data.data.SECRET_KEY2 }}" }
# {{- end -}}
実行結果
ubuntu@ubuntu:~$ kubectl exec -ti -n app-ns $(kubectl get pod -n app-ns |grep app-deploy |awk '{print $1}') -c app -- cat /vault/secrets/kv-secret.txt
data: map[SECRET_KEY1:aaaa SECRET_KEY2:bbbb]
metadata: map[created_time:2024-05-05T07:51:10.054349939Z custom_metadata:<nil> deletion_time: destroyed:false version:1]
ubuntu@ubuntu:~$
5.3. Vault Secret変更時のApp Pod内のSecretの挙動
Vaultに登録しているSecretを変更したところ、App Pod内のSecretも少し待つと自動で更新された。
実行結果を展開する
実行結果
/ $ ## 事前のパラメータ確認
/ $ vault kv get app-secret/kv-secret
====== Secret Path ======
app-secret/data/kv-secret
======= Metadata =======
Key Value
--- -----
created_time 2024-05-05T07:51:10.054349939Z
custom_metadata <nil>
deletion_time n/a
destroyed false
version 1
======= Data =======
Key Value
--- -----
SECRET_KEY1 aaaa
SECRET_KEY2 bbbb
/ $
/ $ ## Secret変更
/ $ vault kv put app-secret/kv-secret SECRET_KEY1="cccc" SECRET_KEY2="dddd"
====== Secret Path ======
app-secret/data/kv-secret
======= Metadata =======
Key Value
--- -----
created_time 2024-05-05T09:20:32.157112084Z
custom_metadata <nil>
deletion_time n/a
destroyed false
version 2
/ $
/ $ ## Secret変更確認
/ $ vault kv get app-secret/kv-secret
====== Secret Path ======
app-secret/data/kv-secret
======= Metadata =======
Key Value
--- -----
created_time 2024-05-05T09:20:32.157112084Z
custom_metadata <nil>
deletion_time n/a
destroyed false
version 2
======= Data =======
Key Value
--- -----
SECRET_KEY1 cccc
SECRET_KEY2 dddd
/ $
/ $
/ $ exit
ubuntu@ubuntu:~$ ## Pod内のSecretを確認
ubuntu@ubuntu:~$ kubectl exec -ti -n app-ns $(kubectl get pod -n app-ns |grep app-deploy |awk '{print $1}') -c app -- cat /vault/secrets/kv-secret.txt
{ SECRET_KEY1: "aaaa", SECRET_KEY2: "bbbb" }
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ kubectl exec -ti -n app-ns $(kubectl get pod -n app-ns |grep app-deploy |awk '{print $1}') -c app -- cat /vault/secrets/kv-secret.txt
{ SECRET_KEY1: "aaaa", SECRET_KEY2: "bbbb" }
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ ## このタイミングでSecretが変更された
ubuntu@ubuntu:~$ kubectl exec -ti -n app-ns $(kubectl get pod -n app-ns |grep app-deploy |awk '{print $1}') -c app -- cat /vault/secrets/kv-secret.txt
{ SECRET_KEY1: "cccc", SECRET_KEY2: "dddd" }
ubuntu@ubuntu:~$
vault-agentのログを確認すると、agent: (runner) rendered "(dynamic)" => "/vault/secrets/kv-secret.txt"というログが出力していたので、sidecar側で更新を検知して、App Pod内の共有ボリュームの書き換えを実施しているみたい。
vault-agentのlog
ubuntu@ubuntu:~$ kubectl logs -n app-ns $(kubectl get pod -n app-ns |grep app-deploy |awk '{print $1}') -c vault-agent
==> Vault Agent started! Log data will stream in below:
==> Vault Agent configuration:
Api Address 1: http://bufconn
Cgo: disabled
Log Level: info
Version: Vault v1.16.1, built 2024-04-03T12:35:53Z
Version Sha: 6b5986790d7748100de77f7f127119c4a0f78946
2024-05-05T09:16:51.018Z [INFO] agent.sink.file: creating file sink
2024-05-05T09:16:51.018Z [INFO] agent.sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----
2024-05-05T09:16:51.018Z [INFO] agent.exec.server: starting exec server
2024-05-05T09:16:51.018Z [INFO] agent.exec.server: no env templates or exec config, exiting
2024-05-05T09:16:51.018Z [INFO] agent.auth.handler: starting auth handler
2024-05-05T09:16:51.018Z [INFO] agent.auth.handler: authenticating
2024-05-05T09:16:51.018Z [INFO] agent.template.server: starting template server
2024-05-05T09:16:51.018Z [INFO] agent: (runner) creating new runner (dry: false, once: false)
2024-05-05T09:16:51.020Z [INFO] agent: (runner) creating watcher
2024-05-05T09:16:51.020Z [INFO] agent.sink.server: starting sink server
2024-05-05T09:16:51.036Z [INFO] agent.auth.handler: authentication successful, sending token to sinks
2024-05-05T09:16:51.036Z [INFO] agent.auth.handler: starting renewal process
2024-05-05T09:16:51.036Z [INFO] agent.sink.file: token written: path=/home/vault/.vault-token
2024-05-05T09:16:51.036Z [INFO] agent.template.server: template server received new token
2024-05-05T09:16:51.036Z [INFO] agent: (runner) stopping
2024-05-05T09:16:51.036Z [INFO] agent: (runner) creating new runner (dry: false, once: false)
2024-05-05T09:16:51.036Z [INFO] agent: (runner) creating watcher
2024-05-05T09:16:51.036Z [INFO] agent: (runner) starting
2024-05-05T09:16:51.052Z [INFO] agent.auth.handler: renewed auth token
2024-05-05T09:21:26.849Z [INFO] agent: (runner) rendered "(dynamic)" => "/vault/secrets/kv-secret.txt"
ubuntu@ubuntu:~$
5.4 Vaultのログ
Vault側のログ確認が必要な場合は、vault audit enableを実行する。
実行結果
/ $ vault audit enable file file_path=/tmp/audit.log log_raw=true
Success! Enabled the file audit device at: file/
/ $ vault audit list
Path Type Description
---- ---- -----------
file/ file n/a
/ $
/ $ tail -f /tmp/audit.log
## vaultのログが出力する
6. 環境削除
動作確認に使用したリソースを削除する。
実行結果
ubuntu@ubuntu:~$ kubectl delete -f app-deployment.yaml
deployment.apps "app-deployment" deleted
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ rm app-deployment.yaml
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ kubectl delete sa app-sa -n app-ns
serviceaccount "app-sa" deleted
ubuntu@ubuntu:~$ kubectl delete ns app-ns
namespace "app-ns" deleted
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ helm uninstall vault -n vault
release "vault" uninstalled
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ helm repo remove hashicorp
"hashicorp" has been removed from your repositories
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ minikube delete
🔥 Deleting "minikube" in docker ...
🔥 Deleting container "minikube" ...
🔥 Removing /home/ubuntu/.minikube/machines/minikube ...
💀 Removed all traces of the "minikube" cluster.
ubuntu@ubuntu:~$
ubuntu@ubuntu:~$ unalias kubectl
ubuntu@ubuntu:~$