Cloud Build + Cloud KMS + kustomizeでGKEへデプロイする

• Cloud Build
• Cloud KMS
• kustomize

を使ってGKEへデプロイするための手順メモ

Cloud KMS

KeyrigとKeyの作成

Credential暗号化用のKeyを作っていきます

$ gcloud kms keyrings create ci-keyring --location=global
$ gcloud kms keyrings list --location global

$ gcloud kms keys create ci-key --keyring ci-keyring --purpose encryption --location global
$ gcloud kms keys list --keyring ci-keyring --location global

認証情報を暗号化

Using encrypted resources

• ファイルの場合

$ gcloud kms encrypt --plaintext-file=.env --ciphertext-file=env.encrypted \
                     --keyring=ci-keyring --key=ci-key --location=global
$ git add env.encrypted
$ git commit

• 環境変数の場合

echo -n $MY_SECRET | gcloud kms encrypt \
  --plaintext-file=- \
  --ciphertext-file=- \
  --location=global \
  --keyring=ci-keyring \
  --key=ci-key | base64

暗号化したものはcommitします

Cloud Build

権限の付与

• Using encrypted resources
• Building, testing, and deploying artifacts

Cloud KMSに対する権限を付ける

$ gcloud kms keys add-iam-policy-binding \
    ci-key --location=global --keyring=ci-keyring \
    --member=serviceAccount:[PROJECT_NUMBER]@cloudbuild.gserviceaccount.com \
    --role=roles/cloudkms.cryptoKeyDecrypter

GKEへのデプロイ権限の付与

$ gcloud projects add-iam-policy-binding [PROJECT_ID]  \
    --member=serviceAccount:[PROJECT_NUMBER]@cloudbuild.gserviceaccount.com \
    --role=roles/container.developer

kustomize

k8sディレクトリを切って書いていく

k8s
├── base
│   ├── deployment.yaml
│   ├── kustomization.yaml
│   └── service.yaml
└── overlays
    ├── dev
    │   ├── deployment.yaml
    │   └── kustomization.yaml
    └── prod
        ├── deployment.yaml
        └── kustomization.yaml

cloudbuild.yamlの記述

• 最初のステップで暗号化したファイルを復号
• dockerビルド時に $COMMIT_SHAをタグとして付ける
• kustomize edit set imageでYAML内のimageをCOMMIT_SHA付きのタグに置換
  • dirは kustomization.yamlがあるディレクトリを指定する
• secretsで環境変数を展開

steps:
- name: gcr.io/cloud-builders/gcloud
  args:
    - kms
    - decrypt
    - --ciphertext-file=ci/dev/env.encrypted
    - --plaintext-file=.env
    - --location=global
    - --keyring=ci-keyring
    - --key=ci-key
- name: 'gcr.io/cloud-builders/docker'
  args: [ 'build', '-t', 'gcr.io/$PROJECT_ID/example:$COMMIT_SHA', '.' ]
- name: 'gcr.io/cloud-builders/docker'
  args: ["push", "gcr.io/$PROJECT_ID/example:$COMMIT_SHA"]
- name: 'gcr.io/$PROJECT_ID/kustomize'
  dir: 'k8s/overlays/dev'
  args:
    - 'edit'
    - 'set'
    - 'image'
    - 'gcr.io/$PROJECT_ID/example=gcr.io/$PROJECT_ID/example:$COMMIT_SHA'
  env:
    - 'CLOUDSDK_COMPUTE_ZONE=asia-northeast1-a'
    - 'CLOUDSDK_CONTAINER_CLUSTER=target-cluster'
# https://github.com/GoogleCloudPlatform/cloud-builders-community/tree/master/kustomize
- id: deploy
  name: 'gcr.io/$PROJECT_ID/kustomize'
  args:
    - 'build'
    - 'k8s/overlays/dev'
  env:
    - 'APPLY=true'
    - 'CLOUDSDK_COMPUTE_ZONE=asia-northeast1-a'
    - 'CLOUDSDK_CONTAINER_CLUSTER=target-cluster'
secrets:
- kmsKeyName: projects/[PROJECT-ID]/locations/global/keyRings/ci-keyring/cryptoKeys/ci-key
  secretEnv:
    MY_SECRET: <base64-encoded encrypted credential>

実行

ローカルから実行する場合は、 COMMIT_SHAを指定して実行する

$ gcloud builds submit --config cloudbuild.yaml . \
       --substitutions=COMMIT_SHA=$(git rev-parse HEAD)

References

Substituting variable values

GKE+Cloud Build+kustomizeのデプロイ