What is sealed secrets?
Sealed secrets are encrypted Kubernetes secrets. The secrets can be encrypted by anyone but can be decrypted only by the sealed secrets controller running in the Kubernetes cluster. The controller decrypts the sealed secrets to generate the Kubernetes secrets and applies it to a cluster
This is how sealed secrets looks like:
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
name: sealed-secrets
namespace: test-namespace
spec:
encryptedData:
foo: De1O3T4ErOm3DcGqVb123URIO.....
bar: chuYOP847NMKleyxp9068NhPP.....
Unlike Kubernetes secrets resources which can be decoded easily, sealed secrets can be decrypted only by the controller. This makes it useful to manage secrets along with other Kubernetes configuration in version control system such as git.
Why do we need to backup the sealed secrets?
Sealed secrets uses public/private RSA key pair generated by sealed secrets controller to achieve the encryption and decryption. So if the controller is removed due to some cluster failure or any other case then we cannot decrypt the sealed secrets even with newly created sealed secrets controller. We need to again create new sealed secrets which might be sometimes tedious and not feasible.
How can I backup and restore the sealed secrets?
By backing up the sealed secrets encryption private keys used to encrypt the secrets.
$ kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml >master.key
And to restore we can simply apply the backed up key in new cluster
$ kubectl apply -f master.key
Remember to apply this key before starting the controller or if the controller has already been started, restart the controller after applying the key
Demo
Check the current Kubernetes context
$ kubectl config current-context
arn:aws:eks:ap-northeast-1:XXXXXXX:cluster/test-eks-cluster
Apply the sealed secrets controller
$ kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.12.6/controller.yaml
It wil applied to kube-system namespace
$ kubectl get deployment -n kube-system sealed-secrets-controller
NAME READY UP-TO-DATE AVAILABLE AGE
sealed-secrets-controller 1/1 1 1 3d
Let's create a namespace for testing
$ kubectl create ns backup-restore
namespace/backup-restore created
$ kubectl get ns backup-restore
NAME STATUS AGE
backup-restore Active 22s
Create a sealed secrets
-
Create secrets file using dry-run
$ kubectl create secret generic test-secret --dry-run -n backup-restore -o yaml --from-literal=test_password1='secrets1' --from-literal=test_password2='secrets2'>test-secret.yaml
- Fetch the certificate to seal the secrets
$ kubeseal --fetch-cert > cert.pem
- Create a sealed secrets
$ kubeseal -o yaml --cert cert.pem < test-secret.yaml > sealed-secrets.yaml
$ cat sealed-secrets.yaml
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
creationTimestamp: null
name: test-secret
namespace: backup-restore
spec:
encryptedData:
test_password1: AgCKzkAeArDQnRrUf2gfZ3MQCYh6+ULC++Uig8ouA7VOUw8WM9DfcTFG/KSIBy+20VuVN4Xp05BEaNUReiR9SoRT/wi+5DUzSdljctUKhuWQCXI/hTvFiDwaeRSvOC9kIVEDPJRelg7kl+6MYREGZMrX/aW4Wgp5ZGbmlqlQ+409/1ehbfVe6j8ld6nwzAQkfj0D+hd5cdVjKHvwuLACYc4bZ+fZ5nEFZ2sDw4iH7zf/c1xNnsj8k1fpsg5uCQVDyaP/8SepYF4D/16LmUCMevMQ9t6zsr8cuLR+l8rajSCG9N0wi9NEWFZup6kkJge+nTiTog4O840GGxAwO6PzuerU8K83xSlgntq0A+VJAC7mPfd9TOwLeEytfbfzxLjbUkbag07ARlRi5lvbAzIQvW22DJXen4OlTFGk6TIdnObS28c0W5V/D23Ws+GWm9cbEURP9Okef0vT+nKRzR/OphJP+vnBlkaAq7syt3RummLBiTfT1aV2QwSjTTgM07bfKcyZv4R2fYVWt5lktl15PKABhdKQnwBX3wGA43c1zsSDnMdpQYjTb9v8kpIeKh8hFOFwqXx3/JuXVOiHvarDOUBIZ+VTeWAOHfe2D5mM4nhWkvie4P77koLvpEz4FPS8wLPWpBRAcoygFkao8WMDxNUuS/zU06Vqf4OR4gqPdO94f3FlLZ4x9oKLTBZETG1swHbpH0YKLOcdRg==
test_password2: AgAoODKJyL7/exmq55yaeXKZsmrPr9chJthdL335Z++OGajMxpRsUxHgcWRSr7ReOQoZsDNbhVFrx33/+yRt8b4diH50rO3buRBASoAqajjJTdAf7lsUD8B6GmvyXbtosBr/C1GDFlewyLzH0x0/4lPbbHsjRFADCDbVWO1kCVnhx5IHO1ClpHuGbmMtmE8IJa3U5BtsyOzw37a5U54FmAeWrCQx08B6fb28fP4ObHyGzl1H3TdqheERMqQV0GBdpjO0ByK6sMb2dPRDaQcrz2BBUNWpFdwSbjPUG++OPfTC72cpijID5pK5w+cWcf4xQ0UY4G5IDE91kUNYeJWnTaoWgAMC5wcarDJBSzqUQpkztUh9/bVzQO7+PGl7Jww6jL175z2yh2C4m8VvglPRVO/KPHf68obYHvKDxzayq9hfHj2AopoXmapzhVHm7NEuqhwKnjY5RDoja7rW20an7zJMnUMNyHhvpKnpRpQJyg243GR7UFaZqn5DMLqxawRodRYBgq9zQsVwqUX7jThZI++XNjNm/tfEXi39Yo7g4bgSQoBGEg5xdd0VPY2QtzCLNrAhXd8O5wqdm5B5cuJ3u2ucV5PvU4/46tn59R2i72mxyfI4ZuKOb4jl0BClV7wtXD6wy2oSBj1tvuBH7OL+C2C51HLBHzRuoNKNvFPzEFojeNezFzzk3AT7bhyrmqEqhz2YukxzZJ6gRQ==
template:
metadata:
creationTimestamp: null
name: test-secret
namespace: backup-restore
- Check the existing secrets with label `sealedsecrets.bitnami.com/sealed-secrets-key`
$ kubectl get secrets -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key
NAME TYPE DATA AGE
sealed-secrets-keypjmxr kubernetes.io/tls 2 10m
## Apply the sealed secrets
$ kubectl apply -f sealed-secrets.yaml
sealedsecret.bitnami.com/test-secret created
$ kubectl get secrets -n backup-restore
NAME TYPE DATA AGE
default-token-zdz8g kubernetes.io/service-account-token 3 12h
test-secret Opaque 2 6s
## Backup the sealed secrets
$ kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml >backup_sealed_secrets.key
## Delete the sealed secrets along with controller for testing
- delete the secrets
$ kubectl delete secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key
secret "sealed-secrets-keypjmxr" deleted
$ kubectl delete -f sealed-secrets.yaml
sealedsecret.bitnami.com "test-secret" deleted
- delete the controller
$ kubectl delete -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.12.6/controller.yaml
serviceaccount "sealed-secrets-controller" deleted
deployment.apps "sealed-secrets-controller" deleted
service "sealed-secrets-controller" deleted
role.rbac.authorization.k8s.io "sealed-secrets-service-proxier" deleted
rolebinding.rbac.authorization.k8s.io "sealed-secrets-controller" deleted
clusterrolebinding.rbac.authorization.k8s.io "sealed-secrets-controller" deleted
customresourcedefinition.apiextensions.k8s.io "sealedsecrets.bitnami.com" deleted
rolebinding.rbac.authorization.k8s.io "sealed-secrets-service-proxier" deleted
role.rbac.authorization.k8s.io "sealed-secrets-key-admin" deleted
clusterrole.rbac.authorization.k8s.io "secrets-unsealer" deleted
## Test if we can create a secrets using existing sealed secrets in new controller
- apply the new sealed secrets controller
$ kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.12.6/controller.yaml
deployment.apps/sealed-secrets-controller created
rolebinding.rbac.authorization.k8s.io/sealed-secrets-service-proxier created
rolebinding.rbac.authorization.k8s.io/sealed-secrets-controller created
clusterrolebinding.rbac.authorization.k8s.io/sealed-secrets-controller created
clusterrole.rbac.authorization.k8s.io/secrets-unsealer created
serviceaccount/sealed-secrets-controller created
customresourcedefinition.apiextensions.k8s.io/sealedsecrets.bitnami.com created
service/sealed-secrets-controller created
role.rbac.authorization.k8s.io/sealed-secrets-service-proxier created
role.rbac.authorization.k8s.io/sealed-secrets-key-admin created
- apply the sealed secrets
$ kubectl apply -f sealed-secrets.yaml
sealedsecret.bitnami.com/test-secret created
$ kubectl get secrets -n backup-restore
NAME TYPE DATA AGE
default-token-zdz8g kubernetes.io/service-account-token 3 12h
Here, we can see the kubernetes secrets is not created.
## Restore the sealed secrets
$ kubectl apply -f backup_sealed_secrets.key
secret/sealed-secrets-keypjmxr created
But we need to restart the controller
## Delete the controller again before restoring
$ kubectl get pods -n kube-system
sealed-secrets-controller-24a9bggd5-t34gn
$ kubectl delete pods sealed-secrets-controller-24a9bggd5-t34gn -n kube-system
## Check secrets
$ kubectl get secrets -n backup-restore
NAME TYPE DATA AGE
default-token-zdz8g kubernetes.io/service-account-token 3 12h
test-secret Opaque 2 57m