Sync HashiCorp Vault Secrets into Kubernetes Native Secrets with ESO
3 min read
External Secrets Operator (ESO) does exactly what the name suggests. It can sync Secrets from remote providers like HashiCorp Vault, AWS SecretManager etc.
Secrets from Vault can be imported into the Kubernetes cluster using csi-secrets-store and hashicorp/vault provider. But the CRD configuration seems a bit complex for noobs like me. So I ended up using ESO which seems easiest.
First, you need to install ESO with Helm inside the cluster. I am using
eso-system as the namespace but feel free to choose another.
helm repo add external-secrets https://charts.external-secrets.io helm install external-secrets external-secrets/external-secrets -n eso-system --create-namespace
Once the release is installed, 3 pods should appear and wait until they all are in both Running and Healthy state.
NAME READY STATUS external-secrets-cert-controller-655d7bb477-kp2ck 1/1 Running external-secrets-f9bc79d45-xk9lm 1/1 Running external-secrets-webhook-5b4779cbf8-mspsl 1/1 Running
To sync Secrets from Vault, we need to authorize the ESO service account to make valid calls to Vault APIs. I used a custom Token with a policy that allows only read access to the Vault KV engine. We need to pass the Token to ESO using Kubernetes secret.
kubectl create secret generic vault-token -n eso-system --from-literal=token=hvs.9lownuRz5bZO221dgkKXG5hQB
Once the Vault Token is ready, we can create a Vault Secret Store and tell ESO to establish a connection with it.
However, you can create multiple SecretStore in multiple namespaces. In that case, each ExternalSecret object will look for SecretStore in the same namespace.
In my case, I used one ClusterSecretStore and referenced multiple ExternalSecret in multiple namespaces to a single Store, to reduce the complexity.
Let's create our first ClusterSecretStore. Apply the following using
kubectl apply -f
apiVersion: external-secrets.io/v1beta1 kind: ClusterSecretStore metadata: name: vault-backend spec: provider: vault: server: "https://vault.your.host:8200" path: "kv" version: "v1" auth: tokenSecretRef: name: "vault-token" namespace: eso-system key: "token"
Now check if the Secret Store is ready to be used. The field
Ready=True is seen if the Vault Token is correct.
$ kubectl get clustersecretstores NAME AGE STATUS READY vault-backend 12h Valid True
When the Secret Store is ready, it's time to put some values in Vault. You can use Vault UI. For my case, I used the CLI to enable Vault v1 KV engine and add some values there.
$ vault kv put kv/webapp-prod SPRING_DATASOURCE_USER=john SPRING_DATASOURCE_PASSWORD=1234qwer $ vault kv get kv/webapp-prod =============== Data =============== Key Value --- ----- SPRING_DATASOURCE_PASSWORD 1234qwer SPRING_DATASOURCE_USER john
Now let's create an ExternalSecret object inside the cluster which will tell SecretStore to sync the above-mentioned secrets from Vault.
apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: webapp-secret namespace: webapp-prod spec: refreshInterval: 30s secretStoreRef: name: vault-backend kind: ClusterSecretStore target: name: webapp-secret creationPolicy: Owner dataFrom: - extract: key: kv/webapp-prod
I have used the
extract option which will fetch all KV pairs from Vault. Also, I used the Kubernetes namespace as the Vault KV Path to easily map multiple secrets into multiple namespaces.
Anyway, let's check if the ExternalSecret syncing is successful.
$ kubectl get externalsecret -n webapp-prod NAME STORE REFRESH STATUS READY webapp-secret vault-backend 30s SecretSynced True
SecretSynced indicates that ESO has successfully fetched the secrets from Vault as per the instruction. Let's check if the Kubernetes native Secret object is created.
$ kubectl get secret webapp-secret -n webapp-prod -o yaml | head -4 apiVersion: v1 data: SPRING_DATASOURCE_HOST: MTI3LjAuMC4x SPRING_DATASOURCE_PASSWORD: MTIzNHF3ZXI=
base64 -d them, you will notice that the values are the same as they are in Vault. Feel free to add more KV pairs in the same path and the new values should appear in Kubernetes Secret as well,