External Secrets Operator (ESO) does exactly what the name suggests. It can sync Secrets from remote providers like HashiCorp Vault, AWS SecretManager etc.
Secrets from Vault can be imported into the Kubernetes cluster using csi-secrets-store and hashicorp/vault provider. But the CRD configuration seems a bit complex for noobs like me. So I ended up using ESO which seems easiest.
First, you need to install ESO with Helm inside the cluster. I am using eso-system as the namespace but feel free to choose another.
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets -n eso-system --create-namespace
Once the release is installed, 3 pods should appear and wait until they all are in both Running and Healthy state.
NAME READY STATUS
external-secrets-cert-controller-655d7bb477-kp2ck 1/1 Running
external-secrets-f9bc79d45-xk9lm 1/1 Running
external-secrets-webhook-5b4779cbf8-mspsl 1/1 Running
To sync Secrets from Vault, we need to authorize the ESO service account to make valid calls to Vault APIs. I used a custom Token with a policy that allows only read access to the Vault KV engine. We need to pass the Token to ESO using Kubernetes secret.
kubectl create secret generic vault-token -n eso-system --from-literal=token=hvs.9lownuRz5bZO221dgkKXG5hQB
Once the Vault Token is ready, we can create a Vault Secret Store and tell ESO to establish a connection with it.
However, you can create multiple SecretStore in multiple namespaces. In that case, each ExternalSecret object will look for SecretStore in the same namespace.
In my case, I used one ClusterSecretStore and referenced multiple ExternalSecret in multiple namespaces to a single Store, to reduce the complexity.
Let's create our first ClusterSecretStore. Apply the following using kubectl apply -f
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
name: vault-backend
spec:
provider:
vault:
server: "https://vault.your.host:8200"
path: "kv"
version: "v1"
auth:
tokenSecretRef:
name: "vault-token"
namespace: eso-system
key: "token"
Now check if the Secret Store is ready to be used. The field Ready=True is seen if the Vault Token is correct.
$ kubectl get clustersecretstores
NAME AGE STATUS READY
vault-backend 12h Valid True
When the Secret Store is ready, it's time to put some values in Vault. You can use Vault UI. For my case, I used the CLI to enable Vault v1 KV engine and add some values there.
$ vault kv put kv/webapp-prod SPRING_DATASOURCE_USER=john SPRING_DATASOURCE_PASSWORD=1234qwer
$ vault kv get kv/webapp-prod
=============== Data ===============
Key Value
--- -----
SPRING_DATASOURCE_PASSWORD 1234qwer
SPRING_DATASOURCE_USER john
Now let's create an ExternalSecret object inside the cluster which will tell SecretStore to sync the above-mentioned secrets from Vault.
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: webapp-secret
namespace: webapp-prod
spec:
refreshInterval: 30s
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
target:
name: webapp-secret
creationPolicy: Owner
dataFrom:
- extract:
key: kv/webapp-prod
I have used the extract option which will fetch all KV pairs from Vault. Also, I used the Kubernetes namespace as the Vault KV Path to easily map multiple secrets into multiple namespaces.
Anyway, let's check if the ExternalSecret syncing is successful.
$ kubectl get externalsecret -n webapp-prod
NAME STORE REFRESH STATUS READY
webapp-secret vault-backend 30s SecretSynced True
The state SecretSynced indicates that ESO has successfully fetched the secrets from Vault as per the instruction. Let's check if the Kubernetes native Secret object is created.
$ kubectl get secret webapp-secret -n webapp-prod -o yaml | head -4
apiVersion: v1
data:
SPRING_DATASOURCE_HOST: MTI3LjAuMC4x
SPRING_DATASOURCE_PASSWORD: MTIzNHF3ZXI=
If you base64 -d them, you will notice that the values are the same as they are in Vault. Feel free to add more KV pairs in the same path and the new values should appear in Kubernetes Secret as well,