External Secrets Operator (ESO) does exactly what the name suggests. It can sync Secrets from remote providers like HashiCorp Vault, AWS SecretManager etc.

Secrets from Vault can be imported into the Kubernetes cluster using csi-secrets-store and hashicorp/vault provider. But the CRD configuration seems a bit complex for noobs like me. So I ended up using ESO which seems easiest.

First, you need to install ESO with Helm inside the cluster. I am using eso-system as the namespace but feel free to choose another.

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets -n eso-system --create-namespace

Once the release is installed, 3 pods should appear and wait until they all are in both Running and Healthy state.

NAME                                                READY   STATUS
external-secrets-cert-controller-655d7bb477-kp2ck   1/1     Running
external-secrets-f9bc79d45-xk9lm                    1/1     Running
external-secrets-webhook-5b4779cbf8-mspsl           1/1     Running

To sync Secrets from Vault, we need to authorize the ESO service account to make valid calls to Vault APIs. I used a custom Token with a policy that allows only read access to the Vault KV engine. We need to pass the Token to ESO using Kubernetes secret.

kubectl create secret generic vault-token -n eso-system --from-literal=token=hvs.9lownuRz5bZO221dgkKXG5hQB

Once the Vault Token is ready, we can create a Vault Secret Store and tell ESO to establish a connection with it.

However, you can create multiple SecretStore in multiple namespaces. In that case, each ExternalSecret object will look for SecretStore in the same namespace.

In my case, I used one ClusterSecretStore and referenced multiple ExternalSecret in multiple namespaces to a single Store, to reduce the complexity.

Let's create our first ClusterSecretStore. Apply the following using kubectl apply -f

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "https://vault.your.host:8200"
      path: "kv"
      version: "v1"
      auth:
        tokenSecretRef:
          name: "vault-token"
          namespace: eso-system
          key: "token"

Now check if the Secret Store is ready to be used. The field Ready=True is seen if the Vault Token is correct.

$ kubectl get clustersecretstores 
NAME            AGE   STATUS   READY
vault-backend   12h   Valid    True

When the Secret Store is ready, it's time to put some values in Vault. You can use Vault UI. For my case, I used the CLI to enable Vault v1 KV engine and add some values there.

$ vault kv put kv/webapp-prod SPRING_DATASOURCE_USER=john SPRING_DATASOURCE_PASSWORD=1234qwer
$ vault kv get kv/webapp-prod
=============== Data ===============
Key                           Value
---                           -----
SPRING_DATASOURCE_PASSWORD    1234qwer
SPRING_DATASOURCE_USER        john

Now let's create an ExternalSecret object inside the cluster which will tell SecretStore to sync the above-mentioned secrets from Vault.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: webapp-secret
  namespace: webapp-prod
spec:
  refreshInterval: 30s
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: webapp-secret
    creationPolicy: Owner
  dataFrom:
  - extract:
      key: kv/webapp-prod

I have used the extract option which will fetch all KV pairs from Vault. Also, I used the Kubernetes namespace as the Vault KV Path to easily map multiple secrets into multiple namespaces.

Anyway, let's check if the ExternalSecret syncing is successful.

$ kubectl get externalsecret -n webapp-prod
NAME            STORE           REFRESH   STATUS         READY
webapp-secret   vault-backend   30s       SecretSynced   True

The state SecretSynced indicates that ESO has successfully fetched the secrets from Vault as per the instruction. Let's check if the Kubernetes native Secret object is created.

$ kubectl get secret webapp-secret -n webapp-prod -o yaml | head -4
apiVersion: v1
data:
  SPRING_DATASOURCE_HOST: MTI3LjAuMC4x
  SPRING_DATASOURCE_PASSWORD: MTIzNHF3ZXI=

If you base64 -d them, you will notice that the values are the same as they are in Vault. Feel free to add more KV pairs in the same path and the new values should appear in Kubernetes Secret as well,