Vault for Beginners: Secrets Management in Kubernetes

The first time I ran kubectl get secret myapp -o yaml and base64-decoded the value, I felt my stomach drop. There was my database password, sitting in etcd, readable by anyone who could reach the API with the right RBAC. Kubernetes Secrets are not secrets. They’re base64-encoded plain text with a fancy name. That’s the default, and it’s the thing nobody warns you about on day one.

Every cloud provider has a fix for sale. AWS has Secrets Manager, Google has Secret Manager, Azure has Key Vault. They work. The catch shows up later: the day you need to migrate, the day you want to know exactly what happens to a secret after you write it, the day you realise your most sensitive data lives in a system you can’t inspect.

HashiCorp Vault is the self-hosted answer. You run it, you control it, and when it breaks at 2am you can actually reason about why.

This post builds Vault up one layer at a time. Start with the smallest thing that works, then add real-world usage, then the advanced bits. Stop reading wherever you have enough.

Why Self-Hosted Secrets?

A cloud KMS hands you a few problems wrapped as features. Your secrets live in a system you don’t operate, so you can’t audit what happens inside it. Migration is brutal because secrets are the stickiest lock-in there is. And you pay per secret, per request, forever.

Self-hosted Vault flips each of those. You see exactly how secrets are stored and accessed. There’s no vendor lock-in on your most critical data. It runs identically across clouds, on-prem, and the edge, so the same config works everywhere. Your secrets never leave infrastructure you control.

That’s the sovereignty angle. As I wrote in Sovereign Infrastructure, black boxes feel like splinters, and secrets are the one place where a splinter can sink your whole platform. The trade-off is real: you now operate Vault, including unsealing and backups. I’ll be honest about that cost throughout.

Vault Architecture Basics

Before any commands, three concepts. Once these click, the rest of Vault stops feeling like magic.

Secrets Engines are where secrets live.

kv (key-value) for static secrets
database for dynamic database credentials
pki for certificate generation

Auth Methods are how clients prove who they are.

kubernetes for pods
token for direct access
userpass, ldap, oidc for humans

Policies decide what an authenticated client is allowed to touch.

flowchart TD
    subgraph vault["Vault"]
        subgraph auth["Auth Methods"]
            K8S["kubernetes"]
            USER["userpass"]
        end

        K8S --> POL["Policies"]
        USER --> POL

        subgraph engines["Secrets Engines"]
            KV["KV"]
            PKI["PKI"]
            DB["DB"]
        end

        POL --> KV
        POL --> PKI
        POL --> DB
    end

The Simplest Version: Installing Vault

Start with the smallest install that runs. The official Helm chart gets you there.

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

helm install vault hashicorp/vault \
  --namespace vault \
  --create-namespace \
  --set server.ha.enabled=true \
  --set server.ha.replicas=3

I don’t actually install anything by hand though. Everything goes through GitOps with ArgoCD, so Vault is declared once and the cluster reconciles it:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: vault
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://helm.releases.hashicorp.com
    chart: vault
    targetRevision: 0.27.0
    helm:
      values: |
        server:
          ha:
            enabled: true
            replicas: 3
            raft:
              enabled: true
          dataStorage:
            size: 10Gi
            storageClass: longhorn
        injector:
          enabled: true
  destination:
    server: https://kubernetes.default.svc
    namespace: vault
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Initializing and Unsealing

Here’s the part that trips up everyone new to Vault. Vault starts sealed. Even with the pods running, it won’t serve a single secret until you unseal it. This is on purpose: the encryption key that protects everything is itself encrypted, and you have to provide enough key shares to reconstruct it. Initialize first:

kubectl exec -it vault-0 -n vault -- vault operator init

This spits out two things: 5 unseal keys, and an initial root token. The keys are Shamir shares, so by default you need any 3 of the 5 to unseal.

Store these somewhere safe and offline. Lose the unseal keys and you lose access to every secret Vault holds. There’s no recovery, no support ticket, no reset link. I keep mine split across separate offline locations, because the whole point is that no single compromise hands someone the keys to everything.

Unseal each pod, feeding it 3 of the 5 keys one at a time:

kubectl exec -it vault-0 -n vault -- vault operator unseal <key1>
kubectl exec -it vault-0 -n vault -- vault operator unseal <key2>
kubectl exec -it vault-0 -n vault -- vault operator unseal <key3>

Repeat for vault-1 and vault-2. Yes, by hand, three times.

Layer 1: Auto-Unseal

The manual dance is fine for a demo. In production it’s a trap. Every pod restart, every node reboot, every chaos event leaves Vault sealed and your apps unable to read secrets until a human shows up with the keys. For a homelab that’s supposed to survive me being asleep, that’s a non-starter.

Auto-unseal solves it by handing the unsealing job to a separate key, either another Vault’s transit engine or a cloud KMS:

# Using another Vault for transit auto-unseal
server:
  ha:
    enabled: true
  config: |
    seal "transit" {
      address = "https://vault-primary.example.com:8200"
      disable_renewal = "false"
      key_name = "autounseal"
      mount_path = "transit/"
      tls_skip_verify = "false"
    }

Or with a cloud KMS. Using cloud here is a deliberate, narrow exception: one tiny key whose only job is to wrap Vault’s master key, not your actual secrets. The blast radius is small and the resilience win is large, so I’ll take that trade:

server:
  config: |
    seal "awskms" {
      region     = "eu-west-1"
      kms_key_id = "alias/vault-unseal"
    }

Enabling the KV Secrets Engine

With Vault unsealed, you can finally store something. The kv engine is the simplest one and a good place to start. Log in, enable it, write a secret:

# Login with root token
kubectl exec -it vault-0 -n vault -- vault login

# Enable KV v2 secrets engine
kubectl exec -it vault-0 -n vault -- vault secrets enable -path=secret kv-v2

# Create a secret
kubectl exec -it vault-0 -n vault -- vault kv put secret/myapp/config \
  database_url="postgres://user:pass@db:5432/myapp" \
  api_key="super-secret-key"

Kubernetes Authentication

A secret in Vault is useless if your pods can’t reach it. The clean way to grant access is the Kubernetes auth method, which lets a pod prove its identity using its own service account token. No long-lived static credential baked into an image, no shared password. Vault verifies the token against the API server and decides whether to issue a short-lived Vault token in return.

Enable and configure it:

# Enable Kubernetes auth
kubectl exec -it vault-0 -n vault -- vault auth enable kubernetes

# Configure it
kubectl exec -it vault-0 -n vault -- vault write auth/kubernetes/config \
  kubernetes_host="https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT"

Then bind a role to a specific service account in a specific namespace. This is where least privilege starts: only the myapp service account in default gets this role, and only for an hour at a time.

kubectl exec -it vault-0 -n vault -- vault write auth/kubernetes/role/myapp \
  bound_service_account_names=myapp \
  bound_service_account_namespaces=default \
  policies=myapp-policy \
  ttl=1h

Creating Policies

The role above referenced myapp-policy, which doesn’t exist yet. Policies are where you spell out exactly what a client may read or list, path by path. Default-deny is baked in: if a path isn’t granted, it’s forbidden. This one lets the app read its own config and nothing else.

# myapp-policy.hcl
path "secret/data/myapp/*" {
  capabilities = ["read"]
}

path "secret/metadata/myapp/*" {
  capabilities = ["list"]
}

Apply it:

kubectl exec -it vault-0 -n vault -- vault policy write myapp-policy - <<EOF
path "secret/data/myapp/*" {
  capabilities = ["read"]
}

path "secret/metadata/myapp/*" {
  capabilities = ["list"]
}
EOF

Layer 2: Injecting Secrets into Pods

So far you’ve been running vault commands by hand. Real workloads need secrets delivered automatically. The Vault Agent Injector does this with nothing but pod annotations: it watches for them, adds an init container that authenticates, and drops the rendered secret onto a file your app reads. No SDK, no Vault-aware code in your application.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: myapp
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "myapp"
        vault.hashicorp.com/agent-inject-secret-config: "secret/data/myapp/config"
        vault.hashicorp.com/agent-inject-template-config: |
          {{- with secret "secret/data/myapp/config" -}}
          export DATABASE_URL="{{ .Data.data.database_url }}"
          export API_KEY="{{ .Data.data.api_key }}"
          {{- end }}
    spec:
      serviceAccountName: myapp
      containers:
        - name: app
          image: myapp:v1.0.0
          command: ["/bin/sh", "-c"]
          args:
            - source /vault/secrets/config && ./myapp

The injector:

Adds an init container that authenticates with Vault
Writes secrets to /vault/secrets/config
Your app reads them at startup

External Secrets Operator Alternative

The injector is great, but it couples your pods to Vault’s annotation dance and mounts secrets as files, which not every app wants. External Secrets Operator takes a different route. It syncs Vault secrets into native Kubernetes Secrets, so your workloads consume them the boring, standard way through envFrom or valueFrom. I reach for this when an app expects plain environment variables and I don’t want to teach it anything about Vault.

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "http://vault.vault:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "myapp"
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: myapp-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: myapp-secrets
    creationPolicy: Owner
  data:
    - secretKey: DATABASE_URL
      remoteRef:
        key: myapp/config
        property: database_url
    - secretKey: API_KEY
      remoteRef:
        key: myapp/config
        property: api_key

This creates a regular Kubernetes Secret that your pods can use normally:

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - name: app
          envFrom:
            - secretRef:
                name: myapp-secrets

Dynamic Database Credentials

This is where Vault stops being a fancy password store and starts being something you can’t easily replicate with a cloud KMS. Static secrets sit there until someone rotates them, which is to say, until someone forgets. Dynamic secrets flip the model: Vault creates a brand new database user on demand, hands it to one pod, and deletes it when the lease expires. Nothing is shared, nothing is permanent.

Enable the database engine, point it at Postgres, and define a role that knows how to mint users:

# Enable database secrets engine
kubectl exec -it vault-0 -n vault -- vault secrets enable database

# Configure PostgreSQL connection
kubectl exec -it vault-0 -n vault -- vault write database/config/mydb \
  plugin_name=postgresql-database-plugin \
  connection_url="postgresql://{{username}}:{{password}}@postgres.default:5432/myapp" \
  allowed_roles="myapp-db" \
  username="vault_admin" \
  password="vault_admin_password"

# Create role that generates credentials
kubectl exec -it vault-0 -n vault -- vault write database/roles/myapp-db \
  db_name=mydb \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

Now a pod requests its own credentials through the same injector annotations as before:

annotations:
  vault.hashicorp.com/agent-inject-secret-db: "database/creds/myapp-db"
  vault.hashicorp.com/agent-inject-template-db: |
    {{- with secret "database/creds/myapp-db" -}}
    export PGUSER="{{ .Data.username }}"
    export PGPASSWORD="{{ .Data.password }}"
    {{- end }}

Each pod gets unique credentials that:

Auto-rotate
Can be revoked individually
Create an audit trail per pod

PKI: Certificate Authority

The same dynamic-secret idea works for certificates. Vault can run as your internal CA and issue short-lived certs on demand, which beats the alternative of long-lived certificates you forget about until they expire on a Sunday. Set up a root CA and a role that constrains what it’ll sign:

# Enable PKI
kubectl exec -it vault-0 -n vault -- vault secrets enable pki

# Configure max TTL
kubectl exec -it vault-0 -n vault -- vault secrets tune -max-lease-ttl=87600h pki

# Generate root CA
kubectl exec -it vault-0 -n vault -- vault write -field=certificate pki/root/generate/internal \
  common_name="My Org Root CA" \
  ttl=87600h > CA_cert.crt

# Create role for issuing certs
kubectl exec -it vault-0 -n vault -- vault write pki/roles/internal-certs \
  allowed_domains="internal,svc.cluster.local" \
  allow_subdomains=true \
  max_ttl=72h

Wire this into cert-manager and certificates rotate themselves across the cluster without you touching anything.

High Availability

A secrets store that goes down takes every dependent app with it, so Vault is the last place you want a single point of failure. Run it in HA mode with the built-in Raft storage, which keeps the data inside Vault itself instead of leaning on an external database you’d also have to make resilient.

server:
  ha:
    enabled: true
    replicas: 3
    raft:
      enabled: true
      setNodeId: true
      config: |
        cluster_name = "vault-cluster"
        storage "raft" {
          path = "/vault/data"
          retry_join {
            leader_api_addr = "http://vault-0.vault-internal:8200"
          }
          retry_join {
            leader_api_addr = "http://vault-1.vault-internal:8200"
          }
          retry_join {
            leader_api_addr = "http://vault-2.vault-internal:8200"
          }
        }

This creates a 3-node Raft cluster:

One leader, two followers
Automatic leader election on failure
Data replicated across all nodes

Backup Strategy

HA protects you from a node dying. It does nothing for a fat-fingered vault policy delete or a corrupted Raft log. For that you need snapshots, and you need them automated, because a backup you have to remember to run is a backup that won’t exist when you need it. Take a manual snapshot to see how it works:

# Create snapshot
kubectl exec -it vault-0 -n vault -- vault operator raft snapshot save /tmp/vault-snapshot.snap

# Copy to local
kubectl cp vault/vault-0:/tmp/vault-snapshot.snap ./vault-snapshot.snap

Then make it boring and automatic with a CronJob, so snapshots happen whether or not I’m paying attention:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: vault-backup
spec:
  schedule: "0 */6 * * *"  # Every 6 hours
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: vault-backup
          containers:
            - name: backup
              image: hashicorp/vault:1.15
              command:
                - /bin/sh
                - -c
                - |
                  vault operator raft snapshot save /backup/vault-$(date +%Y%m%d-%H%M%S).snap
              volumeMounts:
                - name: backup
                  mountPath: /backup
          volumes:
            - name: backup
              persistentVolumeClaim:
                claimName: vault-backup
          restartPolicy: OnFailure

Audit Logging

If you can’t tell who read which secret and when, you don’t really control your secrets, you just store them. Audit logging closes that gap by recording every single request. Turn it on:

kubectl exec -it vault-0 -n vault -- vault audit enable file file_path=/vault/audit/audit.log

Every access lands in the log as structured JSON, ready to ship into Loki or whatever you query with:

{
  "time": "2025-07-02T10:00:00Z",
  "type": "response",
  "auth": {
    "token_type": "service",
    "policies": ["myapp-policy"]
  },
  "request": {
    "path": "secret/data/myapp/config",
    "operation": "read"
  }
}

The Full Picture: My Actual Setup

Remember the three-line Helm install from the start? Here’s where all the layers land in practice. This is the real config running in my homelab, HA Raft, audit storage, sane resource limits, and the UI for the rare moment I need to click something:

server:
  ha:
    enabled: true
    replicas: 3
    raft:
      enabled: true

  dataStorage:
    size: 10Gi
    storageClass: longhorn

  auditStorage:
    enabled: true
    size: 10Gi
    storageClass: longhorn

  resources:
    requests:
      memory: 256Mi
      cpu: 250m
    limits:
      memory: 512Mi
      cpu: 500m

  extraEnvironmentVars:
    VAULT_CACERT: /vault/userconfig/vault-tls/ca.crt

injector:
  enabled: true
  resources:
    requests:
      memory: 64Mi
      cpu: 50m
    limits:
      memory: 128Mi
      cpu: 100m

ui:
  enabled: true

A few choices worth calling out:

HA with Raft keeps storage inside Vault, so there’s no external database to also keep alive.
Audit storage earns its keep the first time you have to answer “who touched this secret?”.
Resource limits stay small because Vault genuinely is lightweight. It barely registers on the cluster.
UI enabled for the occasional manual poke, even though almost everything runs through GitOps.

Migration from Cloud KMS

Already on AWS Secrets Manager or similar? You don’t have to flip a switch and pray. Migrate one app at a time. Install Vault alongside your existing KMS, sync the secrets you want to move, repoint apps individually, and decommission the cloud KMS once it’s empty. Nobody gets a big-bang cutover weekend.

External Secrets Operator makes this painless because it can read from both backends at once during the transition:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-secrets-manager
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-west-1
---
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault
spec:
  provider:
    vault:
      server: "http://vault.vault:8200"

Why This Matters

We started with a base64 string in etcd and ended with dynamic database credentials, a private CA, an HA cluster, automated snapshots, and an audit log of every access. You can stop at any layer and still come out ahead of where the defaults left you.

What you get from running it yourself is visibility into how secrets are actually stored and accessed, the same workflow whether you’re on a cloud, on-prem, or a Raspberry Pi in island mode, no surprise pricing, and the ability to fix things when they break because you understand the whole stack.

I won’t pretend the operational cost is zero. You now own unsealing, backups, and upgrades. But secrets are the one thing I refuse to leave inside a black box, because they’re the keys to everything else I run. Owning that complexity is the price of not depending on a system I can’t inspect, and for this particular thing I’ll pay it every time.

Why Self-Hosted Secrets?#

Vault Architecture Basics#

The Simplest Version: Installing Vault#

Initializing and Unsealing#

Layer 1: Auto-Unseal#

Enabling the KV Secrets Engine#

Kubernetes Authentication#

Creating Policies#

Layer 2: Injecting Secrets into Pods#

External Secrets Operator Alternative#

Dynamic Database Credentials#

PKI: Certificate Authority#

High Availability#

Backup Strategy#

Audit Logging#

The Full Picture: My Actual Setup#

Migration from Cloud KMS#

Why This Matters#