Kubernetes Secrets Management with Vault

Managing secrets securely in Kubernetes is critical for production deployments. While Kubernetes has a native Secrets resource, it stores data as base64-encoded text by default, which is not encryption. HashiCorp Vault provides enterprise-grade secrets management with dynamic secrets, encryption, audit logging, and fine-grained access control. This guide covers installing Vault in Kubernetes, configuring the sidecar injector, CSI drivers, and implementing dynamic secrets.

Table of Contents

• Vault Fundamentals
• Vault Installation on Kubernetes
• Vault Authentication
• Sidecar Injector
• CSI Driver
• Dynamic Secrets
• Vault Policies
• Practical Examples
• Conclusion

Vault Fundamentals

Why Vault?

Kubernetes native Secrets have limitations:

• Default storage in etcd is not encrypted
• No secrets rotation
• No audit logging for secret access
• No dynamic secret generation
• Secrets exposed in pod environment variables

Vault addresses these by providing:

• Encryption at rest and in transit
• Dynamic secret generation
• Audit logging
• Lease management
• Multiple authentication methods
• Fine-grained access control

Vault Concepts

Secrets Engine: Interface for reading/writing secrets (database, AWS, SSH)

Authentication Method: Way to prove identity (Kubernetes, JWT, LDAP)

Policies: Define what paths a client can access

Lease: Time-limited access to secrets with automatic revocation

Audit Log: Complete record of all requests to Vault

Vault Installation on Kubernetes

Prerequisites

• Running Kubernetes cluster
• Helm installed
• kubectl configured

Installing Vault with Helm

Add Vault Helm repository:

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

Create values file for Vault:

# vault-values.yaml
server:
  ha:
    enabled: true
    replicas: 3
    config: |
      ui = true
      
      listener "tcp" {
        address = "[::]:8200"
        tls_disable = false
        tls_cert_file = "/vault/userconfig/vault-ha-tls/tls.crt"
        tls_key_file = "/vault/userconfig/vault-ha-tls/tls.key"
      }
      
      storage "raft" {
        path = "/vault/data"
        node_id = "node-PLACEHOLDER"
        retry_join {
          leader_api_addr = "https://vault-0.vault-internal:8200"
        }
        retry_join {
          leader_api_addr = "https://vault-1.vault-internal:8200"
        }
        retry_join {
          leader_api_addr = "https://vault-2.vault-internal:8200"
        }
      }
      
      service_registration "kubernetes" {}
  
  dataStorage:
    size: 10Gi
    storageClass: fast-ssd
  
  logLevel: "info"
  
  resources:
    requests:
      memory: "256Mi"
      cpu: "100m"
    limits:
      memory: "512Mi"
      cpu: "500m"
  
  affinity: |
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchExpressions:
                - key: app.kubernetes.io/name
                  operator: In
                  values:
                    - vault
            topologyKey: kubernetes.io/hostname

ui:
  enabled: true
  serviceType: ClusterIP

injector:
  enabled: true
  replicas: 1
  
csi:
  enabled: true

Install Vault:

helm install vault hashicorp/vault \
  -n vault \
  --create-namespace \
  -f vault-values.yaml

Wait for Vault to be ready:

kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=vault -n vault --timeout=300s

Initializing Vault

Initialize Vault (creates unseal keys and root token):

kubectl exec -n vault vault-0 -- vault operator init \
  -key-shares=5 \
  -key-threshold=3 \
  > vault-init.txt

Store the unseal keys and root token securely. Extract and unseal:

# Extract unseal keys from vault-init.txt
UNSEAL_KEY_1=$(grep "Unseal Key 1" vault-init.txt | awk '{print $NF}')
UNSEAL_KEY_2=$(grep "Unseal Key 2" vault-init.txt | awk '{print $NF}')
UNSEAL_KEY_3=$(grep "Unseal Key 3" vault-init.txt | awk '{print $NF}')

# Unseal vault-0
kubectl exec -n vault vault-0 -- vault operator unseal $UNSEAL_KEY_1
kubectl exec -n vault vault-0 -- vault operator unseal $UNSEAL_KEY_2
kubectl exec -n vault vault-0 -- vault operator unseal $UNSEAL_KEY_3

# Unseal vault-1 and vault-2
kubectl exec -n vault vault-1 -- vault operator unseal $UNSEAL_KEY_1
kubectl exec -n vault vault-1 -- vault operator unseal $UNSEAL_KEY_2
kubectl exec -n vault vault-1 -- vault operator unseal $UNSEAL_KEY_3

kubectl exec -n vault vault-2 -- vault operator unseal $UNSEAL_KEY_1
kubectl exec -n vault vault-2 -- vault operator unseal $UNSEAL_KEY_2
kubectl exec -n vault vault-2 -- vault operator unseal $UNSEAL_KEY_3

Vault Authentication

Kubernetes Authentication Method

Enable Kubernetes auth:

ROOT_TOKEN=$(grep "Initial Root Token" vault-init.txt | awk '{print $NF}')

kubectl exec -n vault vault-0 -- vault login $ROOT_TOKEN

kubectl exec -n vault vault-0 -- vault auth enable kubernetes

Configure Kubernetes auth:

kubectl exec -n vault vault-0 -- vault write auth/kubernetes/config \
  kubernetes_host="https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  token_reviewer_jwt=@/var/run/secrets/kubernetes.io/serviceaccount/token

Sidecar Injector

Configuring the Injector

The Vault Agent Injector automatically injects secrets into pods via Init and Agent sidecars.

Annotations for pod injection:

apiVersion: v1
kind: Pod
metadata:
  name: my-app
  namespace: production
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/agent-inject-secret-database: "secret/data/database/postgres"
    vault.hashicorp.com/agent-inject-template-database: |
      {{- with secret "secret/data/database/postgres" -}}
      export DB_USER="{{ .Data.data.username }}"
      export DB_PASSWORD="{{ .Data.data.password }}"
      export DB_HOST="{{ .Data.data.host }}"
      {{- end }}
    vault.hashicorp.com/role: "my-app"
spec:
  serviceAccountName: my-app
  containers:
  - name: app
    image: my-app:1.0
    command: ["/bin/sh", "-c"]
    args:
      - |
        source /vault/secrets/database
        ./app

Policy for Sidecar Injector

Create a Vault policy:

kubectl exec -n vault vault-0 -- vault policy write my-app - <<EOF
path "secret/data/database/postgres" {
  capabilities = ["read"]
}

path "auth/token/renew-self" {
  capabilities = ["update"]
}
EOF

Vault Role for Kubernetes

Create a Vault role that binds to ServiceAccount:

kubectl exec -n vault vault-0 -- vault write auth/kubernetes/role/my-app \
  bound_service_account_names=my-app \
  bound_service_account_namespaces=production \
  policies=my-app \
  ttl=24h

CSI Driver

Installing Vault CSI Driver

The CSI driver mounts secrets as files in pods:

# vault-csi-values.yaml
server:
  ha:
    enabled: false

injector:
  enabled: false

csi:
  enabled: true

Install:

helm install vault-csi hashicorp/vault \
  -n vault \
  -f vault-csi-values.yaml

Using CSI Driver

Create a SecretProviderClass:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: vault-database
  namespace: production
spec:
  provider: vault
  parameters:
    vaultAddress: "https://vault.vault.svc:8200"
    vaultKubernetesMountPath: "kubernetes"
    vaultRole: "my-app"
    secretPath: "secret/data/database/postgres"
    objects: |
      - objectName: "username"
        secretPath: "secret/data/database/postgres"
        secretKey: "username"
      - objectName: "password"
        secretPath: "secret/data/database/postgres"
        secretKey: "password"
      - objectName: "host"
        secretPath: "secret/data/database/postgres"
        secretKey: "host"

Mount in pod:

apiVersion: v1
kind: Pod
metadata:
  name: my-app
  namespace: production
spec:
  serviceAccountName: my-app
  containers:
  - name: app
    image: my-app:1.0
    volumeMounts:
    - name: vault-secrets
      mountPath: /mnt/secrets
      readOnly: true
  volumes:
  - name: vault-secrets
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "vault-database"

Dynamic Secrets

Database Secret Engine

Enable database secrets engine:

kubectl exec -n vault vault-0 -- vault secrets enable database

Configure PostgreSQL connection:

kubectl exec -n vault vault-0 -- vault write database/config/postgres \
  plugin_name=postgresql-database-plugin \
  allowed_roles="readonly" \
  connection_url="postgresql://{{username}}:{{password}}@postgres.data.svc:5432/production?sslmode=require" \
  username="vault_admin" \
  password="VaultAdminPassword123"

Create database role:

kubectl exec -n vault vault-0 -- vault write database/roles/readonly \
  db_name=postgres \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

Request dynamic credentials:

kubectl exec -n vault vault-0 -- vault read database/creds/readonly

AWS Secret Engine

Enable AWS secrets:

kubectl exec -n vault vault-0 -- vault secrets enable aws

Configure AWS connection:

kubectl exec -n vault vault-0 -- vault write aws/config/root \
  access_key=AKIAIOSFODNN7EXAMPLE \
  secret_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY \
  region=us-east-1

Create AWS role:

kubectl exec -n vault vault-0 -- vault write aws/roles/app-role \
  credential_type=iam_user \
  policy_document=-<<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::my-bucket/*"
    }
  ]
}
EOF

Vault Policies

Creating Policies

kubectl exec -n vault vault-0 -- vault policy write app-secret-reader - <<EOF
path "secret/data/apps/*" {
  capabilities = ["read", "list"]
}

path "database/creds/readonly" {
  capabilities = ["read"]
}

path "auth/token/renew-self" {
  capabilities = ["update"]
}

path "auth/token/lookup-self" {
  capabilities = ["read"]
}
EOF

Policy Best Practices

Use path templating:

kubectl exec -n vault vault-0 -- vault policy write app-namespace-admin - <<EOF
path "secret/data/{{identity.entity.metadata.namespace}}/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

path "database/creds/{{identity.entity.metadata.namespace}}" {
  capabilities = ["read"]
}
EOF

Practical Examples

Example: Application with Database Secrets

Pod with injected database credentials:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: database-app
  namespace: production
---
apiVersion: v1
kind: Pod
metadata:
  name: app-with-db
  namespace: production
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/agent-inject-secret-db: "database/creds/readonly"
    vault.hashicorp.com/agent-inject-template-db: |
      {{- with secret "database/creds/readonly" -}}
      DB_USER="{{ .Data.username }}"
      DB_PASSWORD="{{ .Data.password }}"
      DB_HOST="postgres.data.svc"
      DB_NAME="production"
      DB_PORT="5432"
      {{- end }}
    vault.hashicorp.com/role: "app-role"
spec:
  serviceAccountName: database-app
  containers:
  - name: app
    image: my-app:1.0
    env:
    - name: APP_ENV
      value: production
    command:
    - /bin/sh
    - -c
    - |
      source /vault/secrets/db
      python app.py

Example: Multi-Secret Injection

Inject multiple secret sources:

apiVersion: v1
kind: Pod
metadata:
  name: multi-secret-app
  namespace: production
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/agent-inject-secret-db: "database/creds/app-db"
    vault.hashicorp.com/agent-inject-secret-aws: "aws/creds/app-role"
    vault.hashicorp.com/agent-inject-secret-api-keys: "secret/data/production/api-keys"
    vault.hashicorp.com/role: "app-role"
spec:
  serviceAccountName: app
  containers:
  - name: app
    image: my-app:1.0
    volumeMounts:
    - name: vault-secrets
      mountPath: /vault/secrets
  volumes:
  - name: vault-secrets
    emptyDir:
      medium: Memory

Conclusion

Vault provides enterprise-grade secrets management for Kubernetes deployments on VPS and baremetal infrastructure. By implementing the sidecar injector or CSI driver for automatic secret injection, leveraging dynamic secrets for automatic rotation, and enforcing fine-grained policies, you create a secure secrets management system. Start with basic static secrets, then gradually implement dynamic secrets and advanced features. Regular audits of Vault logs and policy access ensure ongoing security and compliance with your organization's requirements.