Docker Secrets Management

Docker Secrets provide a secure mechanism for managing sensitive data in containerized environments, particularly within Docker Swarm. This comprehensive guide covers creating, storing, and managing secrets securely, differentiating between environment variables and secrets, implementing secrets in Docker Compose, rotating credentials, and best practices for production deployments. Proper secrets management prevents credential exposure and ensures compliance with security standards.

Table of Contents

• Understanding Docker Secrets
• Creating and Managing Secrets
• Using Secrets in Services
• Secrets vs Environment Variables
• Docker Compose Secrets
• Secret Rotation and Updates
• Security Best Practices
• External Secrets Management
• Troubleshooting Secrets
• Conclusion

Understanding Docker Secrets

Docker Secrets are encrypted data blobs stored in the Swarm manager's database and passed securely to authorized services. Secrets are never written to disk in plaintext and are only accessible to services explicitly granted access.

Secret architecture:

• Secrets encrypted at rest in manager database using Raft encryption
• Transmitted securely over TLS to worker nodes
• Decrypted only in memory for authorized containers
• Never exposed in process listings or environment
• Audit trail available through API

# Check Swarm status (required for Secrets)
docker swarm status

# Initialize Swarm if needed
docker swarm init

# List existing secrets
docker secret ls

# Inspect secret metadata
docker secret inspect <secret-name>

# Get secret value (manager-only, use carefully)
docker secret inspect --pretty <secret-name>

Secrets vs Configs (related feature):

• Secrets: For sensitive data (passwords, tokens, keys)
• Configs: For non-sensitive data (config files, public keys)
• Both encrypted in transit, Configs not encrypted at rest

Creating and Managing Secrets

Create secrets from files, stdin, or external sources.

Create secrets from stdin:

# Create secret from piped input
echo "MyDatabasePassword123!" | docker secret create db_password -

# Create API key secret
echo "sk-1234567890abcdef" | docker secret create api_key -

# Create TLS certificate
cat /path/to/cert.crt | docker secret create app_cert -

# Create TLS key
cat /path/to/key.key | docker secret create app_key -

# Create SSH key
cat ~/.ssh/id_rsa | docker secret create ssh_key -

Create secrets from files:

# Create secret from file
docker secret create db_config ./db_config.json

# Create from environment file
docker secret create app_env ./app.env

# Create from certificate
docker secret create tls_cert ./domain.crt

# Create from key
docker secret create tls_key ./domain.key

Generate and create secrets:

# Generate random password and create secret
openssl rand -base64 32 | docker secret create db_password -

# Generate API token
head -c 32 /dev/urandom | base64 | docker secret create api_token -

# Generate encryption key
openssl rand -hex 32 | docker secret create encryption_key -

# Create secret with random data
python3 -c "import secrets; print(secrets.token_urlsafe(32))" | docker secret create jwt_secret -

List and inspect secrets:

# List all secrets
docker secret ls

# Get detailed secret information
docker secret inspect db_password

# Inspect multiple secrets
docker secret inspect db_password api_key ssh_key

# Format output
docker secret ls --format "table {{.Name}}\t{{.CreatedAt}}"

# View secret creation date and ID
docker secret inspect --pretty db_password

# Check secret availability
docker secret inspect db_password --format='{{.ID}}'

Remove secrets:

# Remove unused secret
docker secret rm api_key

# Remove multiple secrets
docker secret rm secret1 secret2 secret3

# List secrets before cleanup
docker secret ls | grep -v CREATED

# Remove all secrets (dangerous - confirm services don't use them)
docker secret ls --format '{{.Name}}' | xargs docker secret rm

Using Secrets in Services

Deploy services with access to secrets through secure mounts.

Deploy service with secret:

# Create secret
echo "db_user=admin" | docker secret create db_creds -

# Deploy service with secret access
docker service create \
  --name myapp \
  --secret db_creds \
  --replicas 2 \
  myapp:latest

# Secret accessible at /run/secrets/db_creds in container

# Verify service has access
docker service inspect myapp | grep -A 5 Secrets

# Check secret in running task
docker exec <container-id> cat /run/secrets/db_creds

Configure secret targets (different filename):

# Secret accessible as different filename
docker service create \
  --name api \
  --secret source=api_key,target=api_secret \
  --secret source=db_password,target=password \
  myapi:latest

# In container:
# cat /run/secrets/api_secret
# cat /run/secrets/password

Multiple secrets for one service:

# Create multiple secrets
echo "admin" | docker secret create db_user -
echo "MyPassword123!" | docker secret create db_pass -
echo "sk-123abc" | docker secret create api_key -
echo "-----BEGIN CERTIFICATE-----..." | docker secret create tls_cert -

# Deploy with multiple secrets
docker service create \
  --name webapp \
  --secret db_user \
  --secret db_pass \
  --secret api_key \
  --secret source=tls_cert,target=certificate \
  --replicas 3 \
  webapp:latest

# Application reads from /run/secrets/

Service update with secrets:

# Add new secret to running service
docker service update \
  --secret-add new_secret \
  myapp

# Remove secret from service
docker service update \
  --secret-rm old_secret \
  myapp

# Replace secret reference
docker service update \
  --secret-rm old_api_key \
  --secret-add api_key \
  myapp

Secrets vs Environment Variables

Understand when to use secrets versus environment variables.

Comparison:

# Environment variables (NOT for secrets)
# - Visible in ps output
# - Visible in container inspect
# - Visible in environment
# - Passed to child processes
# - Logged in Docker history

# Secrets (for sensitive data)
# - Not in ps output
# - Not in inspect output
# - Only accessible via /run/secrets/
# - Not inherited by child processes
# - Encrypted at rest and in transit

Environment variables example:

# Create service with environment variables (non-sensitive)
docker service create \
  --name app \
  --env ENVIRONMENT=production \
  --env LOG_LEVEL=info \
  --env API_TIMEOUT=30 \
  myapp:latest

# Check env vars in running container
docker exec <container-id> env | grep -E "ENVIRONMENT|LOG_LEVEL"

Secrets example (sensitive):

# Create secrets for sensitive data
echo "MyDatabasePassword!" | docker secret create db_password -
echo "sk-abc123def456" | docker secret create api_key -

# Create service with secrets
docker service create \
  --name app \
  --secret db_password \
  --secret api_key \
  --env ENVIRONMENT=production \
  --env LOG_LEVEL=info \
  myapp:latest

# In application, read sensitive data from /run/secrets/
# Read config from environment variables

Secure application pattern:

# Dockerfile showing best practice
cat > Dockerfile <<'EOF'
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt

COPY app.py .

# App reads secrets from /run/secrets/
# App reads config from environment variables

EXPOSE 5000
CMD ["python", "app.py"]
EOF

# Application code
cat > app.py <<'EOF'
import os

# Read non-sensitive config from environment
ENVIRONMENT = os.getenv("ENVIRONMENT", "development")
LOG_LEVEL = os.getenv("LOG_LEVEL", "info")

# Read sensitive data from secrets
try:
    with open("/run/secrets/db_password", "r") as f:
        DB_PASSWORD = f.read().strip()
except FileNotFoundError:
    DB_PASSWORD = os.getenv("DB_PASSWORD")  # fallback

try:
    with open("/run/secrets/api_key", "r") as f:
        API_KEY = f.read().strip()
except FileNotFoundError:
    API_KEY = os.getenv("API_KEY")  # fallback

# Configure application with loaded values
EOF

Docker Compose Secrets

Use secrets in Docker Compose for development and testing.

Compose file with secrets:

cat > docker-compose.yml <<'EOF'
version: '3.9'

services:
  db:
    image: postgres:15-alpine
    environment:
      POSTGRES_DB: mydb
      POSTGRES_USER: admin
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
    secrets:
      - db_password
    volumes:
      - db_data:/var/lib/postgresql/data

  app:
    image: myapp:latest
    depends_on:
      - db
    secrets:
      - db_password
      - api_key
    environment:
      ENVIRONMENT: production
      DATABASE_HOST: db
    ports:
      - "5000:5000"

secrets:
  db_password:
    file: ./secrets/db_password.txt
  api_key:
    file: ./secrets/api_key.txt

volumes:
  db_data:

EOF

# Create secret files
mkdir -p secrets
echo "MyDatabasePassword123!" > secrets/db_password.txt
echo "sk-1234567890abcdef" > secrets/api_key.txt

# Start services
docker-compose up -d

# Access secrets from running container
docker-compose exec app cat /run/secrets/db_password

External secrets source:

cat > docker-compose.yml <<'EOF'
version: '3.9'

services:
  web:
    image: nginx:alpine
    secrets:
      - source: tls_cert
        target: certificate.crt
      - source: tls_key
        target: certificate.key
    ports:
      - "443:443"

secrets:
  tls_cert:
    file: /etc/ssl/certs/domain.crt
  tls_key:
    file: /etc/ssl/private/domain.key

EOF

docker-compose up -d

Secret Rotation and Updates

Safely update secrets without disrupting running services.

Rotation strategy:

# Current state: services using old_secret

# Step 1: Create new secret
echo "NewPassword2024!" | docker secret create db_password_v2 -

# Step 2: Update services to use new secret
docker service update \
  --secret-add db_password_v2 \
  myapp

# Step 3: Give containers time to read new secret
sleep 10

# Step 4: Restart containers to pick up new secret
docker service update \
  --force \
  myapp

# Step 5: Verify services using new secret
docker service ps myapp

# Step 6: Remove old secret after verification
sleep 30
docker secret rm db_password_v1

Rolling secret updates:

# Create new secret version
echo "UpdatedPassword123!" | docker secret create api_key_v2 -

# Service with rolling update strategy
docker service update \
  --secret-rm api_key \
  --secret-add api_key_v2 \
  --update-parallelism 1 \
  --update-delay 30s \
  webapp

# Monitor rolling update
watch -n 2 'docker service ps webapp'

# Verify all tasks updated
docker service ps webapp --filter desired-state=running

Automated rotation script:

cat > rotate-secrets.sh <<'EOF'
#!/bin/bash

SERVICE="myapp"
SECRET="db_password"
NEW_PASSWORD=$(openssl rand -base64 32)
TIMESTAMP=$(date +%s)

# Create versioned secret
docker secret create "${SECRET}_${TIMESTAMP}" <<< "$NEW_PASSWORD"

# Update service to use new secret
docker service update \
  --secret-add "${SECRET}_${TIMESTAMP}" \
  "$SERVICE"

# Wait for service to stabilize
sleep 10

# Remove old secrets (keep last 3 versions)
SECRETS=$(docker secret ls --format '{{.Name}}' | grep "^${SECRET}_" | sort -r)
VERSION_COUNT=0
while IFS= read -r old_secret; do
    ((VERSION_COUNT++))
    if [ $VERSION_COUNT -gt 3 ]; then
        docker secret rm "$old_secret"
    fi
done <<< "$SECRETS"

echo "Secret rotation completed for $SECRET"
EOF

chmod +x rotate-secrets.sh

# Schedule rotation
0 0 * * * /path/to/rotate-secrets.sh >> /var/log/secret-rotation.log 2>&1

Security Best Practices

Implement comprehensive security measures for secrets management.

Secure secret creation and storage:

# Never expose secrets in command history
cat > /dev/null <<< "MyPassword" | docker secret create db_pass -

# Use secure file permissions during creation
umask 0077
echo "SecretToken" > /tmp/token.txt
docker secret create token - < /tmp/token.txt
shred -vfz -n 3 /tmp/token.txt

# Verify secret not in bash history
history | grep secret  # Should show nothing

# Verify secret not in Docker history
docker history myimage | grep -i secret  # Should show nothing

Audit and monitoring:

# Enable secret audit logging
# Configure Docker daemon with audit policy

# Monitor secret access
docker events --filter type=secret

# Check secret creation history
docker inspect <secret-id> --format='{{.CreatedAt}}'

# Log service updates affecting secrets
docker events --filter type=service

# Rotate secrets regularly
# Track secret versions for compliance

Principle of least privilege:

# Only grant secrets to services that need them
# Avoid wildcard secret grants

docker service create \
  --name specific_service \
  --secret db_password \
  --secret api_key \
  myapp:latest

# Only specific services should have each secret
# Not all services need all secrets

Encryption and compliance:

# Verify Swarm encryption enabled
docker swarm inspect self | grep -i encrypt

# Use external secret management for compliance
# Integrate with Vault, AWS Secrets Manager, etc.

# Require authentication for secret operations
docker node update --role manager \
  <node-id>

# Managers-only can create/remove secrets
# Restricted access control

External Secrets Management

Integrate with external secrets management systems.

HashiCorp Vault integration:

# Example Vault integration setup
cat > Dockerfile <<'EOF'
FROM ubuntu:22.04
RUN apt-get update && apt-get install -y curl jq

COPY vault-client.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/vault-client.sh

WORKDIR /app
COPY app.py .

ENTRYPOINT ["/usr/local/bin/vault-client.sh"]
CMD ["python", "app.py"]
EOF

cat > vault-client.sh <<'EOF'
#!/bin/bash

# Authenticate with Vault
VAULT_TOKEN=$(curl -X POST \
  http://vault:8200/v1/auth/kubernetes/login \
  -d @- <<< '{"jwt":"'$VAULT_SA_TOKEN'","role":"app"}' \
  | jq -r '.auth.client_token')

# Fetch secret from Vault
SECRET=$(curl -H "X-Vault-Token: $VAULT_TOKEN" \
  http://vault:8200/v1/secret/data/app/database \
  | jq -r '.data.data.password')

# Export as environment variable
export DB_PASSWORD="$SECRET"

# Execute application
exec "$@"
EOF

chmod +x vault-client.sh

AWS Secrets Manager integration:

# Docker service with AWS Secrets Manager access
docker service create \
  --name app \
  --env AWS_REGION=us-east-1 \
  --mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock \
  myapp:latest

# Application code to fetch secrets
cat > fetch-secret.py <<'EOF'
import boto3
import json

client = boto3.client('secretsmanager')

def get_secret(secret_name):
    try:
        response = client.get_secret_value(SecretId=secret_name)
        return json.loads(response['SecretString'])
    except Exception as e:
        print(f"Error: {e}")
        return None

# Usage
db_secret = get_secret('prod/database')
db_password = db_secret['password']
EOF

Troubleshooting Secrets

Diagnose and resolve secrets-related issues.

Debug secret access:

# Verify secret exists
docker secret ls | grep db_password

# Check secret metadata
docker secret inspect db_password

# Verify service has secret access
docker service inspect myapp | grep -A 20 Secrets

# Test secret availability in container
docker exec <container-id> ls -la /run/secrets/

# Read secret content from container
docker exec <container-id> cat /run/secrets/db_password

# Check file permissions
docker exec <container-id> stat /run/secrets/db_password

Common issues and solutions:

# Issue: Service can't access secret
# Solution: Verify secret is assigned to service

docker service update --secret-add missing_secret myapp

# Issue: Secret not found in container
# Solution: Verify correct secret name

docker exec <container-id> ls /run/secrets/

# Issue: Secret file permissions denied
# Solution: Secrets have 600 permissions, run as correct user

# In Dockerfile:
# RUN addgroup -S app && adduser -S app -G app
# USER app

# Issue: Old secret still accessible after update
# Solution: Restart containers to reload

docker service update --force myapp

Conclusion

Docker Secrets provide a production-grade mechanism for managing sensitive data in containerized environments. By understanding the distinction between secrets and environment variables, implementing proper access controls, and integrating with orchestration systems, you create secure infrastructure that meets compliance requirements. Start with basic secret creation and service assignment, progress to automated rotation, and eventually integrate with external secrets management systems for enterprise deployments. Regular audits of secret access and rotation schedules ensure your secrets remain protected as your infrastructure evolves. Treat secrets management as a foundational security practice, not an afterthought.