Search…

Kubernetes configuration and secrets

In this series (14 parts)
  1. Why Kubernetes exists
  2. Kubernetes architecture
  3. Core Kubernetes objects
  4. Kubernetes networking
  5. Storage in Kubernetes
  6. Kubernetes configuration and secrets
  7. Resource management and autoscaling
  8. Kubernetes workload types
  9. Kubernetes observability
  10. Kubernetes security
  11. Helm and package management
  12. GitOps with ArgoCD
  13. Kubernetes cluster operations
  14. Service mesh concepts

Hardcoding configuration into container images is a bad idea. You cannot change settings without rebuilding. You cannot use the same image across environments. And you definitely should not bake database passwords into a Docker layer. Kubernetes solves this with ConfigMaps and Secrets.

ConfigMaps as environment variables

The simplest approach: inject key-value pairs as environment variables.

apiVersion: v1
kind: ConfigMap
metadata:
  name: api-config
  namespace: production
data:
  DATABASE_HOST: "postgres.production.svc.cluster.local"
  DATABASE_PORT: "5432"
  LOG_LEVEL: "info"
  CACHE_TTL: "300"
  MAX_CONNECTIONS: "100"

Reference it in a Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api-server
  template:
    metadata:
      labels:
        app: api-server
    spec:
      containers:
        - name: api
          image: myregistry/api:3.2.0
          envFrom:
            - configMapRef:
                name: api-config
          ports:
            - containerPort: 8080

envFrom loads every key from the ConfigMap as an environment variable. You can also pick individual keys:

env:
  - name: DB_HOST
    valueFrom:
      configMapKeyRef:
        name: api-config
        key: DATABASE_HOST

ConfigMaps as mounted files

Some applications read configuration from files rather than environment variables. Mount a ConfigMap as a volume.

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
  namespace: production
data:
  nginx.conf: |
    worker_processes auto;
    events {
        worker_connections 1024;
    }
    http {
        upstream backend {
            server api-server.production.svc.cluster.local:8080;
        }
        server {
            listen 80;
            location / {
                proxy_pass http://backend;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
            }
            location /health {
                return 200 'ok';
                add_header Content-Type text/plain;
            }
        }
    }

Mount it:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-proxy
  namespace: production
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx-proxy
  template:
    metadata:
      labels:
        app: nginx-proxy
    spec:
      containers:
        - name: nginx
          image: nginx:1.27
          ports:
            - containerPort: 80
          volumeMounts:
            - name: nginx-conf
              mountPath: /etc/nginx/nginx.conf
              subPath: nginx.conf
              readOnly: true
      volumes:
        - name: nginx-conf
          configMap:
            name: nginx-config

Using subPath mounts a single file without overwriting the entire directory.

Auto-reload caveat

Mounted ConfigMaps update automatically when the ConfigMap changes (with a delay of up to a minute). But the running process does not know the file changed. For nginx, you need a sidecar or init script that watches for file changes and runs nginx -s reload. Environment variables never update without a pod restart.

Secrets and the base64 caveat

Secrets look like ConfigMaps but are intended for sensitive data. There is a critical misunderstanding: base64 encoding is not encryption. Anyone with API access to the namespace can decode secrets.

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: production
type: Opaque
stringData:
  username: app_user
  password: "correct-horse-battery-staple"
  connection-string: "postgresql://app_user:correct-horse-battery-staple@postgres:5432/appdb?sslmode=require"

Use stringData for plain text input. Kubernetes encodes it to base64 on storage. Retrieve and decode:

kubectl get secret db-credentials -n production -o jsonpath='{.data.password}' | base64 -d

Mount secrets the same way as ConfigMaps:

env:
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: db-credentials
        key: password

What base64 does not protect against

  • A compromised pod with access to the Kubernetes API can read all secrets in its namespace.
  • etcd stores secrets in plain text by default. Enable encryption at rest in the API server.
  • Secret values appear in pod specs, which are visible to anyone with get pods permission.
  • Environment variable secrets show up in process listings and crash dumps.

For production systems, you need stronger tools. See secrets management for broader strategies.

Encrypting secrets at rest

Enable etcd encryption in the API server configuration:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: "base64-encoded-32-byte-key-here"
      - identity: {}

The identity provider at the end allows reading unencrypted secrets that existed before encryption was enabled. After migrating all secrets, remove it.

Sealed Secrets

Sealed Secrets by Bitnami let you commit encrypted secrets to Git. A controller in the cluster decrypts them.

Install the controller and CLI:

# Install controller
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.27.0/controller.yaml

# Install kubeseal CLI (macOS)
brew install kubeseal

Encrypt a secret:

kubectl create secret generic db-credentials \
  --from-literal=password=correct-horse-battery-staple \
  --dry-run=client -o yaml | \
  kubeseal --format yaml > sealed-db-credentials.yaml

The resulting SealedSecret is safe to commit:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: db-credentials
  namespace: production
spec:
  encryptedData:
    password: AgBy3i4OJSWK+PiTySYZZA9rO...

The controller decrypts it and creates a regular Secret. Only the controller’s private key can decrypt the values.

ExternalSecrets Operator

ExternalSecrets Operator (ESO) syncs secrets from external providers into Kubernetes Secrets. It supports AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager, Azure Key Vault, and more.

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets
  namespace: production
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets
    kind: SecretStore
  target:
    name: db-credentials
    creationPolicy: Owner
  data:
    - secretKey: password
      remoteRef:
        key: production/database
        property: password
    - secretKey: username
      remoteRef:
        key: production/database
        property: username

ESO polls the external store and keeps the Kubernetes Secret in sync. If you rotate the password in AWS Secrets Manager, the Kubernetes Secret updates within the refresh interval.

Vault Agent Injector

HashiCorp Vault Agent Injector uses a mutating webhook to inject secrets into pods as files. No changes to application code are needed.

Annotate your pod:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api-server
  template:
    metadata:
      labels:
        app: api-server
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "api-server"
        vault.hashicorp.com/agent-inject-secret-db-creds: "secret/data/production/db"
        vault.hashicorp.com/agent-inject-template-db-creds: |
          {{- with secret "secret/data/production/db" -}}
          export DB_USER="{{ .Data.data.username }}"
          export DB_PASS="{{ .Data.data.password }}"
          {{- end }}
    spec:
      serviceAccountName: api-server
      containers:
        - name: api
          image: myregistry/api:3.2.0
          command: ["/bin/sh", "-c", "source /vault/secrets/db-creds && exec /app/server"]

The Vault Agent sidecar authenticates with Vault using the pod’s service account, retrieves secrets, and writes them to /vault/secrets/. The secrets are never stored in Kubernetes.

Choosing a secret management approach

ApproachSecrets in GitRotationComplexity
Plain SecretsNo (or leaked)ManualLow
Sealed SecretsYes (encrypted)ManualLow
ExternalSecrets OperatorNoAutomaticMedium
Vault Agent InjectorNoAutomaticHigh

Start with Sealed Secrets for small teams. Move to ESO or Vault when you need automatic rotation or centralized management.

What comes next

Configuration and secrets keep your applications flexible and secure. The next article covers resource management and autoscaling: setting CPU and memory limits, understanding QoS classes, and scaling pods automatically with HPA.

Start typing to search across all content
navigate Enter open Esc close