Kubernetes configuration and secrets
In this series (14 parts)
- Why Kubernetes exists
- Kubernetes architecture
- Core Kubernetes objects
- Kubernetes networking
- Storage in Kubernetes
- Kubernetes configuration and secrets
- Resource management and autoscaling
- Kubernetes workload types
- Kubernetes observability
- Kubernetes security
- Helm and package management
- GitOps with ArgoCD
- Kubernetes cluster operations
- Service mesh concepts
Hardcoding configuration into container images is a bad idea. You cannot change settings without rebuilding. You cannot use the same image across environments. And you definitely should not bake database passwords into a Docker layer. Kubernetes solves this with ConfigMaps and Secrets.
ConfigMaps as environment variables
The simplest approach: inject key-value pairs as environment variables.
apiVersion: v1
kind: ConfigMap
metadata:
name: api-config
namespace: production
data:
DATABASE_HOST: "postgres.production.svc.cluster.local"
DATABASE_PORT: "5432"
LOG_LEVEL: "info"
CACHE_TTL: "300"
MAX_CONNECTIONS: "100"
Reference it in a Deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: api-server
namespace: production
spec:
replicas: 3
selector:
matchLabels:
app: api-server
template:
metadata:
labels:
app: api-server
spec:
containers:
- name: api
image: myregistry/api:3.2.0
envFrom:
- configMapRef:
name: api-config
ports:
- containerPort: 8080
envFrom loads every key from the ConfigMap as an environment variable. You can also pick individual keys:
env:
- name: DB_HOST
valueFrom:
configMapKeyRef:
name: api-config
key: DATABASE_HOST
ConfigMaps as mounted files
Some applications read configuration from files rather than environment variables. Mount a ConfigMap as a volume.
apiVersion: v1
kind: ConfigMap
metadata:
name: nginx-config
namespace: production
data:
nginx.conf: |
worker_processes auto;
events {
worker_connections 1024;
}
http {
upstream backend {
server api-server.production.svc.cluster.local:8080;
}
server {
listen 80;
location / {
proxy_pass http://backend;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
location /health {
return 200 'ok';
add_header Content-Type text/plain;
}
}
}
Mount it:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-proxy
namespace: production
spec:
replicas: 2
selector:
matchLabels:
app: nginx-proxy
template:
metadata:
labels:
app: nginx-proxy
spec:
containers:
- name: nginx
image: nginx:1.27
ports:
- containerPort: 80
volumeMounts:
- name: nginx-conf
mountPath: /etc/nginx/nginx.conf
subPath: nginx.conf
readOnly: true
volumes:
- name: nginx-conf
configMap:
name: nginx-config
Using subPath mounts a single file without overwriting the entire directory.
Auto-reload caveat
Mounted ConfigMaps update automatically when the ConfigMap changes (with a delay of up to a minute). But the running process does not know the file changed. For nginx, you need a sidecar or init script that watches for file changes and runs nginx -s reload. Environment variables never update without a pod restart.
Secrets and the base64 caveat
Secrets look like ConfigMaps but are intended for sensitive data. There is a critical misunderstanding: base64 encoding is not encryption. Anyone with API access to the namespace can decode secrets.
apiVersion: v1
kind: Secret
metadata:
name: db-credentials
namespace: production
type: Opaque
stringData:
username: app_user
password: "correct-horse-battery-staple"
connection-string: "postgresql://app_user:correct-horse-battery-staple@postgres:5432/appdb?sslmode=require"
Use stringData for plain text input. Kubernetes encodes it to base64 on storage. Retrieve and decode:
kubectl get secret db-credentials -n production -o jsonpath='{.data.password}' | base64 -d
Mount secrets the same way as ConfigMaps:
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: db-credentials
key: password
What base64 does not protect against
- A compromised pod with access to the Kubernetes API can read all secrets in its namespace.
- etcd stores secrets in plain text by default. Enable encryption at rest in the API server.
- Secret values appear in pod specs, which are visible to anyone with
get podspermission. - Environment variable secrets show up in process listings and crash dumps.
For production systems, you need stronger tools. See secrets management for broader strategies.
Encrypting secrets at rest
Enable etcd encryption in the API server configuration:
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: "base64-encoded-32-byte-key-here"
- identity: {}
The identity provider at the end allows reading unencrypted secrets that existed before encryption was enabled. After migrating all secrets, remove it.
Sealed Secrets
Sealed Secrets by Bitnami let you commit encrypted secrets to Git. A controller in the cluster decrypts them.
Install the controller and CLI:
# Install controller
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.27.0/controller.yaml
# Install kubeseal CLI (macOS)
brew install kubeseal
Encrypt a secret:
kubectl create secret generic db-credentials \
--from-literal=password=correct-horse-battery-staple \
--dry-run=client -o yaml | \
kubeseal --format yaml > sealed-db-credentials.yaml
The resulting SealedSecret is safe to commit:
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
name: db-credentials
namespace: production
spec:
encryptedData:
password: AgBy3i4OJSWK+PiTySYZZA9rO...
The controller decrypts it and creates a regular Secret. Only the controller’s private key can decrypt the values.
ExternalSecrets Operator
ExternalSecrets Operator (ESO) syncs secrets from external providers into Kubernetes Secrets. It supports AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager, Azure Key Vault, and more.
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: aws-secrets
namespace: production
spec:
provider:
aws:
service: SecretsManager
region: us-east-1
auth:
jwt:
serviceAccountRef:
name: external-secrets-sa
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: db-credentials
namespace: production
spec:
refreshInterval: 1h
secretStoreRef:
name: aws-secrets
kind: SecretStore
target:
name: db-credentials
creationPolicy: Owner
data:
- secretKey: password
remoteRef:
key: production/database
property: password
- secretKey: username
remoteRef:
key: production/database
property: username
ESO polls the external store and keeps the Kubernetes Secret in sync. If you rotate the password in AWS Secrets Manager, the Kubernetes Secret updates within the refresh interval.
Vault Agent Injector
HashiCorp Vault Agent Injector uses a mutating webhook to inject secrets into pods as files. No changes to application code are needed.
Annotate your pod:
apiVersion: apps/v1
kind: Deployment
metadata:
name: api-server
namespace: production
spec:
replicas: 3
selector:
matchLabels:
app: api-server
template:
metadata:
labels:
app: api-server
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/role: "api-server"
vault.hashicorp.com/agent-inject-secret-db-creds: "secret/data/production/db"
vault.hashicorp.com/agent-inject-template-db-creds: |
{{- with secret "secret/data/production/db" -}}
export DB_USER="{{ .Data.data.username }}"
export DB_PASS="{{ .Data.data.password }}"
{{- end }}
spec:
serviceAccountName: api-server
containers:
- name: api
image: myregistry/api:3.2.0
command: ["/bin/sh", "-c", "source /vault/secrets/db-creds && exec /app/server"]
The Vault Agent sidecar authenticates with Vault using the pod’s service account, retrieves secrets, and writes them to /vault/secrets/. The secrets are never stored in Kubernetes.
Choosing a secret management approach
| Approach | Secrets in Git | Rotation | Complexity |
|---|---|---|---|
| Plain Secrets | No (or leaked) | Manual | Low |
| Sealed Secrets | Yes (encrypted) | Manual | Low |
| ExternalSecrets Operator | No | Automatic | Medium |
| Vault Agent Injector | No | Automatic | High |
Start with Sealed Secrets for small teams. Move to ESO or Vault when you need automatic rotation or centralized management.
What comes next
Configuration and secrets keep your applications flexible and secure. The next article covers resource management and autoscaling: setting CPU and memory limits, understanding QoS classes, and scaling pods automatically with HPA.