GitOps
In this series (8 parts)
Everything in this series has treated Git as a tool for managing application source code. GitOps extends that role. In a GitOps workflow, Git is the single source of truth for infrastructure, configuration, and deployment state. Every change goes through a pull request. Every deployment is a commit.
For background on infrastructure as code, see the IaC introduction. For ArgoCD specifics, see ArgoCD.
What GitOps is
GitOps is an operational framework that applies DevOps best practices for application development (version control, collaboration, CI/CD) to infrastructure automation.
Four principles define GitOps:
- Declarative configuration. The desired state of the system is described declaratively (Kubernetes manifests, Terraform files, Helm charts).
- Versioned and immutable. The desired state is stored in Git. Every change is a commit with full audit history.
- Pulled automatically. Software agents automatically pull the desired state from Git and apply it.
- Continuously reconciled. Agents monitor the actual state and correct any drift from the desired state.
Push vs pull model
There are two approaches to applying changes from Git to the target environment.
Push-based deployment
A CI/CD pipeline detects a change in Git and pushes it to the target.
Developer commits -> CI pipeline runs -> Pipeline pushes to cluster
This is the traditional CD model. Jenkins, GitHub Actions, and GitLab CI operate this way. The pipeline needs credentials to the target environment.
Pull-based deployment
An agent running inside the target environment pulls the desired state from Git and applies it.
Developer commits -> Agent detects change -> Agent applies to cluster
The agent lives inside the cluster. It does not expose cluster credentials to the CI system. This is more secure and aligns with the principle of least privilege.
graph TD subgraph "Git Repository" GR["Desired state<br/>(manifests)"] end subgraph "Kubernetes Cluster" AGENT["GitOps Agent<br/>(ArgoCD / Flux)"] ACTUAL["Running workloads"] end DEV["Developer"] -->|push commit| GR AGENT -->|poll / webhook| GR AGENT -->|compare desired<br/>vs actual| ACTUAL AGENT -->|reconcile| ACTUAL ACTUAL -->|status| AGENT
The pull-based GitOps reconciliation loop. The agent continuously compares Git (desired state) with the cluster (actual state) and corrects drift.
The reconciliation loop
The core of GitOps is the reconciliation loop. It runs continuously.
- Observe. The agent reads the desired state from Git.
- Compare. The agent diffs the desired state against the actual state in the cluster.
- Act. If they differ, the agent applies the changes to bring actual state in line with desired state.
- Report. The agent reports the sync status.
This loop runs every few minutes (configurable). It means the system is self-healing. If someone manually changes a resource in the cluster, the agent reverts it on the next reconciliation cycle.
Sync strategies
| Strategy | Behavior |
|---|---|
| Auto-sync | Agent applies changes immediately when drift is detected |
| Manual sync | Agent detects drift but waits for human approval |
| Auto-prune | Agent deletes resources removed from Git |
| Self-heal | Agent reverts manual cluster changes |
Most teams start with manual sync and auto-prune disabled, then enable automation as confidence grows.
Flux
Flux is a CNCF graduated project. It runs as a set of controllers inside Kubernetes.
Core components
- Source Controller: Watches Git repositories for changes.
- Kustomize Controller: Applies Kustomize overlays.
- Helm Controller: Manages Helm releases.
- Notification Controller: Sends alerts on events.
Basic setup
# GitRepository source
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: my-app
namespace: flux-system
spec:
interval: 1m
url: https://github.com/org/infra-repo
ref:
branch: main
---
# Kustomization to apply manifests
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: my-app
namespace: flux-system
spec:
interval: 5m
sourceRef:
kind: GitRepository
name: my-app
path: ./clusters/production
prune: true
Flux polls the Git repository every minute. When it detects a change, it applies the manifests from the specified path. The prune: true setting deletes resources that no longer exist in Git.
ArgoCD
ArgoCD provides a UI-driven experience on top of the same pull-based model.
Application definition
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: my-app
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/org/infra-repo
targetRevision: main
path: apps/my-app
destination:
server: https://kubernetes.default.svc
namespace: production
syncPolicy:
automated:
prune: true
selfHeal: true
ArgoCD features
- Web UI with real-time sync visualization.
- Application sets for generating applications from templates.
- Multi-cluster deployment from a single ArgoCD instance.
- Rollback to any previous Git commit.
- SSO integration for access control.
Repository structure
GitOps repositories typically follow one of two patterns.
Single repo (app + config)
my-app/
src/ # application code
k8s/ # Kubernetes manifests
base/
overlays/
staging/
production/
Simple but couples application and configuration releases.
Separate repos
# App repo: contains source code, Dockerfile, CI pipeline
my-app/
src/
Dockerfile
.github/workflows/ci.yml
# Config repo: contains deployment manifests
infra-config/
apps/
my-app/
base/
overlays/
staging/
production/
The CI pipeline in the app repo builds a container image and updates the image tag in the config repo. The GitOps agent watches the config repo and deploys the new image.
This separation provides cleaner audit trails and decouples build from deploy.
Secrets in GitOps
Storing secrets in Git is not an option. But GitOps requires everything to be in Git. Several approaches resolve this tension.
Sealed Secrets
Bitnami Sealed Secrets encrypts secrets with a cluster-specific key. The encrypted version is safe to commit.
# Encrypt a secret
kubeseal --format yaml < secret.yaml > sealed-secret.yaml
# Commit sealed-secret.yaml to Git
The Sealed Secrets controller in the cluster decrypts it at runtime.
SOPS (Secrets OPerationS)
Mozilla SOPS encrypts values within YAML/JSON files using AWS KMS, GCP KMS, Azure Key Vault, or PGP keys.
apiVersion: v1
kind: Secret
metadata:
name: db-credentials
data:
password: ENC[AES256_GCM,data:...,type:str]
Only the values are encrypted. Keys and structure remain readable. Flux has native SOPS integration.
External Secrets Operator
Fetches secrets from external stores (AWS Secrets Manager, HashiCorp Vault, Azure Key Vault) and creates Kubernetes secrets.
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: db-credentials
spec:
refreshInterval: 1h
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
target:
name: db-credentials
data:
- secretKey: password
remoteRef:
key: secret/data/db
property: password
The secret reference is committed to Git. The actual value lives in Vault.
Drift detection
Drift happens when the actual cluster state diverges from the desired state in Git. Causes include:
- Manual
kubectledits. - Operators that modify resources.
- Admission controllers that inject sidecars.
Handling drift
Self-heal mode (ArgoCD) or prune and force (Flux) revert drift automatically. This enforces Git as the source of truth.
Alert on drift without auto-correcting. Useful when some drift is intentional (auto-scaling, for example).
Teams must agree: is the cluster or Git authoritative? GitOps says Git. Any manual change is temporary.
GitOps workflow in practice
A typical deployment workflow:
- Developer opens a PR in the config repo changing the image tag.
- CI validates the manifests (kubeval, OPA policies).
- Reviewer approves. PR is merged.
- GitOps agent detects the change within its polling interval.
- Agent applies the updated manifests.
- Agent reports sync status.
- If something breaks, revert the PR. The agent deploys the previous state.
Rollback is a git revert. No special tooling needed.
Benefits and challenges
Benefits
- Audit trail: Every change is a commit with author, timestamp, and review.
- Rollback: Revert a commit to undo a deployment.
- Consistency: The same workflow for application code and infrastructure.
- Security: Cluster credentials stay inside the cluster (pull model).
- Developer experience: Familiar Git workflows for operations.
Challenges
- Secret management: Requires additional tooling.
- Debugging: “Why did this deploy?” requires tracing from cluster to Git commit.
- Learning curve: Teams must adopt declarative configuration.
- Tooling overhead: Running ArgoCD or Flux adds operational burden.
What comes next
This article concludes the Git and Version Control series. You have gone from Git’s internal object model through everyday commands, branching strategies, history rewriting, hooks, monorepo management, and now GitOps. Git is not just a version control tool. It is the foundation for modern software delivery. The patterns and practices covered here apply whether you are a solo developer or part of a thousand-person engineering organization.