Search…

GitOps

In this series (8 parts)
  1. Git internals: how Git actually works
  2. Everyday Git: the commands that matter
  3. Branching and merging
  4. Branching strategies for teams
  5. Git rebase and history rewriting
  6. Git hooks and automation
  7. Monorepos and large repo management
  8. GitOps

Everything in this series has treated Git as a tool for managing application source code. GitOps extends that role. In a GitOps workflow, Git is the single source of truth for infrastructure, configuration, and deployment state. Every change goes through a pull request. Every deployment is a commit.

For background on infrastructure as code, see the IaC introduction. For ArgoCD specifics, see ArgoCD.

What GitOps is

GitOps is an operational framework that applies DevOps best practices for application development (version control, collaboration, CI/CD) to infrastructure automation.

Four principles define GitOps:

  1. Declarative configuration. The desired state of the system is described declaratively (Kubernetes manifests, Terraform files, Helm charts).
  2. Versioned and immutable. The desired state is stored in Git. Every change is a commit with full audit history.
  3. Pulled automatically. Software agents automatically pull the desired state from Git and apply it.
  4. Continuously reconciled. Agents monitor the actual state and correct any drift from the desired state.

Push vs pull model

There are two approaches to applying changes from Git to the target environment.

Push-based deployment

A CI/CD pipeline detects a change in Git and pushes it to the target.

Developer commits -> CI pipeline runs -> Pipeline pushes to cluster

This is the traditional CD model. Jenkins, GitHub Actions, and GitLab CI operate this way. The pipeline needs credentials to the target environment.

Pull-based deployment

An agent running inside the target environment pulls the desired state from Git and applies it.

Developer commits -> Agent detects change -> Agent applies to cluster

The agent lives inside the cluster. It does not expose cluster credentials to the CI system. This is more secure and aligns with the principle of least privilege.

graph TD
subgraph "Git Repository"
  GR["Desired state<br/>(manifests)"]
end
subgraph "Kubernetes Cluster"
  AGENT["GitOps Agent<br/>(ArgoCD / Flux)"]
  ACTUAL["Running workloads"]
end
DEV["Developer"] -->|push commit| GR
AGENT -->|poll / webhook| GR
AGENT -->|compare desired<br/>vs actual| ACTUAL
AGENT -->|reconcile| ACTUAL
ACTUAL -->|status| AGENT

The pull-based GitOps reconciliation loop. The agent continuously compares Git (desired state) with the cluster (actual state) and corrects drift.

The reconciliation loop

The core of GitOps is the reconciliation loop. It runs continuously.

  1. Observe. The agent reads the desired state from Git.
  2. Compare. The agent diffs the desired state against the actual state in the cluster.
  3. Act. If they differ, the agent applies the changes to bring actual state in line with desired state.
  4. Report. The agent reports the sync status.

This loop runs every few minutes (configurable). It means the system is self-healing. If someone manually changes a resource in the cluster, the agent reverts it on the next reconciliation cycle.

Sync strategies

StrategyBehavior
Auto-syncAgent applies changes immediately when drift is detected
Manual syncAgent detects drift but waits for human approval
Auto-pruneAgent deletes resources removed from Git
Self-healAgent reverts manual cluster changes

Most teams start with manual sync and auto-prune disabled, then enable automation as confidence grows.

Flux

Flux is a CNCF graduated project. It runs as a set of controllers inside Kubernetes.

Core components

  • Source Controller: Watches Git repositories for changes.
  • Kustomize Controller: Applies Kustomize overlays.
  • Helm Controller: Manages Helm releases.
  • Notification Controller: Sends alerts on events.

Basic setup

# GitRepository source
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: my-app
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/org/infra-repo
  ref:
    branch: main
---
# Kustomization to apply manifests
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: my-app
  namespace: flux-system
spec:
  interval: 5m
  sourceRef:
    kind: GitRepository
    name: my-app
  path: ./clusters/production
  prune: true

Flux polls the Git repository every minute. When it detects a change, it applies the manifests from the specified path. The prune: true setting deletes resources that no longer exist in Git.

ArgoCD

ArgoCD provides a UI-driven experience on top of the same pull-based model.

Application definition

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/org/infra-repo
    targetRevision: main
    path: apps/my-app
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

ArgoCD features

  • Web UI with real-time sync visualization.
  • Application sets for generating applications from templates.
  • Multi-cluster deployment from a single ArgoCD instance.
  • Rollback to any previous Git commit.
  • SSO integration for access control.

Repository structure

GitOps repositories typically follow one of two patterns.

Single repo (app + config)

my-app/
  src/               # application code
  k8s/               # Kubernetes manifests
    base/
    overlays/
      staging/
      production/

Simple but couples application and configuration releases.

Separate repos

# App repo: contains source code, Dockerfile, CI pipeline
my-app/
  src/
  Dockerfile
  .github/workflows/ci.yml

# Config repo: contains deployment manifests
infra-config/
  apps/
    my-app/
      base/
      overlays/
        staging/
        production/

The CI pipeline in the app repo builds a container image and updates the image tag in the config repo. The GitOps agent watches the config repo and deploys the new image.

This separation provides cleaner audit trails and decouples build from deploy.

Secrets in GitOps

Storing secrets in Git is not an option. But GitOps requires everything to be in Git. Several approaches resolve this tension.

Sealed Secrets

Bitnami Sealed Secrets encrypts secrets with a cluster-specific key. The encrypted version is safe to commit.

# Encrypt a secret
kubeseal --format yaml < secret.yaml > sealed-secret.yaml
# Commit sealed-secret.yaml to Git

The Sealed Secrets controller in the cluster decrypts it at runtime.

SOPS (Secrets OPerationS)

Mozilla SOPS encrypts values within YAML/JSON files using AWS KMS, GCP KMS, Azure Key Vault, or PGP keys.

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
data:
  password: ENC[AES256_GCM,data:...,type:str]

Only the values are encrypted. Keys and structure remain readable. Flux has native SOPS integration.

External Secrets Operator

Fetches secrets from external stores (AWS Secrets Manager, HashiCorp Vault, Azure Key Vault) and creates Kubernetes secrets.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: db-credentials
  data:
    - secretKey: password
      remoteRef:
        key: secret/data/db
        property: password

The secret reference is committed to Git. The actual value lives in Vault.

Drift detection

Drift happens when the actual cluster state diverges from the desired state in Git. Causes include:

  • Manual kubectl edits.
  • Operators that modify resources.
  • Admission controllers that inject sidecars.

Handling drift

Self-heal mode (ArgoCD) or prune and force (Flux) revert drift automatically. This enforces Git as the source of truth.

Alert on drift without auto-correcting. Useful when some drift is intentional (auto-scaling, for example).

Teams must agree: is the cluster or Git authoritative? GitOps says Git. Any manual change is temporary.

GitOps workflow in practice

A typical deployment workflow:

  1. Developer opens a PR in the config repo changing the image tag.
  2. CI validates the manifests (kubeval, OPA policies).
  3. Reviewer approves. PR is merged.
  4. GitOps agent detects the change within its polling interval.
  5. Agent applies the updated manifests.
  6. Agent reports sync status.
  7. If something breaks, revert the PR. The agent deploys the previous state.

Rollback is a git revert. No special tooling needed.

Benefits and challenges

Benefits

  • Audit trail: Every change is a commit with author, timestamp, and review.
  • Rollback: Revert a commit to undo a deployment.
  • Consistency: The same workflow for application code and infrastructure.
  • Security: Cluster credentials stay inside the cluster (pull model).
  • Developer experience: Familiar Git workflows for operations.

Challenges

  • Secret management: Requires additional tooling.
  • Debugging: “Why did this deploy?” requires tracing from cluster to Git commit.
  • Learning curve: Teams must adopt declarative configuration.
  • Tooling overhead: Running ArgoCD or Flux adds operational burden.

What comes next

This article concludes the Git and Version Control series. You have gone from Git’s internal object model through everyday commands, branching strategies, history rewriting, hooks, monorepo management, and now GitOps. Git is not just a version control tool. It is the foundation for modern software delivery. The patterns and practices covered here apply whether you are a solo developer or part of a thousand-person engineering organization.

Start typing to search across all content
navigate Enter open Esc close