GitOps Practices
GitOps is the foundational delivery model for Kubernetes platforms at scale. This section covers the principles, repository patterns, and CI/CD workflows I use to keep platforms declarative, auditable, and drift-free.
What is GitOps
I stick to the canonical definitions from the GitOps community (e.g., Codefresh and OpenGitOps): desired state lives in Git, and a pull‑based controller continuously reconciles clusters to match that state.
The value is simple: auditability, automated drift reconciliation, safe rollbacks, and repeatability.
Best Practices
- Fully declarative everything: clusters, platform components, and app workloads.
- Separate source code, infra, argocd app crds and platform manifests into dedicated repositories. Different lifecycles and different owners means strict separation of concerns.
- Eliminate drift with Kustomize or Helm and reconciliation. Kubernetes' superpower is reconciliation and a consistent API. Use it.
- Environment folders, not long‑lived branches. Promotion is a PR from
dev→stage→prodwith version pins. Branches are for work, not for state. - Trunk‑based development. Keep changes small, reviewable, and flowing.
- Agents reconcile automatically and are the only writers to the cluster. Humans commit to Git, controllers write the cluster. This closed loop eliminates silent configuration drift.
Repository Structure
You want clear separation of concerns. Infra and app manifests should not live in the same repo. I use three repos for the platform layer and then 2 repos per dev team or domain.
| Repository | Purpose | Ownership |
|---|---|---|
platform | Installs Argo CD and core platform components | Platform (cluster-admin) |
argocd-apps | Argo CD root application (app-of-apps pattern) | Platform |
k8s-resources | Platform workload manifests (custom apps, post-deploy tasks, etc.) | Platform |
<team>-argocd-apps | Team Argo CD root app (projects, RBAC, cluster targets scaffolded) | Dev Team |
<team>-k8s-resources | App manifests (Helm/Kustomize) + optional CI that bumps env version pins | Dev Team |
Why this split?
- Platform is minimal and stable. It brings up Argo CD and only the bare minimum required components for argocd to run (ESO, ingress controller, ...).
- Environments are the source of truth for what runs where.
- ArgoCD root apps hold all the config related to RBAC. When onboarding a new dev team a default scaffolding template is provided by the platform team that has the correct argocd projects and cluster names pre-configured.
- Team app repos own their runtime and declare only what they need, not how the platform is built.
CI/CD Separation
Teams who grew up on gitflow often practiced Continuous Deployment: pipelines that built, tested, and deployed straight into production by committing to the production branch. They relied on long-lived branches to promote between environments.
The OpenGitOps principles shift the paradigm to Continuous Delivery, where pipelines produce artifacts and update desired state in Git, while agents handle deployment through reconciliation.
Key Distinction
The pipeline handles Continuous Integration.
The agent handles Continuous Delivery.
Keeping them separate keeps runtime state aligned with Git at all times.
Treat it as two feedback loops that reinforce each other:
- CI produces artifacts that are provably ready.
- CD reconciles the cluster until it matches the declared state.
Continuous Integration
Developers commit into a source repository that owns the CI workflow. That pipeline runs the usual CI tasks:
scan -> test -> build -> sign -> publish to artifactory
The pipeline publishes signed artifacts to a registry or artifact store. That is what gives the agent something trustworthy to pull later. In regulated environments it also creates an evidence trail for auditors.
Continuous Delivery
Once a new version is ready, you can either tweak the desired state yourself or let CI handle it by updating the version tag in your manifests or Helm values.
That promotion logic allows your development environment to act as a preview environment automatically, with the CI pipeline overlapping into the CD process, but only as a trigger not by mutating the cluster directly. Alternatively, you can configure Argo CD's ApplicationSets to detect the change from a pull request and deploy the corresponding applications. This enables extra functionalities like better sandboxing and auto-cleanup.
Agent driven continuous delivery handles execution. The GitOps agent watches the Git repository, pulls desired state, and reconciles until the cluster matches. With a trunk based flow Argo CD only needs to watch the default branch. Use a folder per environment with Kustomize overlays or Helm charts. I default to Kustomize and wrap any Helm charts with Kustomize when I need them.
Promotions become copying or adjusting version pins across environment overlays. Rollbacks are a simple git revert on the manifest change.
Adopting this model consistently leads to fewer production surprises and less time chasing config snowflakes.
Why This Matters
WARNING
Push‑based pipelines that apply manifests directly to clusters are an anti‑pattern for Kubernetes. They fight reconciliation, hide drift and fragment audit. If you want to adopt a push based approach, reconsider using Kubernetes.
Kubernetes is built around a control loop reconciliation model where controllers continuously watch cluster state and work to move the current state closer to the desired state.
This is a fundamental design principle that should be taken into account when we imagine how we want to deploy software on kubernetes. I therefore advocate for a pull-based approach with agents that reconcile cluster state against Git. This naturally aligns with the GitOps model.