GitOps Workflow for Kubernetes: Implementing Continuous Delivery with Flux and ArgoCD
GitOps has emerged as a powerful paradigm for managing Kubernetes infrastructure and applications. By using Git as the single source of truth for declarative infrastructure and applications, GitOps enables teams to increase deployment velocity, improve system reliability, and enhance security posture. This article provides a comprehensive guide to implementing GitOps workflows in enterprise environments using popular tools like Flux and ArgoCD.
Understanding GitOps Principles
GitOps is built on four key principles:
- Declarative Configuration: The entire system is described declaratively, typically using Kubernetes manifests, Helm charts, or Kustomize configurations.
- Version Control: The declarative configuration is stored in Git, providing a single source of truth with full audit trail and version control.
- Automated Synchronization: Approved changes to the Git repository are automatically applied to the system.
- Convergence: Software agents continuously monitor the actual system state and reconcile it with the desired state defined in Git.
graph LR
A[Developers] -->|Push Changes| B[Git Repository]
B -->|Webhook| C[CI Pipeline]
C -->|Validate & Test| B
D[GitOps Operator] -->|Pull Changes| B
D -->|Apply Changes| E[Kubernetes Cluster]
D -->|Report Status| B
F[Drift Detection] -->|Monitor| E
F -->|Reconcile| D
Repository Structure for GitOps
A well-organized repository structure is crucial for GitOps success. Here’s a recommended structure for enterprise environments:
├── clusters/
│ ├── production/
│ │ ├── infrastructure/
│ │ │ ├── namespaces.yaml
│ │ │ ├── network-policies/
│ │ │ ├── storage-classes/
│ │ │ └── rbac/
│ │ ├── applications/
│ │ │ ├── team-a/
│ │ │ ├── team-b/
│ │ │ └── shared/
│ │ └── kustomization.yaml
│ ├── staging/
│ │ ├── infrastructure/
│ │ ├── applications/
│ │ └── kustomization.yaml
│ └── development/
│ ├── infrastructure/
│ ├── applications/
│ └── kustomization.yaml
├── base/
│ ├── applications/
│ │ ├── app-a/
│ │ │ ├── deployment.yaml
│ │ │ ├── service.yaml
│ │ │ ├── ingress.yaml
│ │ │ └── kustomization.yaml
│ │ └── app-b/
│ └── infrastructure/
│ ├── cert-manager/
│ ├── ingress-controller/
│ ├── monitoring/
│ └── logging/
└── .github/
└── workflows/
├── validate.yaml
└── preview.yaml
Implementing GitOps with Flux
Flux is a set of continuous and progressive delivery solutions for Kubernetes that are open and extensible. Let’s implement a GitOps workflow using Flux v2.
Installing Flux CLI
# macOS
brew install fluxcd/tap/flux
# Linux
curl -s https://fluxcd.io/install.sh | sudo bash
Bootstrapping Flux on a Kubernetes Cluster
# Export your GitHub personal access token
export GITHUB_TOKEN=<your-token>
# Bootstrap Flux
flux bootstrap github \
--owner=your-github-username \
--repository=gitops-fleet \
--branch=main \
--path=clusters/production \
--personal
Creating a Flux Source
# clusters/production/infrastructure/sources/git-repository.yaml
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
name: infrastructure-repo
namespace: flux-system
spec:
interval: 1m
url: https://github.com/your-org/infrastructure-repo
ref:
branch: main
secretRef:
name: github-credentials
Defining Kustomization for Infrastructure Components
# clusters/production/infrastructure/kustomization.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
name: infrastructure
namespace: flux-system
spec:
interval: 10m
path: ./clusters/production/infrastructure
prune: true
sourceRef:
kind: GitRepository
name: infrastructure-repo
validation: client
healthChecks:
- apiVersion: apps/v1
kind: Deployment
name: ingress-nginx-controller
namespace: ingress-nginx
Implementing Helm Releases with Flux
# clusters/production/applications/team-a/helm-release.yaml
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
name: bitnami
namespace: team-a
spec:
interval: 30m
url: https://charts.bitnami.com/bitnami
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: redis
namespace: team-a
spec:
interval: 5m
chart:
spec:
chart: redis
version: "16.x"
sourceRef:
kind: HelmRepository
name: bitnami
namespace: team-a
values:
architecture: standalone
auth:
enabled: true
password: ${REDIS_PASSWORD}
master:
persistence:
enabled: true
storageClass: "ebs-sc"
size: 8Gi
metrics:
enabled: true
serviceMonitor:
enabled: true
Flux Notification Controller for Alerts
# clusters/production/infrastructure/notifications/slack-alert.yaml
apiVersion: notification.toolkit.fluxcd.io/v1beta2
kind: Provider
metadata:
name: slack
namespace: flux-system
spec:
type: slack
channel: gitops-notifications
secretRef:
name: slack-url
---
apiVersion: notification.toolkit.fluxcd.io/v1beta2
kind: Alert
metadata:
name: on-call-webapp
namespace: flux-system
spec:
providerRef:
name: slack
eventSeverity: info
eventSources:
- kind: GitRepository
name: webapp
- kind: Kustomization
name: webapp
- kind: HelmRelease
name: webapp
namespace: default
exclusionList:
- ".*upgrade.*succeeded.*"
Image Automation with Flux
# clusters/production/applications/team-a/image-automation.yaml
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageRepository
metadata:
name: webapp
namespace: flux-system
spec:
image: ghcr.io/your-org/webapp
interval: 1m
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImagePolicy
metadata:
name: webapp
namespace: flux-system
spec:
imageRepositoryRef:
name: webapp
policy:
semver:
range: '>=1.0.0 <2.0.0'
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
name: webapp
namespace: flux-system
spec:
interval: 1m
sourceRef:
kind: GitRepository
name: flux-system
git:
checkout:
ref:
branch: main
commit:
author:
email: fluxcdbot@users.noreply.github.com
name: fluxcdbot
messageTemplate: 'Update webapp image to {{.NewTag}}'
push:
branch: main
update:
path: ./clusters/production/applications/team-a
strategy: Setters
Implementing GitOps with ArgoCD
ArgoCD is a declarative, GitOps continuous delivery tool for Kubernetes. Let’s implement a GitOps workflow using ArgoCD.
Installing ArgoCD
# argocd/install.yaml
apiVersion: v1
kind: Namespace
metadata:
name: argocd
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
patchesStrategicMerge:
- |-
apiVersion: apps/v1
kind: Deployment
metadata:
name: argocd-server
namespace: argocd
spec:
template:
spec:
containers:
- name: argocd-server
command:
- argocd-server
- --insecure
- --staticassets
- /shared/app
Configuring ArgoCD Projects
# argocd/projects/team-a.yaml
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: team-a
namespace: argocd
spec:
description: Team A Applications
sourceRepos:
- 'https://github.com/your-org/gitops-repo'
destinations:
- namespace: team-a-*
server: https://kubernetes.default.svc
- namespace: team-a
server: https://kubernetes.default.svc
clusterResourceWhitelist:
- group: '*'
kind: 'Namespace'
namespaceResourceBlacklist:
- group: ''
kind: 'ResourceQuota'
- group: ''
kind: 'LimitRange'
roles:
- name: developer
description: Developer role
policies:
- p, proj:team-a:developer, applications, get, team-a/*, allow
- p, proj:team-a:developer, applications, sync, team-a/*, allow
groups:
- team-a
Defining ArgoCD Applications
# argocd/applications/team-a-webapp.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: team-a-webapp
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: team-a
source:
repoURL: https://github.com/your-org/gitops-repo
targetRevision: HEAD
path: clusters/production/applications/team-a/webapp
destination:
server: https://kubernetes.default.svc
namespace: team-a
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
- PruneLast=true
- ApplyOutOfSyncOnly=true
retry:
limit: 5
backoff:
duration: 5s
factor: 2
maxDuration: 3m
ArgoCD ApplicationSet for Multi-Environment Deployments
# argocd/applicationsets/webapp-all-environments.yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: webapp-all-environments
namespace: argocd
spec:
generators:
- list:
elements:
- environment: development
namespace: team-a-dev
- environment: staging
namespace: team-a-staging
- environment: production
namespace: team-a
template:
metadata:
name: 'webapp-{{environment}}'
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: team-a
source:
repoURL: https://github.com/your-org/gitops-repo
targetRevision: HEAD
path: clusters/{{environment}}/applications/team-a/webapp
destination:
server: https://kubernetes.default.svc
namespace: '{{namespace}}'
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
Implementing a Comprehensive GitOps Workflow
Let’s implement a complete GitOps workflow that combines CI/CD with GitOps principles.
CI Pipeline with GitHub Actions
# .github/workflows/ci.yaml
name: CI Pipeline
on:
push:
branches: [ main ]
paths:
- 'src/**'
pull_request:
branches: [ main ]
paths:
- 'src/**'
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
- name: Login to GitHub Container Registry
uses: docker/login-action@v2
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract metadata
id: meta
uses: docker/metadata-action@v4
with:
images: ghcr.io/${{ github.repository }}/webapp
tags: |
type=semver,pattern={{version}}
type=sha,format=short
- name: Build and push
uses: docker/build-push-action@v3
with:
context: ./src
push: ${{ github.event_name != 'pull_request' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
CD Pipeline for Manifest Generation
# .github/workflows/cd.yaml
name: CD Pipeline
on:
workflow_run:
workflows: ["CI Pipeline"]
types:
- completed
branches: [main]
jobs:
update-manifests:
if: ${{ github.event.workflow_run.conclusion == 'success' }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Get image tag
id: image
run: |
IMAGE_TAG=$(curl -s https://api.github.com/repos/${{ github.repository }}/actions/runs/${{ github.event.workflow_run.id }}/artifacts | jq -r '.artifacts[] | select(.name == "image-tag") | .archive_download_url')
curl -L -H "Authorization: token ${{ secrets.GITHUB_TOKEN }}" $IMAGE_TAG -o image-tag.zip
unzip image-tag.zip
echo "tag=$(cat image-tag.txt)" >> $GITHUB_OUTPUT
- name: Update Kubernetes manifests
run: |
cd base/applications/webapp
kustomize edit set image ghcr.io/${{ github.repository }}/webapp:${{ steps.image.outputs.tag }}
- name: Commit changes
uses: stefanzweifel/git-auto-commit-action@v4
with:
commit_message: "Update webapp image to ${{ steps.image.outputs.tag }}"
file_pattern: base/applications/webapp/kustomization.yaml
Implementing Progressive Delivery with Flagger
# clusters/production/applications/team-a/webapp/canary.yaml
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
name: webapp
namespace: team-a
spec:
provider: istio
targetRef:
apiVersion: apps/v1
kind: Deployment
name: webapp
progressDeadlineSeconds: 600
service:
port: 80
targetPort: 8080
gateways:
- istio-system/public-gateway
hosts:
- webapp.example.com
analysis:
interval: 30s
threshold: 10
maxWeight: 50
stepWeight: 5
metrics:
- name: request-success-rate
thresholdRange:
min: 99
interval: 1m
- name: request-duration
thresholdRange:
max: 500
interval: 1m
webhooks:
- name: load-test
url: http://flagger-loadtester.test/
timeout: 15s
metadata:
cmd: "hey -z 1m -q 10 -c 2 http://webapp-canary.team-a:80/"
GitOps Security Best Practices
Implementing GitOps securely requires attention to several key areas:
1. Secure Git Repository Access
# clusters/production/infrastructure/sources/git-repository-with-ssh.yaml
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
name: secure-infrastructure-repo
namespace: flux-system
spec:
interval: 1m
url: ssh://git@github.com/your-org/infrastructure-repo
secretRef:
name: ssh-credentials
2. Implement Sealed Secrets for Sensitive Data
# Install Sealed Secrets controller
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.18.0/controller.yaml
# Example of a sealed secret
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
name: database-credentials
namespace: team-a
spec:
encryptedData:
username: AgBy8hCM8FQUo3...
password: AgCZdvtCLtZ7Q3...
template:
metadata:
name: database-credentials
namespace: team-a
type: Opaque
3. Implement Policy Enforcement with OPA Gatekeeper
# clusters/production/infrastructure/policies/require-labels.yaml
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
name: require-team-label
spec:
match:
kinds:
- apiGroups: [""]
kinds: ["Namespace"]
excludedNamespaces:
- kube-system
- flux-system
- argocd
parameters:
labels:
- key: team
allowedRegex: "^(team-a|team-b|platform)$"
4. Implement RBAC for GitOps Operators
# clusters/production/infrastructure/rbac/flux-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: flux-reconciler
namespace: flux-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: flux-reconciler
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]
# Add more granular permissions as needed
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: flux-reconciler
subjects:
- kind: ServiceAccount
name: flux-reconciler
namespace: flux-system
roleRef:
kind: ClusterRole
name: flux-reconciler
apiGroup: rbac.authorization.k8s.io
Monitoring and Observability for GitOps
Effective monitoring is crucial for GitOps workflows. Here’s how to implement comprehensive observability:
1. Prometheus Monitoring for GitOps Operators
# clusters/production/infrastructure/monitoring/gitops-monitoring.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: flux-system
namespace: monitoring
spec:
selector:
matchLabels:
app: flux
namespaceSelector:
matchNames:
- flux-system
endpoints:
- port: http
interval: 30s
2. Grafana Dashboard for GitOps
{
"annotations": {
"list": []
},
"editable": true,
"gnetId": null,
"graphTooltip": 0,
"id": 1,
"links": [],
"panels": [
{
"datasource": "Prometheus",
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 1
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 0
},
"id": 2,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"text": {},
"textMode": "auto"
},
"pluginVersion": "7.5.5",
"targets": [
{
"expr": "sum(flux_reconcile_condition{type=\"Ready\",status=\"False\"})",
"interval": "",
"legendFormat": "",
"refId": "A"
}
],
"title": "Not Ready Reconciliations",
"type": "stat"
}
],
"schemaVersion": 27,
"style": "dark",
"tags": [],
"templating": {
"list": []
},
"time": {
"from": "now-6h",
"to": "now"
},
"timepicker": {},
"timezone": "",
"title": "GitOps Dashboard",
"uid": "gitops",
"version": 1
}
3. Alerting for GitOps Issues
# clusters/production/infrastructure/monitoring/gitops-alerts.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: gitops-alerts
namespace: monitoring
spec:
groups:
- name: gitops.rules
rules:
- alert: GitOpsReconciliationFailure
expr: flux_reconcile_condition{type="Ready",status="False"} == 1
for: 15m
labels:
severity: critical
team: platform
annotations:
summary: "GitOps reconciliation failure"
description: "{{ $labels.namespace }}/{{ $labels.name }} has been failing to reconcile for more than 15 minutes"
- alert: GitRepositoryFetchFailure
expr: flux_reconcile_condition{kind="GitRepository",type="Ready",status="False"} == 1
for: 5m
labels:
severity: critical
team: platform
annotations:
summary: "Git repository fetch failure"
description: "{{ $labels.namespace }}/{{ $labels.name }} has been failing to fetch for more than 5 minutes"
Implementing GitOps at Scale
For large enterprises with multiple teams and clusters, consider these strategies:
1. Multi-Cluster Management with Cluster API and GitOps
# clusters/management/cluster-api/clusters/production-cluster.yaml
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
name: production-cluster
namespace: clusters
spec:
clusterNetwork:
pods:
cidrBlocks: ["192.168.0.0/16"]
services:
cidrBlocks: ["10.96.0.0/12"]
controlPlaneRef:
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
name: production-control-plane
infrastructureRef:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AWSCluster
name: production-cluster
---
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
name: production-control-plane
namespace: clusters
spec:
replicas: 3
version: v1.24.0
machineTemplate:
infrastructureRef:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AWSMachineTemplate
name: production-control-plane
kubeadmConfigSpec:
clusterConfiguration:
apiServer:
extraArgs:
cloud-provider: aws
controllerManager:
extraArgs:
cloud-provider: aws
2. Fleet Management with ArgoCD ApplicationSets
# argocd/applicationsets/fleet-applications.yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: fleet-applications
namespace: argocd
spec:
generators:
- matrix:
generators:
- git:
repoURL: https://github.com/your-org/gitops-repo
revision: HEAD
directories:
- path: clusters/*/applications/*
- clusters:
selector:
matchLabels:
environment: production
template:
metadata:
name: '{{path.basename}}-{{path.basenameNormalized}}'
spec:
project: default
source:
repoURL: https://github.com/your-org/gitops-repo
targetRevision: HEAD
path: '{{path}}'
destination:
server: '{{server}}'
namespace: '{{path.basename}}'
syncPolicy:
automated:
prune: true
selfHeal: true
Conclusion
GitOps provides a powerful framework for managing Kubernetes infrastructure and applications. By leveraging Git as the single source of truth and implementing automated reconciliation with tools like Flux and ArgoCD, organizations can achieve faster, more reliable, and more secure deployments.
Key takeaways from this guide:
- Start with a well-structured repository that clearly separates infrastructure from applications and environments.
- Choose the right GitOps tool based on your organization’s needs and existing workflows.
- Implement proper security controls including RBAC, sealed secrets, and policy enforcement.
- Set up comprehensive monitoring to quickly identify and resolve issues in your GitOps workflow.
- Scale your GitOps implementation with multi-cluster management and fleet management tools.
By following these best practices, you can successfully implement GitOps in your organization and realize the benefits of increased deployment velocity, improved system reliability, and enhanced security posture.