Overview
Managing microservice deployments across disparate Kubernetes clusters (AKS, GKE, and EKS) frequently introduces operational drift, divergent deployment tooling, and inconsistent secret injection mechanisms.
This project established a declarative, automated, and secure Kubernetes GitOps delivery platform where cluster configurations and workloads are defined and reconciled as version-controlled software.
Platform Goals
- Reduce Configuration Drift: Establish declarative source-of-truth repositories where cluster configurations are continuously reconciled via GitOps.
- Standardize Workload Delivery: Provide application teams with structured delivery patterns across development, staging, and production clusters.
- Progressive Rollouts: Support canary rollout workflows to stage service revisions safely during deployment windows.
- Unified Secret Management: Prevent credentials from being committed to source repositories while standardizing secret synchronization across cloud providers.
- Operate the Cluster: Manage ingress, TLS certificates, autoscaling, policy, and telemetry as versioned platform services, not one-time cluster setup.
Architecture & Workflows
GitOps Core: Argo CD ApplicationSets & Helmfile
We structured cluster deployment workflows using the ApplicationSet pattern in Argo CD alongside Helmfile and Kustomize:
Cluster Services After Provisioning
A running AKS, GKE, or EKS control plane was only the starting point. The platform repository also managed the controllers and configuration that made each cluster usable for applications:
- Traffic and TLS: Ingress controllers routed requests to workloads; cert-manager managed certificate resources. Environment-specific routing and TLS configuration lived alongside application definitions. The platform initially used Ingress NGINX and later began moving toward Traefik, Istio, and Gateway API patterns where appropriate.
- Scaling and policy: KEDA supplied event-driven autoscaling, while Kyverno enforced workload policy. Cluster-specific node, GPU, storage, and ComputeClass configuration stayed in the same GitOps operating model.
- Secrets and visibility: External Secrets Operator synchronized Vault-backed references into workload namespaces. Observability components provided metrics, logs, and traces so operators could inspect controller and workload health after reconciliation.
Platform services and application workloads had separate Argo CD ownership boundaries. That let us update a controller or cluster-specific setting without treating every application as one release.
The release path kept distinct states: pipelines produced artifacts and scanned images with Trivy, Git configuration described the target version, Argo CD reconciled that version, and Argo Rollouts managed traffic during a canary release. Production deployment required approval. A healthy container image alone did not mean the route, certificate, secret reference, or rollout was ready.
Progressive Delivery with Argo Rollouts
To improve deployment safety beyond standard Kubernetes rolling updates, we integrated Argo Rollouts:
- Canary Deployments: Configured Argo Rollout resources with dedicated stable and canary service bindings to expose new workload revisions to targeted traffic subsets.
- Controlled Progression: Allows testing and validation of newly deployed revisions in live environments before shifting full traffic.
- Rollback Capability: Supports returning traffic to the previous stable ReplicaSet when a rollout is aborted.
The reusable Helm templates wired stable and canary Services to each Rollout. That contract mattered as much as the Rollout object: routing needed both Services before a revision could progress safely.
Declarative Secret Synchronization
To prevent hardcoded credentials in deployment manifests:
- Workloads utilize the External Secrets Operator (ESO) integrated with HashiCorp Vault.
- The operator authenticates against Vault using Kubernetes Service Account tokens and Azure Workload Identity patterns.
- ExternalSecrets resources define secret references declaratively in Git, with native Kubernetes secrets created dynamically in target namespaces.
Decisions & Trade-offs
| Decision | Why We Made It | Trade-off / Mitigation |
|---|---|---|
| Argo CD ApplicationSets | Generated repeated application definitions across cluster and environment targets. | Centralized reconciliation requires explicit ownership boundaries and controller health monitoring. |
| Helmfile + Kustomize Hybrid | Helmfile manages platform add-ons (Traefik, Prometheus); Kustomize overlays manage environment-specific configurations without excessive chart templating. | Requires team familiarity with both tools; mitigated through standardized repo templates and linting checks. |
Outcomes & Verification
- Declarative Reconciliation: Cluster drift is automatically identified and corrected according to Git source definitions.
- Consistent Delivery Workflow: Standardized release procedures across AKS, GKE, and EKS clusters.
- Secret References in Git: Kept secret values out of workload repositories while External Secrets Operator materialized values from Vault for Kubernetes workloads.