Scaling Argo CD GitOps Across 34 Kubernetes Clusters

Running GitOps across dozens of clusters changes the central engineering problem: the challenge is less about making a controller reconcile state and more about controlling the reach of every change. At Lucid Motors, one Argo CD instance managed 34 clusters spread across 22 AWS accounts and an Oracle Cloud tenancy, spanning five regions on three continents. A repository with 174 Helm charts produced 1,214 Argo CD Application objects.

That scale makes ordinary shortcuts consequential. A chart value that works in one environment can break many others; a default deletion policy can turn a routine edit into a fleet-wide incident. The design that held this system together rested on three ideas: make each cluster conform to a small contract, render the full impact of changes before merge, and keep failure within a manageable boundary.

Make the cluster contract the source of truth

Each cluster was represented by a values file containing about 20 keys. The fields described its identity, account boundary, region, environment, API access, DNS and certificate delegation, and any opt-outs. From that contract, the platform derived the cluster’s Applications, IAM role names, Cluster API identity, DNS delegation and cost tags.

This kept related settings from drifting into separate configuration paths. Adding a cluster to a fleet meant changing a label on its registration. Adding a component to a fleet meant adding one line to a list. ApplicationSet matrix generators handled the resulting fan-out.

The same contract also tested whether the platform could absorb a meaningful exception. A Middle East expansion added an Oracle Cloud tenancy to the fleet. It joined through the established cluster model rather than a one-off configuration path. That’s a practical measure of a contract’s value: it can accommodate a new boundary without turning that boundary into a special case.

Centralize deployment while keeping authority local

The hub contained the only Argo CD instance responsible for platform components. It held cluster registrations, the identity used by the Cluster API controller, and three ApplicationSets. One deployed a per-cluster baseline through an app-of-apps chart. A second placed shared platform services on the clusters that hosted them. The third managed the Cluster API definitions for the clusters themselves.

Spoke clusters retained control over whether to trust the hub. Each spoke’s authentication configuration decided whether to admit the hub’s role; the spoke also owned the role Cluster API used within it. Workloads received cloud roles through the identity provider configured on their own cluster.

On AWS, the hub stored no long-lived cross-account credentials. It used short-lived credentials, while each access grant was declared and reviewed in Git on the spoke side. Oracle Cloud did not offer the same arrangement in this deployment: cluster registration still read a stored credential from a vault. That difference matters in a multi-provider design. Documenting where the security model changes makes the exception visible and gives teams a clear place to improve it.

Make CI show the rendered fleet impact

With 174 charts in play, reviewing template edits alone would leave too much guesswork. A values change might render successfully for one cluster and fail across many others. The team addressed that risk by making CI render the complete configuration graph on every merge request.

CI checked each chart independently, rendered each app-of-apps chart against every cluster contract, rendered cluster definitions, and rendered the charts referenced by Applications with the exact values those Applications would receive. This approach tested the actual combinations in use, including the interactions between shared charts and cluster-specific configuration.

CI also rendered the target branch and attached the resulting manifest diff to the merge request. Reviewers could inspect the Kubernetes resources that would change on clusters, rather than trying to infer deployment effects from Helm template edits. That shifts review toward the operational consequence of a change, where a mistaken value or unexpected resource update is easier to spot.

Measure drift without overstating the result

Reported out-of-band drift fell from 24.3 incidents per month before cutover to 0.9 afterward, based on two 90-day periods. The figures point in a favorable direction, but they don’t support a clean percentage comparison: the earlier count came from incident tickets tagged as drift, and tagging was incomplete. The later count came from controller diff events with a confirmed out-of-band cause.

The useful operational finding is that instrumented drift was rare after the change, and the controller corrected every confirmed case within its reconciliation interval without manual intervention. For teams establishing their own baseline, consistent measurement before and after a rollout would make the comparison more reliable.

Limit destructive actions and isolate outages

At fleet scale, the most concerning failure can be a validly merged but harmful commit. The deployment model therefore constrained what reconciliation could delete. ApplicationSets preserved deployed resources when an entry was removed, preventing that removal from cascading into resource deletion. Cluster objects carried a no-prune setting, so reverting a commit could not delete a running cluster.

The principle is to match automated action to its blast radius and reversibility. Controllers should still converge on declared state, but the consequences of removing an entry should reflect the cost of losing the resources behind it.

Git availability is another boundary to plan around. GitHub had an outage on Oct. 7, 2026, affecting Actions, Git operations and pull requests, following smaller incidents on Oct. 5 and 6. Red Hat had published a walkthrough on Oct. 6 showing Argo CD ApplicationSets provisioning hosted control planes from Git. Together, those events underscore how much infrastructure now depends on Git while also depending on it being reachable.

In a hub-and-spoke setup, workloads run in the spokes. If Git or the hub becomes unavailable, workloads already running can continue, while GitOps-driven deployments and configuration changes pause. The design objective is to prevent a control-plane interruption from becoming a workload interruption. The hub deserves particular protection, too: a compromised repository or CI pipeline can inherit the hub’s reach once its change is merged.

Onboarding becomes a repeatable operation

Adding a cluster followed four steps: create the patterned IAM roles once in Terraform, write the cluster’s values file, open a merge request for CI to render the graph and publish its diff, then merge and label the registration. Older clusters that had been built manually could be adopted in place through an explicit import path.

Mean onboarding time was reported as 4.0 days across eight earlier clusters and 1.7 hours across eleven clusters after cutover. Those figures come from different methods and small samples: the earlier estimate was reconstructed from change tickets, while the later measure ran from merge to first successful sync. The cutover also removed manual approval steps, so GitOps alone cannot account for the full change. The evidence supports a faster, more repeatable process, while the exact size of the improvement remains uncertain.

The broader lesson is that cluster count magnifies the cost of informal practices. A credential kept in the hub, a values file that CI never renders, or an unchecked prune default may seem manageable in a small deployment. Across 34 clusters, each can become a fleet-level incident. A small contract, full rendered diffs and deliberate deletion behavior make the control plane easier to extend while keeping its failure modes contained.