Configuration Drift

What is configuration drift?

Configuration drift is the gradual divergence of a system’s actual state from its intended or defined state, as undocumented manual changes accumulate over time. A server, cluster, or environment starts in a known configuration, but then someone SSHes in to fix an urgent issue, tweaks a setting in a console, installs a package by hand, or applies a quick patch — and each of these small, unrecorded changes moves reality further from the documented baseline. Multiply that across many servers and months, and no environment quite matches its definition anymore. Drift is the silent root cause of a whole class of frustrating, hard-to-debug problems.

Why drift is dangerous

The danger of drift is that it makes systems inconsistent, unpredictable, and irreproducible. The classic symptom is “works in staging but fails in production” (or “works on this server but not that one”) — because the environments have silently drifted apart, code that runs fine in one place breaks in another, and the cause is invisible because the differences were never recorded. Drift also breaks infrastructure-as-code: if your Terraform says the infrastructure should look one way but someone changed it manually, the IaC and reality disagree — the next terraform apply may try to “correct” the manual change (potentially undoing an important fix) or fail, and your source-of-truth config no longer describes what’s actually running. It undermines security and compliance (a manually-relaxed firewall rule or a disabled logging setting that drifted from the secure baseline creates an unaudited hole), makes incident response harder (you can’t trust the documented config when diagnosing), and makes disaster recovery unreliable (rebuilding “from the definition” produces something different from what was actually running). In short, drift erodes the fundamental assumption that you know what your systems look like.

Preventing and detecting drift

There are two complementary defenses, and the modern answer leans hard on the first. Prevent drift by construction with immutable infrastructure: if servers are never modified after deployment — changes require building a new image and replacing instances — then drift literally cannot accumulate, because nothing is ever changed in place. This is why immutable infrastructure and containers are the strongest antidote: they remove the “manually change a running server” step that causes drift in the first place. Combined with GitOps (where a controller continuously reconciles the live system back to the Git-declared state, actively reverting any drift) and IaC as the single source of truth, drift is designed out. Detect drift where it can still occur: tools like AWS Config continuously record configuration and flag divergence from desired baselines, terraform plan reveals differences between IaC and reality, and GitOps operators surface (and heal) drift automatically. The essential cultural discipline underpinning all of it is stopping manual changes: routing every change through code (IaC, GitOps, image builds) rather than ad-hoc console/SSH edits — the technical tools only work if the team actually commits to not hand-editing production. The recurring mistakes: making “just one quick manual fix” (the origin of nearly all drift), having IaC but bypassing it with console changes (so the code lies about reality), and no drift detection at all (so divergence is discovered only when something breaks).

What people get wrong

• The “just one quick fix” habit — every manual, unrecorded change is drift; the discipline of routing all changes through code (IaC/GitOps/images) is what actually prevents it.
• Having IaC but bypassing it: console or SSH changes that go around Terraform make the source-of-truth config disagree with reality, so the next apply may revert fixes or fail.
• No detection — without AWS Config, terraform plan checks, or GitOps reconciliation, drift is invisible until it causes a “works here but not there” failure or a security gap.

Primary source: AWS Config: Detecting configuration drift