← Seth Bergman

Infrastructure · Terraform

Apply complete! · replacement never joined

Every test was green. The cluster could not heal.

A plan that succeeds is not a system that works

Every check proves something narrower than its color suggests. The useful habit is writing down what each one proves, and spending money only on the part nothing cheaper can reach.

I have spent the last few months building a production-shaped HashiCorp Vault platform in public: a three-node Raft cluster, KMS auto-unseal, Terraform for AWS and Azure, Ansible for configuration, and around forty CI jobs. These are the lessons from it I would most want another infrastructure engineer to take away.

Layers, cheapest first

The repository tests at four levels, each catching what the one below it cannot:

An emulator implements the API, not the service. Nothing boots, no health check runs, no scaling group replaces anything, and KMS returns plausible answers without doing cryptography.

The failure only a real apply could find

The first real AWS apply took about two hours in a sandbox account and cost under a dollar. Everything upstream was green. It found ten defects, five of them before Vault had started at all.

The one that mattered was the reason for the whole architecture. I terminated the leader. A new leader was elected in seconds, the load balancer dropped the dead target, and the autoscaling group launched a replacement inside 75 seconds. Then the replacement exited with error loading TLS cert and never became a Vault node.

Its certificate only ever arrived through an Ansible run, and was named after an instance id that did not exist until after launch. Nothing in the automated path could issue one.

The cluster survived losing a node. It could not replace one.

The fix was to let a replacement node sign its own certificate. The second apply, a week later, watched it auto-unseal and join Raft with nobody touching it. Azure's first apply was harder: nineteen fixes, and not one was reachable by the mocked tests or the emulator. Most shared a shape. A policy says who, a firewall says from where, and both have to allow it.

A restore that did nothing still looks healthy

The disaster-recovery drill seeds a canary secret, takes a Raft snapshot, destroys the node and its storage, and restores into a fresh one. Checking that the new cluster is unsealed and healthy proves nothing, because a restore that silently failed leaves exactly that.

So the drill reads the canary back using the original root token, and checks that the fresh node's own token stops working. That token flip is what distinguishes a restore from a cluster that came up empty.

The drill also deliberately leaves the unseal key alone. A snapshot of an auto-unsealed Vault is encrypted under that key, so the snapshot is only half a backup. In the cloud, the KMS key has to survive, and be backed up, independently of everything else.

Fail closed when the control plane is gone

Vault's database engine issues a fresh Postgres user per consumer, with an hour's lease. The detail that matters is one clause in the creation statement: VALID UNTIL '{{expiration}}'. It puts the deadline on the database role itself, so if Vault is unreachable when the lease expires, Postgres still ends the credential on schedule.

MySQL has no equivalent, so on that engine Vault's revocation is the only thing that ends a credential. The tests assert the difference rather than smoothing it over.

CI can break with no code change

A documentation-only pull request failed ShellCheck this week, on a script it never touched. A cleanup() function that runs only through trap had its warning suppressed as SC2317. ShellCheck 0.11 moved "function is never invoked" into a new code, SC2329, and the old directive stopped covering it. The main branch was already broken; it just had not run yet.

# shellcheck disable=SC2317,SC2329  # invoked through the trap below

Naming both codes keeps the line passing on the old version and the new one, so the fix does not depend on which ShellCheck the runner happens to install.

The general shape

Each of these is the same lesson from a different side. A green result answers a narrower question than it appears to, and the gap between the question asked and the one that matters is where outages live.

So the project keeps a written record of what every check settles and what it does not, including the gap it has not closed: no identity narrower than an administrator has driven a real apply yet. Saying that plainly is worth more than another green badge. Know what your checks cannot see, and decide on purpose which of those gaps you will pay to close.