September 2026
Checking it worked is what broke it
Seal migration has a window a few seconds
long where the cluster reports the new seal type but has not
committed to it. Restarting a node to check auto-unseal works —
the obvious verification — is what puts it back into migration
mode, where it will not auto-unseal.
The error that meant it worked
A 400 that meant the operation had
succeeded, and three more signals I read backwards — an absence
that meant interference, a health check that stayed green on a
node serving nothing, and two of three healthy with no quorum.
None of them were bugs in Vault or AWS.
Nobody had watched these tests fail
A mutation table is a list of claims about your
test suite. Mine had thirteen rows and a heading admitting none had
been run — six assertions turned out to have no input that could make
them red, including one that was 24 <= 24 for every
input in existence.
August 2026
Turning on audit logging can take Vault down
Vault refuses any request it cannot audit, which
is correct — and makes a single audit device into a cluster-wide
outage the day the disk fills. Two of the wrong kind do not help
either.
August 2026
I had to delete a passing test to fix the bug
Three assertions that were holding defects in
place — one guaranteeing a cluster could never start, one dropping
every client packet, one I wrote myself against a mock that had
flattened the values it compared.
August 2026
A hash chain is not tamper evidence
Chaining an audit log stops whoever cannot
recompute the digests. That is not the person you are worried about:
anyone who can edit the log can rewrite the chain over it, and the
result verifies perfectly. What actually breaks that, and the test
that proves it was needed.
August 2026
The alert that could not fire
A threshold alert on a metric that stops being
reported never fires — it has no series to evaluate, so it is silent
in exactly the situation it was written for. Pairing every freshness
rule with absent(), and proving in a test that the
threshold really does stay quiet.
August 2026
Three green timers and zero backups
A test double can only encode what you already
believe. Three incidents from a HashiCorp Vault platform — a shim that
agreed with the bug, one that disagreed with the environment, and a
mock that never modelled the consumer at all.