Three green timers and zero backups

A test double can only encode what you already believe. Where the belief is wrong, the double agrees with you — and agreement is indistinguishable from a passing test.

The integration suite had been running against a real three-node HashiCorp Vault cluster for about ninety seconds when it printed that line. On the active node.

That was the first of three. All three are the same mistake wearing different clothes, and it took me until the third to see what they had in common.

OneThe double agreed with the bug

Scheduled Raft snapshots run hourly from a systemd timer on all three nodes. Only the leader should take one — two standbys writing redundant snapshots every hour is triple the storage for no additional recovery capability. So the script checked whether it was the leader:

HA_MODE="$(jq -r '.ha_mode // "unknown"' <<< "$STATUS_JSON")"

if [[ "$HA_MODE" != "active" ]]; then
    log "Node is ${HA_MODE}, not active; the leader takes the snapshot."
    exit 0
fi

Run vault status and you see HA Mode active. Run vault status -format=json and ha_mode is not there. Those HA fields are rendered by the CLI for its text output; they are not keys in the JSON. Reported upstream since 2020 (hashicorp/vault#9185).

So .ha_mode was null, // "unknown" substituted the string, "unknown" is not "active", and the script exited 0 having done nothing. On every node. Every hour.

Three green timers. Zero backups.

There was a test suite for that script. It ran the real script against a fake vault binary on PATH, and it asserted real things: a standby takes no snapshot, an empty snapshot is never uploaded, a failed upload is a failure. It was green. It had always been green.

"status -format=json")
    printf '{"sealed":%s,"ha_mode":"%s","initialized":true}\n' \
        "${FAKE_SEALED:-false}" "${FAKE_HA_MODE:-active}"
    exit 0
    ;;

Both sides agreed · Neither was right

I wrote the script believing vault status -format=json contained ha_mode. Then I wrote the shim from the same belief. The suite then confirmed, correctly and repeatedly, that the script read a field which existed only because I had put it there.

Not "the mock drifted out of date." The mock was never right. It was a faithful reproduction of my misconception.

What else was green at the same time

Two other things, and both were honest.

The systemd timers were green because the script exited 0. The unit ran and succeeded. It succeeded at doing nothing.

The disaster-recovery drill was green because it genuinely works: it takes a snapshot, destroys the node and its storage, restores into a replacement, and verifies a secret written beforehand reads back. It passes on every pull request. But it calls vault operator raft snapshot save directly. It never goes through the scheduled path, so it proved snapshots restore while proving nothing about whether any were being taken.

Neither was lying. The bug lived in the gap between two true statements.

TwoThe double disagreed with reality

The next day, a different suite. This one drives a script that migrates a cluster onto Vault-issued TLS certificates, and it has to decide whether a node is already migrated. The driver reads the CA's subject with the real openssl; the shim answers the "what certificate is this node serving" question with a canned issuer:

pki) printf 'ISSUER:CN=vault pki CA\n' ;;

37 assertions, all green, on my machine. On CI, five failed:

Refusing to prune: vault-0 vault-1 vault-2 still serve a non-PKI certificate

Every node read as un-migrated. The driver was right to refuse — on the evidence it had.

OpenSSL renders a distinguished name differently between builds. Mine printed CN=vault pki CA. The runner's OpenSSL 3.0 printed CN = vault pki CA, with spaces. The substring comparison failed, and nothing was migrated as far as the driver could tell.

This one is the first one inverted. There, the double agreed with the code and hid a real bug. Here, the double disagreed with the environment and invented one that did not exist. Same root: the double asserted something about the outside world that had been checked on exactly one machine.

The fix was not to normalise whitespace. It was to stop the shim having an opinion at all — it now receives the subject, read from the fixture by the same binary the driver will use, so the two cannot disagree about formatting regardless of what is installed.

ThreeThe double never modelled the consumer

The third has no shim in it, which is why it took me longest to file under the same heading.

Both cloud profiles are covered by terraform test against mocked providers. That catches more than it sounds like — a Key Vault name over Azure's 24-character limit, an IAM policy granting delete on the snapshot bucket. Every test passed.

The Azure profile's cloud-init rendered this:

auto_join = "provider=azure tag_name=VaultCluster
             tag_value=${CLUSTER_NAME}
             subscription_id=${subscription_id}
             resource_group=${resource_group}"

go-discover's Azure provider accepts tag_name + tag_value, or resource_group + vm_scale_set. It rejects any mix of the two with "unclear configuration."

That cluster would never have formed. And the failure is quiet in the particular way that matters: Vault starts, every node reports healthy, each one sits alone believing itself fine. The only evidence is a line in one node's log.

No test caught it because no test could. A mocked provider validates the shape of a configuration, not whether the program that eventually reads it will accept the contents. terraform test never runs go-discover. Nothing in the suite knew that string had a grammar.

It was found by reading go-discover's source, which is not a testing strategy.

There is a coda. The Terraform-to-Ansible handoff script had a guard against writing a broken config file. The guard called die() on a missing output — but from inside a heredoc, which exits the command substitution's subshell while the enclosing cat succeeds. So it wrote the broken file it existed to prevent, and exited 0. The thing meant to catch the failure had the failure.

The shape

Three incidents, one sentence:

A test double can only encode what you already believe.

It follows that a double is worthless precisely where your belief is wrong, which is exactly where you needed the test. And the three failures above are the three ways that plays out:

None of this is an argument against test doubles. Shims are the right trade for most of what this repository does: they are fast, need no credentials, and reach failure modes a live cluster will not reproduce on demand — a corrupt snapshot, a failed upload, a node sealed at exactly the wrong moment. Deleting them would cost more than it saved.

It is an argument that a green suite of doubles is evidence about internal consistency, and you have to go and get the other kind separately.

What actually changed

Doubles mirror the real response shape, and are not allowed opinions of their own. The vault shim no longer emits ha_mode, because the real command does not. The openssl shim is handed the DN rather than knowing one.

Every claim that rested on a document got a real instance. An integration suite stands up the real three-node cluster and runs the operational scripts against it. It found the first bug on its first run. That is not a coincidence; it is what the suite is for.

Assertions are checked by mutation, not counted. Every suite here has to be made to fail on purpose. When I fixed the leadership check, I confirmed that restoring the // form turned it red and that letting a standby proceed turned it red. Later, an assertion in a different suite — "an unhealthy node aborts the run" — turned out to pass with the health check deleted, because an earlier check aborted first. It had never tested anything. Mutation is how that surfaced.

And I ask one question of any green suite: what does this assume that nothing inside it can check?

For snapshots the answer was a field name. It cost every backup I thought I had, for as long as nobody looked in the bucket.