The alert that could not fire

Every failure this system produced looked like success from the outside. The graphs did not go red — they had one fewer line than they should have, and nobody counts lines.

Here is an alert rule. Read it and decide whether it does what it says.

- alert: VaultSnapshotStale
  expr: time() - vault_snapshot_last_success_timestamp_seconds > 7200

"Alert if the last successful snapshot was more than two hours ago." That is what it looks like. It is not what it does.

If snapshots stop entirely — the timer breaks, the script exits early, the pushgateway becomes unreachable — then nothing reports vault_snapshot_last_success_timestamp_seconds any more. The expression has no series to evaluate. It produces an empty result, and Prometheus treats an empty result as nothing to alert on.

The alert is silent in exactly the situation it was written for.

Not delayed. Not noisy. Silent. A rule that fires when backups are stale and goes quiet when backups stop is worse than no rule, because it comes with a feeling of coverage.

How I found out

I had three green systemd timers and zero backups, for weeks. The leadership check in the snapshot script read a field that does not exist in vault status -format=json, so every node concluded it was not the leader and skipped its turn. I have written that one up separately; the short version is that the failure was completely invisible.

What stuck with me afterwards was not the bug. It was the realisation that I had had a staleness alert the whole time, and it had never been in a position to fire.

Once I went looking, the pattern was everywhere in this repository. Every significant failure it has produced looked like success from the outside:

• Snapshots were never taken on any node, for weeks, with three green timers.
• A certificate reload failed and Vault kept serving the old certificate, while the reload command exited 0.
• A hostname check accepted every certificate it was given, because openssl reports that particular result in its output rather than its exit code.

None of those would have been caught by a dashboard. Nothing was wrong on screen. The graph simply had one fewer line than it should have, and a missing line does not draw attention to itself the way a spike does.

Absence is a different failure mode

Monitoring is mostly built around thresholds: something got too big, too slow, too old. Thresholds all share a hidden precondition — that the thing is still reporting.

That precondition is invisible in the rule. Nothing in time() - metric > 7200 tells you it depends on metric existing. You have to already know.

And the precondition fails in precisely the circumstances you care about. A metric stops being reported when the exporter dies, the job is misconfigured, the scrape target list is empty, the push never happens, or the thing being measured stopped happening at all. Those are not edge cases. Several of them are the outage.

The fix is boring, which is the point

Prometheus has absent(). It returns a value when a series does not exist, so it fires on the case the threshold cannot see:

- alert: VaultSnapshotMetricMissing
  expr: absent(vault_snapshot_last_success_timestamp_seconds)
  for: 2m
  annotations:
    description: >-
      No snapshot has ever been recorded, or the pushgateway is
      unreachable. VaultSnapshotStale cannot fire while this is true,
      because it has no series to evaluate.

Nothing clever. The interesting part is not the function; it is treating the pair as the unit. A freshness alert without its absence partner is not a smaller amount of coverage — it is a false report of coverage, which is worse than a gap you know about.

Proving the threshold really is silent

It is easy to write both alerts and assume the split works. I wanted the claim tested, because the whole point is that the failure is invisible.

promtool can feed a synthetic series to a rule file and assert which alerts fire. It also understands stale, which makes a series stop existing rather than merely go to zero — exactly the case in question:

- name: snapshots stopping entirely is caught by absent, not by staleness
  interval: 15s
  input_series:
  - series: vault_snapshot_last_success_timestamp_seconds{instance="vault-0"}
    values: 0+0x4 stale
  alert_rule_test:
  - eval_time: 5m
    alertname: VaultSnapshotStale
    exp_alerts: []          # <- the threshold alert does NOT fire
  - eval_time: 5m
    alertname: VaultSnapshotMetricMissing
    exp_alerts:
    - exp_labels:
        severity: critical

The load-bearing line is exp_alerts: []. The test asserts that VaultSnapshotStale stays quiet when snapshots stop. If someone later "fixes" the rules so the threshold alert covers this case, that assertion fails and the split gets re-examined rather than silently abandoned.

A second case covers the other half: a snapshot that is merely old, where the series still exists, fires VaultSnapshotStale and not the absence alert. Two rules, two behaviours, both pinned.

Making the rule survive the next contributor

Testing the pairs I wrote does nothing about the pair somebody adds next year. So there is a structural check over the rule file itself:

for name, r in alerts.items():
    expr = r["expr"]
    if "time()" in expr and not expr.strip().startswith("absent("):
        metrics = [...]                     # metric names in the expression
        if not any(m in guarded for m in metrics):
            failures.append(f"{name}: freshness check with no absent() guard")

Any rule doing freshness arithmetic on a metric that no absent() alert guards fails CI. The same pass requires every alert to carry a severity label, a summary, and a runbook annotation — because an alert nobody knows how to action is its own kind of absence.

This is the part I would keep if I kept only one thing. The insight is cheap and easy to agree with; it is worthless the first time somebody adds a rule at 2am without having read this file. An invariant enforced in CI outlives the person who understood it.

It keeps happening

Recently I migrated the cluster's TLS certificates from a bootstrap CA to Vault's own PKI. The final step drops the old CA from the trust bundle.

Prometheus and the blackbox exporter both mount that same bundle to verify the nodes they scrape. They read it once, at startup. After the migration they were validating against a CA that no longer signs anything, so the TLS probe failed and probe_ssl_earliest_cert_expiry stopped being reported.

Certificate expiry monitoring was gone. No error, no red panel — the series just stopped. And the thing that caught it was the absence alert written after the snapshot incident, firing in a case I had not thought of when I wrote it.

That is the argument, really. Thresholds catch the failures you anticipated. Absence catches a whole category at once, including the ones you have not imagined yet, because "this stopped reporting" is the shape almost every silent failure eventually takes.

What to do on Monday

Go through your alert rules and find every expression that does arithmetic on a metric — freshness, ratios, error rates. For each one, ask: if this metric stopped being reported entirely, would anything fire?

Where the answer is no, you have an alert that cannot fire in its own worst case. Pair it with absent(), then write a test that asserts the threshold rule stays quiet — otherwise you have only moved the assumption, not removed it.