A hash chain is not tamper evidence

It stops the person who cannot recompute it. Whoever can — which is whoever could edit the log in the first place — is unaffected, and the result verifies perfectly.

The obvious way to protect an audit log is to hash-chain it. Each entry carries the hash of the entry before, so altering anything downstream breaks every link after it:

<seq> <sha256(entry)> <sha256(previous_chain_hash + entry_hash)>

I built exactly this for a Vault cluster's audit trail. It is genuinely useful, and I want to be clear about what it buys before explaining why I did not stop there.

What chaining actually buys

Real things, all of them worth having:

• Corruption is visible. A truncated write, a bad disk, a shipper that dropped a line — the chain diverges at a specific sequence number and you know where.
• Careless tampering is visible. Someone deletes the lines recording what they did and puts the file back. Every link after the deletion is now wrong.
• The failure is located, not just detected. "Entry 4,812 does not hash to what the chain recorded" is a starting point. "Something is wrong somewhere" is not.

That is a corruption detector, and a decent one. It is not tamper evidence, and the gap between those is the whole point of this post.

The attacker it does nothing about

A hash chain catches whoever cannot recompute it.

Consider who can edit the audit log. They have write access to the volume holding it. The chain file is on that same volume, written by the same process, using a published algorithm over the entries themselves. There is no key. Nothing about recomputing it is privileged.

So the attack is: delete the entries recording what you did, recompute every hash from that point forward, write both files back.

prev=$GENESIS; seq=0
while IFS= read -r line; do
    seq=$((seq + 1))
    eh="$(printf '%s\n' "$line" | sha256sum | cut -d' ' -f1)"
    ch="$(printf '%s%s' "$prev" "$eh" | sha256sum | cut -d' ' -f1)"
    printf '%s %s %s\n' "$seq" "$eh" "$ch" >> chain.log
    prev="$ch"
done < audit.log

Twelve lines. The result passes every check that reads only the log and the chain, because it is consistent — the chain correctly describes the history it was given.

It is internally consistent and completely false.

This is the part I think gets skipped. "We hash-chain our audit log" is said as though it settles the question, and against the threat people usually have in mind — an insider covering their tracks — it settles nothing at all.

What does break it

A copy of the head hash, held somewhere the attacker is not.

Once the head for sequence N is written down elsewhere, any later rewrite of entries at or before N produces a different head. The rewritten chain is still self-consistent; it just no longer agrees with the record of what it used to say. The attacker cannot fix the disagreement without also reaching the anchors.

So the collector gained a sibling: a separate service whose only job is to copy the chain head somewhere else, periodically.

audit-anchor:
  build: ../audit-collector
  entrypoint: ["/usr/local/bin/anchor.sh"]
  volumes:
    - audit-logs:/collector:ro     # read-only: it cannot alter the trail
    - audit-anchors:/anchors       # and the collector has no mount here

Two properties, and both matter:

The audit volume is mounted read-only. The anchor service cannot alter the thing it is attesting to.

The collector has no mount for the anchors at all. Compromising the process that writes the log does not give you the record of what the log used to say.

Anchoring is periodic rather than per-entry, which is a real limitation stated plainly: entries written since the last anchor are covered by the chain but not yet by an anchor. That interval is the window in which a thorough attacker can still rewrite history undetected. Shorter is safer and noisier.

Testing a negative

Here is the part I would want a reviewer to look at, because it is easy to build all of this and never check that the anchors are doing anything.

The test builds a trail, anchors the real head, deletes an entry, recomputes the whole chain — and then asserts both halves:

# First half: it passes everything that reads only the log and the chain.
run_verify_no_anchors "$D"
assert_rc    "a rewritten chain passes without anchors" 0
assert_says  "and claims the chain is intact"           "chain intact"

# Second half: the anchor remembers a head that no longer exists.
run_verify "$D"
assert_rc   "the anchor catches the rewrite"   1
assert_says "and says the trail was truncated" "truncated"

The first assertion is the interesting one. It asserts that verification succeeds on a forged trail — that the chain alone is fooled.

That looks like a test of the bug rather than the fix, and that is exactly what it is for. It is the evidence that anchoring is load-bearing rather than decoration. If that assertion ever starts failing, it means chaining alone became sufficient somehow, and the right response is to re-read the reasoning rather than to "fix" the test.

I checked both directions by mutation, because an assertion that has never been made to fail is not yet known to test anything. Dropping the previous hash from the link turns eight assertions red; disabling the anchor comparison turns three red.

What this still is not

Both volumes live on the same Docker daemon.

A sufficiently privileged compromise of that host reaches the anchors too, so what exists is tamper evidence — you can tell that something was altered — and not tamper proofing. Only the first is claimed, in the code comments, the docs, and the README.

A production anchor belongs where the audited host cannot write at all: object-locked storage with a retention policy, a different cloud account, or an external timestamping service. The anchor format is three fields of text — timestamp, sequence, hash — specifically so that shipping it somewhere else is a change of destination and not a redesign.

The general shape

Every integrity mechanism has an implicit against whom, and it is usually unstated.

A hash chain answers "against someone who can modify the data but not recompute the digests." For a transmission error or a clumsy edit, that is the right threat model. For an insider with write access — the threat an audit log exists for — it is the wrong one, and the mechanism is approximately decorative.

So the question worth asking of any integrity control is not "is this cryptographically sound." It usually is. The question is: who is it that cannot do this? If the answer turns out to be "nobody who could attack us anyway", you have built a corruption detector and called it something else.