Turning on audit logging can take Vault down

It is filed under security. It is an availability decision with a security benefit, and the difference matters at three in the morning.

Vault sends every request and response to all enabled audit devices, and guarantees the entry reaches at least one of them. If it cannot write to any enabled device, it stops answering.

Not degrades. Stops. A request Vault cannot audit is a request Vault will not serve.

That is the correct behaviour, and I want to be clear about why before complaining about it. A secrets manager that quietly stops recording who read what is worse than one that is down, because down is obvious and unaudited is not. If you cannot prove afterwards which credentials an attacker touched, you have to assume all of them.

So the guarantee is right. The consequence is what nobody mentions when they tell you to turn audit logging on.

One device is a single point of failure for the whole cluster

Enable exactly one file audit device — the obvious first step, the one every tutorial shows — and you have made your log partition filling up into a total Vault outage.

Not a lost log. Not a warning. Vault refuses requests until something can be written again.

You added the device for safety, and the naive move made the system less available than it was before.

It fails on the day the disk fills, which is not a day you chose.

bootstrap-audit.sh in my Vault platform therefore enables two by default. --no-second exists, and its help text says what it costs rather than presenting it as a preference.

Two devices, but not any two

Here is the part I got wrong first.

The obvious redundancy is a second file device. Two files, two paths, twice the safety. Except both live on the same partition, so the same disk fills for both, at the same moment, for the same reason. That is one failure domain wearing two hats, and it buys you nothing at the only moment you needed it.

The secondary has to fail differently:

The asymmetry that catches people

You would think, having established that, that any two devices on different domains would do. They will not, and the reason is worth knowing.

A socket device alone is more dangerous than a file device alone. When its endpoint goes away — collector restarts, network partitions, someone redeploys the log stack — Vault has nowhere to write and blocks. A remote dependency now gates every request to your secrets manager.

Paired with a file device, it cannot do that. The file keeps satisfying the at-least-one guarantee while the socket is unreachable, so the collector going down costs you a degraded audit trail rather than an outage.

So the shape that works is file primary, socket secondary — which is what HashiCorp recommend, and it is not arbitrary advice. It is the only arrangement where the remote device adds a failure domain without adding a dependency.

Vault gives you one free check

There is a nice piece of design here that makes the socket case survivable.

Vault writes a test entry when a device is enabled. So a destination it cannot reach fails the vault audit enable command, immediately, rather than at the first real request:

vault audit enable -path=socket socket \
    address=collector:9090 socket_type=tcp
# fails here if nothing is listening

That is the difference between a failed command at a keyboard and an outage in production. My bootstrap script leans on it and says so, rather than treating the enable as fire-and-forget.

The gap in re-enabling

One more sharp edge, found by asking what --force should do.

Re-enabling an existing device means disabling it first. If it is the only device, there is a window between the two where Vault is running with no audit device at all — and anything written in that window goes unrecorded, which is precisely the state the whole feature exists to prevent.

The script refuses:

Refusing to re-enable 'file/' with --force while it is the only device:
disabling it first would leave Vault unaudited

Refuse rather than narrate. A warning printed into a terminal nobody is reading is not a mitigation.

Testing it means breaking one on purpose

Everything above is reasoning. What makes it real is that the integration suite stops the collector container and then checks Vault is still serving — still accepting writes, still healthy, still recording to the surviving device.

That test only means something because the two devices are on different failure domains. Two files would have failed together, and the suite would have been asserting that nothing broke while nothing was capable of breaking independently.

Which is the same trap as everywhere else: a test that passes because the scenario it describes cannot occur is not evidence, it is decoration.

What to take away

If you are turning on audit logging:

• Enable two devices, not one. One makes a full disk into an outage.
• Put them on different failure domains. Two files on one partition is nominal redundancy.
• Make the primary a non-socket device. A socket alone turns a remote service into a hard dependency for every Vault request.
• Check that the enable actually enabled something. Vault's test entry gives you that for free; use it rather than assuming.

And when you write it down for whoever runs this after you, say what the single-device option costs. The failure is quiet in exactly the way audit logging exists to prevent, and the person who hits it will not have read the tutorial you did.