Cloud & DevOps

The Backup System That Took Down the Database It Was Backing Up

A production Postgres box in a crash-restart loop, reported as "the database is maxed out." It was sitting at 9 of 100 connections. The real cause was one character of difference in a filesystem path.

August 22, 2026 · 6 min read · Cartolinks Engineering

Photo by panumas nikhomkhai on Pexels

The ticket said the database was maxed out. Apps were down, Postgres was in a crash-restart loop, and the obvious read was load — too many connections, not enough CPU, time to resize the box.

The box was at 9 of 100 connections. CPU was fine. Memory was fine. The actual error, once we got to the logs, was this:

FATAL:  could not write lock file "postmaster.pid": No space left on device

The OS disk was 100% full. 75 GB of 75 GB, zero bytes free. Postgres could not write its own PID file, so it FATALed on every single start attempt. Nothing was wrong with the database. Something was wrong with the disk.

One character

The box backed up nightly to a 1 TB network storage volume, mounted over CIFS at /mnt/storagebox. The backup container bind-mounted a subdirectory of that mount and wrote gzipped dumps into it. Simple, and it had worked for months.

Except the dumps were landing in /mnt/storagebox.local — a plain local directory on the OS disk — not /mnt/storagebox, the actual network mount. Ten days of nightly dumps at roughly 6 GB each had quietly piled up to 60 GB on a 75 GB disk.

Meanwhile the 1 TB destination those backups were supposed to be going to sat 99.99% empty.

Retention was working perfectly

This is the part worth sitting with. The nightly job had a retention policy, and the retention policy ran successfully every night. It pruned dumps older than the cutoff — in the directory it was configured to write to.

It had no idea that 60 GB of stranded dumps existed somewhere else. Retention only ever prunes the *current* backup directory. The moment your backups start landing somewhere unexpected, your cleanup stops covering them, and the two failures compound: you're filling a disk you didn't mean to fill, and nothing is cleaning it up.

Every individual component reported success. The backup job exited zero. Retention exited zero. Disk usage climbed about 6 GB a night for ten nights with no alert configured to notice.

The fix, and how fast it was

1. Moved the stranded dumps to the real network mount with rsync --remove-source-files, which frees space incrementally as it goes rather than at the end. Disk went from 100% to 21%.
2. Postgres recovered on its own the moment space appeared. No restore, no intervention, no data loss — the full backup history from all ten days was intact, just on the wrong disk.
3. Fixed the backup job's path handling for the CIFS mount and recreated the container. A manual run then exited clean: all databases dumped, tiers hardlinked, retention pruned.
4. Deleted the empty local directory tree so nothing could bind-mount into it again.

Total data loss: none. Total time the fix took once we understood the problem: minutes. Total time the problem had been building: ten days.

We had already solved this. Somewhere else.

The genuinely uncomfortable part came later. The MySQL tier in the same estate has a hard guard against exactly this failure. Its backup script refuses to run if the backup directory shares a device with the data directory:

if [[ "$(stat -c %d "$BACKUP_DIR")" == "$(stat -c %d "$DATA_DIR")" ]]; then
  echo "✗ BACKUP_DIR is on the SAME volume as DATA_DIR."
  exit 1
fi

Its README says it in plain language: *backups live on their own dedicated volume — never on the data volume, because a runaway dump can fill it and take the database down.* That is a precise description of the incident, written before the incident, by the same team.

The Postgres box was built on a different path, at a different time, and inherited none of those guards. The lesson is not "add a disk check." It is that hard-won operational knowledge doesn't propagate on its own — it lives in whichever script the person who learned it happened to be editing that week. If a guard matters, it belongs in a shared bootstrap that every tier runs, not in one tier's backup script.

What we changed

• Backups get their own volume, everywhere. Not their own directory — their own block device. Then a runaway dump fills a volume nobody depends on.
• Boot ordering is a real failure mode. If the container starts before the network mount is up, Docker will happily create the mount point as a local directory and you reproduce this exact incident. A systemd drop-in makes Docker wait for the mount — using Wants, not Requires, so a storage outage delays backups rather than taking the database down.
• Alert on disk before it matters. A single "filesystem above 90% for 10 minutes" alert would have caught this on roughly day six.
• Verify the destination, not the exit code. The backup job's own report of success told us nothing. df -h against the backup path is now part of the runbook.
• A backup you have never restored is not a backup. Restore drills are the only check that actually exercises the whole chain.

The offsite backup program for this estate is dated the day after this incident. That is not a coincidence, and it is the most honest thing in the write-up: a well-run system is usually a system that has already failed in an instructive way.