How to migrate a stateful Docker app off a server with a dying disk

Short answer: When the source server’s disk fails on writes but still reads fine, treat it as read-only hardware. Stream pg_dump -Fc over SSH straight to the target machine, pull Docker images from your registry instead of rebuilding end-of-life Dockerfiles, and anchor every rsync exclude with a leading slash. That moved a 14-service app with a 12 GB PostgreSQL database onto a new node without a single write to the failing disk.

The source was an old OpenVZ box with a degraded RAID and a disk that had quietly stopped accepting writes. Reads were healthy. On it lived a Dockerized PHP application: 14 services (nginx, php-fpm, eight queue workers, PostgreSQL, Redis), 12 GB of database, 5 GB of media. The job was to stand up a production-like copy on a new Proxmox node for testing, with the real traffic cutover scheduled later.

TL;DR

• Fix: run pg_dump -Fc on the source but pipe it through ssh -T to the target’s disk — the source only reads, the target does all the writing.
• Fix: don’t rebuild images based on EOL distributions; pull the exact production images from the registry, reusing /root/.docker/config.json from the source host.
• Trap: rsync --exclude=log/ without a leading slash matches log at every depth. It silently removed vendor/yiisoft/yii2/log/ from the copy and the app answered HTTP 500.
• Trap: a dump to a dying disk hangs without an error. Watch the output file’s size delta, not the process list.

Why pg_dump hangs on a dying disk instead of failing

A dump writing to a dead disk doesn’t error out; it stops making progress while the process stays alive. My first attempt was the obvious one: pg_dump -Fc into a local backup/ directory on the source. The file grew to 1,601,044,831 bytes and froze. Two size checks six seconds apart returned the same number, with the process still running. The disk had died on the write path specifically, and a hung dump was exactly how that looked from userspace.

The liveness check lied to me too. The watcher used pgrep -f pg_dump, and its own command line contained the string pg_dump, so it matched itself and kept reporting the dump as alive. ps -C pg_dump (exact process name match) or a size delta on the output file are the checks that don’t fool themselves.

Streaming the dump: keep every write off the source

The fix is to leave the source with the read half of the work and move the write half to another machine. pg_dump runs on the source, but its output goes over SSH to the target’s healthy disk:

1
2
3

ssh -T old-host \
  'docker exec app-postgres pg_dump -Fc -U app appdb' \
  > /srv/dumps/appdb.dump

Dumping from a live database is fine here. The PostgreSQL 11 documentation states it directly: “pg_dump does not block other users accessing the database (readers or writers)” (pg_dump reference).

Two details matter for the transport. ssh -T disables pseudo-terminal allocation, and docker exec must run without -t: a tty mangles the binary custom-format stream. With both in place the 1.6 GB dump completed with exit code 0, and on the target pg_restore --no-owner -j 4 -d appdb unpacked it into 9.6 GB across 115 tables, with the rum extension intact.

Pull EOL images, don’t rebuild them

If production images already exist in a registry, pull them; rebuilding their Dockerfiles years later is a dead end. My first docker compose pull on the new host failed with Error response from daemon: error from registry: access forbidden — a private GitLab registry the new machine wasn’t authorized against. Instead of fixing auth, I detoured into rebuilding the images from the same Dockerfiles production once used. That went like this:

Buster I could route to archive.debian.org. Then the stretch-based postgres:11 build died on the PGDG repository, which no longer serves packages for an EOL major at all. I stopped there and admitted the detour was me solving the wrong problem.

The actual answer was sitting on the dying host the whole time: it pulls those images itself, which means working registry credentials live in its /root/.docker/config.json (that’s where docker login stores them). Copying that one file to the new host let docker compose pull fetch all 5 images — the exact bytes production runs, no build step.

The rsync exclude that deleted framework files

--exclude=log/ without a leading slash matches every directory named log at any depth, not just the top-level one you meant. I added it to skip 2.2 GB of application logs during the code-and-media rsync. The copy finished clean, and the app came up with HTTP 500:

include(/app/vendor/yiisoft/yii2/log/Logger.php):
failed to open stream: No such file or directory

Nothing in that message points at rsync. A missing framework file reads like a corrupted vendor directory, and I chased that theory first. The real cause: the unanchored pattern had stripped vendor/yiisoft/yii2/log/ out of the transfer. The rsync man page is explicit about the rule: “if the pattern starts with a / then it is anchored to a particular spot in the hierarchy of files, otherwise it is matched against the end of the pathname.”

The fix is one character: --exclude=/log/ anchors the pattern to the transfer root. A second rsync run then sent only the log directories it had wrongly skipped inside the code, a few seconds of incremental transfer.

Testing against live integrations without side effects

Queue workers and cron are what produce outbound effects; a web tier on top of a copied database is safe to start. This app talks to a real payment provider, CRM, SMTP, and ClickHouse, and the test copy had to come up without firing any of that. The compose setup was already split into docker-compose.yml (web) and docker-compose.queue.yml (workers and scheduler), so the whole safety switch was one environment variable:

1

COMPOSE_FILE=docker/docker-compose.yml docker compose up -d

Web and API up, all eight queue workers and the scheduler down, database an isolated copy. Smoke test: API returned 200, the site returned its usual 302, the admin backend rendered the login page.

One integration still broke: ClickHouse refused connections from the new node’s IP (“Connection refused”), and backend pages that touch it answered 503. External allowlists (database, mail, analytics) are part of the migration checklist, not something to discover during cutover.

Step by step

1. Establish the disk’s failure mode first: “reads work, writes hang” changes every decision after it.
2. Copy /root/.docker/config.json from the source to the target; docker compose pull the production images.
3. Stream the database: ssh -T source 'docker exec db pg_dump -Fc -U app appdb' > dump on the target.
4. Restore with pg_restore --no-owner -j 4 -d appdb.
5. Rsync code and media with anchored excludes only (--exclude=/log/, not --exclude=log/).
6. Bring up the web tier alone via COMPOSE_FILE, leaving queues and cron down.
7. Smoke-test, and collect every external service that needs the new IP allowlisted.
8. Cut over later in a quiet window: fresh dump, DNS, allowlists.

Bottom line

Dying hardware changes the rules: the source is read-only whether you accept it or not, so structure the whole migration around that. Stream the dump instead of writing it locally, pull images instead of rebuilding EOL bases, and anchor rsync excludes before they cut into vendor/. The copy went from “disk is dying” to a responding production clone in a few hours, and most of the lost time was me fighting battles (EOL apt archives, a phantom vendor corruption) that the constraints had already decided.