BitPage

Mass segfaults on a ZFS host: not bad RAM, an unsigned underflow in zfs_fillpage()

BitPage — Sun, 06 Sep 2026 04:15:00 +0000

Short answer: unrelated processes segfaulting all at once on a ZFS host, plus systemd[1]: Freezing execution in the journal, is kernel arithmetic rather than failing memory. In OpenZFS before 2.4.4 and 2.3.9, zfs_fillpage() computed the read length as io_len = i_size - io_off on unsigned types. Truncate a file in place at the moment an mmap page is faulted in, i_size drops below io_off, the difference underflows to near 2⁶⁴, and dmu_read() zero-fills physical memory far past the page it was asked for. The machine survives that. What finishes it off is systemd, whose CrashAction defaults to freeze.

From outside it looked like a powered-off box: no SSH, no HTTP, no hypervisor UI. A hard reset from the provider’s panel brought it back, which left the question nobody enjoys: what that was, and whether it happens again tomorrow.

An hour of digging produced two answers instead of one. The ZFS bug is the trigger. The price of the failure was set by a systemd default.

TL;DR

Cause: unsigned underflow in zfs_fillpage() when an mmap read races an in-place truncate. The kernel zero-fills physical pages belonging to whatever else was there, so the victims are random processes.
Fixed in: OpenZFS 2.4.4 and 2.3.9, both published 21 August 2026. On Proxmox, zfsutils-linux 2.4.4-pve1.
Not fixed in: 2.2.11, published the same minute as the other two. Verified by reading the code in the tag, not the release notes.
Why the outage runs for hours: CrashAction=freeze is the systemd default. A crashed PID 1 hangs forever, nothing restarts services, systemctl blocks, sshd dies later with nobody to bring it back.
One-minute test against bad RAM: ECC counters in /sys/devices/system/edac/mc/mc0/ at zero and no Machine Check in the journal means the DIMMs are not your problem.
Tell in the dump: the length register of memset is not garbage, it is a tidy negative number, exactly minus another register. That is a size computation gone wrong, not failing hardware.
One-line mitigation: CrashAction=reboot in a drop-in turns a repeat from an overnight outage into a 10-second one.

Why the machine looks powered off when it is not

It looks identical to a powered-off machine from outside, which is why the first hypothesis is always wrong. No port answers, the provider’s panel reports power as normal, and support has nothing to add. The obvious theory is that the box went down, so you start looking for the cause of a shutdown: an ACPI event, a watchdog, a power dip in the rack.

The journal kills that theory. journalctl --list-boots shows the previous boot, and its entries run right up to a point a couple of minutes before the new boot began, which is the board’s POST. The last line before the reset was an nginx worker dying:

nginx[…]: [alert] …: worker process … exited on signal 11

The machine never powered off. It ran the whole time and kept writing to the journal that it was in trouble. Nobody outside could hear it.

That yields a second, nastier fact. The moment things actually broke sits much earlier than the “crash time” you get from last. last -x will show crash without saying when it started. The real start is the first segfault in the avalanche, and it can be a whole night before the reset.

Why it is not bad RAM, even though it looks exactly like bad RAM

Simultaneous segfaults across unrelated processes is the classic signature of a failing DIMM, and the reflex it triggers is the wrong one. Before you schedule downtime for memtest86+, spend a minute on two counters:

1
2
3


cat /sys/devices/system/edac/mc/mc0/ce_count # corrected errors
cat /sys/devices/system/edac/mc/mc0/ue_count # uncorrected
journalctl -b -1 -k | grep -i "machine check"

Zeros in all three places on an ECC system mean memory as a subsystem is healthy. A single flipped bit would have been caught and logged. While you are there, confirm ECC is actually active rather than merely printed on the invoice:

1

dmidecode -t memory | grep -E "Error Correction Type|Total Width|Data Width"

Total Width needs to be 8 bits wider than Data Width, 72 against 64, because those are the physical bits doing the correction. Equal widths mean you have no ECC, whatever the spec sheet claims.

That minute saves hours of downtime spent diagnosing the wrong subsystem. ECC protects against flipped bits, not against code that walks the ordinary execution path and writes zeros into the wrong place.

Four dead ends, all through the same function

Every hypothesis that came from searching the symptom was wrong, and they cost more time than the answer did. Writing them out because each one closes a direction: if your stack looks like this, you can skip these branches.

Dead end 1: a Direct IO race. The first sensible search on the call stack landed on a real race in OpenZFS, where a concurrent O_DIRECT write can set db->db_data = NULL and dmu_read_impl() then faults on the dereference. The function names matched mine exactly. Checking the tunable made it look even better:

# cat /sys/module/zfs/parameters/zfs_dio_enabled
1

I said out loud that this was the leading theory. The registers disagreed: what I had was not a NULL dereference but a memset with a sensible destination and an absurd length. Different mechanism. I should have checked that in the first minute, since the registers were on screen the whole time.

Dead end 2: SLUB corruption on encrypted ZFS. Issue #18427 describes my symptom nearly word for word — mass userspace segfaults, reproducible across several releases. One command ended it:

# zfs get -r encryption rpool
NAME PROPERTY VALUE SOURCE
rpool encryption off default

Dead end 3: an already-fixed regression in dmu_read_impl. Issue #17886 is an oops in the same function, also with memory clobbered by a bad length, closed by PR #17915. It read like “known bug, fix exists, go upgrade.” Then I read the PR: it restores lost parentheses in the BRT_RANGESIZE_TO_NBLOCKS() macro, and the author states the blast radius plainly — “this could cause small memory corruptions for vdevs bigger than 64TB+”. A web-workload pool of a few terabytes is an order of magnitude short of that threshold. The stack in that bug also runs through brt_load() during pool import rather than through mmap under load. Same crash site, different bug.

Dead end 4: “the server was off for hours.” Covered above. It died against a continuous journal from the previous boot.

The unpleasant summary: all four trails ran through dmu_read_impl. It is a popular function, “memory corruption” is far too general a symptom, and searching by symptom in that situation returns an endless feed of other people’s misfortunes. The break came from arithmetic, not from searching.

The giveaway: the memset length is exactly minus another register

The whole answer sits in the oops dump, and it only needs counting. Trimmed to what matters:

BUG: unable to handle page fault for address: ffff8983e8400000
#PF: supervisor write access in kernel mode
#PF: error_code(0x0003) - permissions violation
Comm: php-fpm
RIP: 0010:memset+0xb/0x20
RAX: 0000000000020000 RSI: 0000000000000000 RDI: ffff8983e8400000
RDX: ffffffffff7a4000 R15: 000000000085c000
Call Trace:
? dmu_read_impl+0x21e/0x230 [zfs]
dmu_read [zfs]
zfs_fillpage [zfs]
zfs_getpage [zfs]
zpl_read_folio [zfs]
filemap_read_folio
filemap_fault
__do_fault
handle_mm_fault

Read the memset arguments. Under the x86-64 SysV ABI those are RDI for destination, RSI for the fill byte, RDX for the count.

RSI = 0, so it is zero-filling. RDX = 0xffffffffff7a4000 looks like garbage at a glance. Now take R15 = 0x85c000 = 8,765,440. This is the part worth slowing down for: 0xffffffffff7a4000 is exactly −8,765,440 in two’s complement. Not a random value, not a clobbered register. A tidy negative number.

A negative length does not appear from nowhere. It means something subtracted a larger number from a smaller one in unsigned arithmetic. The culprit is not whoever corrupted memory, it is whoever computed the size.

RAX = 0x20000 fits the picture too: 128 KiB, the default ZFS recordsize. An ordinary record read whose length went backwards.

With that hypothesis the search stopped being a search by symptom and became a search by function name plus exact version. The right issue was the first result.

What is broken in ZFS: an invariant that does not survive the release build

Two lines in zfs_fillpage() do the damage, and the guard in front of them does not exist in release builds. Here is the code from the 2.4.3 tag:

1
2
3
4
5
6
7
8


loff_t i_size = i_size_read(ip);
u_offset_t io_off = page_offset(pp);
size_t io_len = PAGE_SIZE;

ASSERT3U(io_off, <, i_size);

if (io_off + io_len > i_size)
 io_len = i_size - io_off;

The invariant is stated in plain sight: the page offset must be smaller than the file size. One line down, a subtraction is built on it. All honest, right up until you open the definition of ASSERT3U for a non-debug build in include/os/linux/spl/sys/debug.h:

1
2
3
4
5
6
7


/*
 * Debugging disabled (--disable-debug)
 */
#ifdef NDEBUG

#define ASSERT3U(x, y, z) \
 ((void) sizeof ((uintptr_t)(x)), (void) sizeof ((uintptr_t)(z)))

Two sizeof expressions that check nothing. The header says so directly: assertions are “compiled out when NDEBUG is defined, this is the default behavior for the SPL.” Distributions, Proxmox included, build ZFS that way. So the invariant is documented, and unenforced exactly where enforcement would have mattered.

Truncating the file in place while a page is being faulted in breaks that invariant. i_size drops below io_off, the subtraction underflows, and io_len becomes a number close to 2⁶⁴. dmu_read() takes it at face value and fills.

The fix adds the runtime check the assertion pretended to be. From the 2.4.4 tag:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


 /*
 * The page may be faulted in after the file has been truncated.
 * There is no data to read; just zero-fill the page.
 */
 if (io_off >= i_size) {
 void *zva = kmap(pp);
 memset(zva, 0, PAGE_SIZE);
 kunmap(pp);
 ClearPageError(pp);
 SetPageUptodate(pp);
 return (0);
 }

The commit is 223b8bc446, “linux: handle mmap read beyond file size”, merged 30 June 2026 by Brian Behlendorf. Its trailers are worth reading: Reported-by: Iliya Polihronov (@vnsavage) (Automattic), Reviewed-by: Alexander Motin. WordPress.com hit this independently, which fits — it is a PHP-shaped bug. readfile() on a cache file that another request is rewriting in place is the exact race.

The crashing process in my dump is php-fpm, running under an unprivileged LXC. Its uid maps to 82 inside the container, which is www-data in Alpine, so it was PHP from an Alpine image in Docker inside LXC, on a Debian host. Four layers down from the kernel, and none of them at fault.

Which versions are actually fixed, and which release from the same day is not

Three OpenZFS releases went out on 21 August 2026, and the fix is in two of them. Checking release notes is not enough here, so I read zfs_vnops_os.c in each tag:

Branch	Release	Published	`zfs_fillpage()` guarded
2.4	zfs-2.4.4	2026-08-21	yes
2.3	zfs-2.3.9	2026-08-21	yes
2.2	zfs-2.2.11	2026-08-21	no

The three tags were published within about a minute of each other. Release notes for 2.4.4 and 2.3.9 both carry the line linux: handle mmap read beyond file size #18715; the notes for 2.2.11 do not. In the 2.2.11 tree, ASSERT3U(io_off, <, i_size); and the bare subtraction are still there. The same holds in the 2.1 branch, so this is an old bug rather than a fresh regression, and downgrading to an older kernel does not help.

Check your own build rather than trusting a version number:

1
2
3
4


zfs version
# then, for the tag you are on:
curl -sf "https://raw.githubusercontent.com/openzfs/zfs/zfs-$(zfs version | head -1 | cut -d- -f2 | cut -d_ -f1)/module/os/linux/zfs/zfs_vnops_os.c" \
 | grep -c 'io_off >= i_size'

Nonzero means the guard is in. Use curl -sf: without -f, a nonexistent tag returns a 404 page that greps to “0 matches”, which is indistinguishable from “the fix is missing.” I nearly recorded that as a result once.

On Proxmox the fixed package is already in the ordinary repository. In pve-no-subscription for trixie, zfsutils-linux is at 2.4.4-pve1, so this is a plain apt upgrade plus a reboot rather than anything exotic.

Upstream the story is not closed. Issue #18787, opened 12 July 2026 by the Incus maintainer, who was losing hosts full of LAMP containers, is still open today even though the fix landed two weeks before it was filed. Track the PR number, not the ticket state. A regression test followed on 18 July.

Why the outage lasts hours: CrashAction defaults to freeze

The bug corrupts memory in milliseconds; the outage lasts all night because of a systemd default. Milliseconds after the corruption started, PID 1 caught its own crash:

systemd[1]: Caught <ABRT>, from our own process.
systemd[1]: Caught <ABRT>, dumped core as pid …
systemd[1]: Freezing execution.

CrashAction= arrived in systemd 256, alongside the systemd.crash_action= kernel command line option. The manual is unambiguous about the default:

Takes one of “freeze”, “reboot” or “poweroff”. Defaults to “freeze”. If set to “freeze”, the system will hang indefinitely when the system manager (PID 1) crashes. If set to “reboot”, the system manager (PID 1) will reboot the machine automatically when it crashes, after a 10s delay.

Freezing makes sense on a desktop, or anywhere an operator is standing next to the console and wants the corpse preserved. On rented hardware in someone else’s data centre it means “hang forever and wait for a human to notice.”

With PID 1 frozen the system loses the ability to repair itself. Nothing restarts crashed services, systemctl blocks, a clean shutdown is impossible. sshd went next, and after its last line there is not one further mention of it in the journal: no login attempts, no refusals. Port 22 simply had nobody listening. nginx meanwhile kept crashing at the same instruction pointer, tens of thousands of times, roughly twice a second, all night. The master was alive and forking workers; each worker inherited poisoned memory and died at once. A restart would have fixed it, and there was no one left to issue one.

The mitigation is one drop-in file:

1
2
3


# /etc/systemd/system.conf.d/10-crash-reboot.conf
[Manager]
CrashAction=reboot

1
2


systemd-analyze cat-config systemd/system.conf # inspect before applying
systemctl daemon-reexec # apply to the running PID 1

A repeat now costs ten seconds instead of a night. That is the whole trade: I cannot remove the bug from a release that does not contain the fix, so I bought recovery time instead of prevention.

Side gotchas worth stealing

Two smaller findings came out of the same morning, and both are worth checking on any Docker host.

A restart: on-failure policy survives a crash but may not survive a clean reboot. The containers came back after the hard reset because their processes were killed by SIGKILL, so the exit code was nonzero and the policy fired. On a graceful reboot dockerd sends SIGTERM, and anything that exits 0 (nginx does) is not a failure, so on-failure leaves it down. The configuration survives the disaster and can fail the maintenance window. Switching everything to unless-stopped with docker update fixes it live, no recreation and no downtime.

A container with no restart: line at all defaults to no. One monitoring container was in that state, so after every reboot it stayed down, and the proxy that depended on it spun through dozens of restarts inside half an hour with host not found in upstream. depends_on does not help here: it orders docker compose up and has no effect when the daemon starts containers on boot.

What to do right now

Check whether your ZFS build carries the guard: zfs version, then look for io_off >= i_size in zfs_fillpage() for that tag. Anything on 2.2.x or 2.1.x is exposed, including 2.2.11 from 21 August 2026.
Upgrade to 2.4.4 or 2.3.9. On Proxmox trixie that is zfsutils-linux 2.4.4-pve1 from pve-no-subscription, then a reboot into the new module.
If you cannot upgrade yet, cut the recovery time: drop in CrashAction=reboot and run systemctl daemon-reexec. Ten seconds against an open-ended hang.
Add an availability check from outside the host. Mine was found by a person arriving at work, which is a bigger hole than the kernel bug.
Audit Docker restart policies: docker inspect -f '{{.Name}} {{.HostConfig.RestartPolicy.Name}}' $(docker ps -aq). Anything showing no or on-failure is a machine that may not come back from a planned reboot.

Bottom line

Three layers of blame, and you can only fix two of them. The ZFS underflow is upstream’s to fix, and as of today it is fixed in 2.4.4 and 2.3.9 while 2.2.11 from the same day still carries it. The systemd default turned a memory-corruption event into an overnight outage, and that costs one line of configuration to change. The missing availability check is what stretched the outage until morning, and that one is nobody’s fault but mine.

The transferable part is the diagnostic move. When the kernel faults inside memset or memcpy, look at the length register before you look at anything else, and check whether it is a tidy negative number. If it is, stop investigating the hardware: someone computed a size wrong, and the fastest path to the answer is the function names in the stack plus your exact version, not the symptom.

Primary sources: commit 223b8bc446 and PR #18715 in OpenZFS, issue #18787, the zfs-2.4.4 release, and systemd(1) for systemd.crash_action=.

PostgreSQL conflict with recovery: why both standard fixes make it worse under Patroni

BitPage — Thu, 06 Aug 2026 07:31:00 +0000

Short answer: Both top search results for canceling statement due to conflict with recovery are wrong on a Patroni cluster with replication slots and one replica. hot_standby_feedback=on parks the xmin horizon inside the slot, where it survives the replica’s death and bloats the primary; raising max_standby_streaming_delay converts held-back replay into the exact lag your load balancer ejects the replica for. What I shipped instead was removing on-marked-down shutdown-sessions from the read backend, after measuring that replica lag runs in single-digit milliseconds at the median against a health-check threshold of 10 MB.

The cluster: one primary and one replica under Patroni 3.0.2 with etcd, PostgreSQL 15, synchronous replication, use_slots: true. Web requests from PHP-FPM read through HAProxy 2.4.30 against the replica; CLI work (queues, scheduler) goes to the primary. Once every quarter hour a cron job wipes and rebuilds a large table in full, and it belongs to someone else’s business logic, so it is not negotiable. The application was catching SQLSTATE[40001], and a developer had already written an interceptor at the PostgresConnection level that retries the cancelled read on the primary. The question I was handed was whether he’d done it right. That took a day to answer and turned out to be about something else entirely.

TL;DR

Fix: drop on-marked-down shutdown-sessions from the read backend — it was killing dozens of live sessions per ejection, and the application does not see 40001 for those, it sees a dead socket.
Don’t: hot_standby_feedback=on with replication slots and a single replica — the xmin outlives the replica, so the failure mode moves from “a report dies” to “the primary fills its disk.”
Don’t: raise max_standby_streaming_delay when the load balancer health-checks replayed position — you pay for it with fan-out read outages instead of a couple of dozen cancelled queries a day.
Measure: replay_lag sampled from the primary once a second for twenty minutes. Median in single-digit milliseconds, p95 in tens, worst case a couple of seconds — against a byte threshold that a ~170 MB burst clears in a few seconds.
Trap: Patroni’s lag= parameter is bytes-only. A requirement stated in seconds cannot be expressed in it at all.

Why hot_standby_feedback is the wrong first move with replication slots

Replication slots change what hot_standby_feedback=on costs you, because the xmin stops being tied to a live connection. The PostgreSQL 15 documentation is upfront about the general risk: the parameter “can be used to eliminate query cancels caused by cleanup records, but can cause database bloat on the primary for some workloads” (runtime-config-replication). That sentence is what every answer on the internet is implicitly waving away when it recommends the setting.

The part that applies specifically here is one section over, in the description of slots: “Replication slots provide an automated way to ensure that the primary does not remove WAL segments until they have been received by all standbys, and that the primary does not remove rows which could cause a recovery conflict even when the standby is disconnected” (warm-standby).

Read that as an operator rather than as a feature description. Without slots, feedback xmin lives as long as the session does; kill the replica and the primary is free again. With slots, the xmin settles into pg_replication_slots.xmin and stays there while the replica is gone. There’s one replica in this cluster and no spare. Turning the setting on would mean any replica outage becomes unbounded bloat on the primary, and I’d have traded a couple of dozen cancelled read queries a day for a way to take down the whole database. That one didn’t need a benchmark to reject.

Why raising max_standby_streaming_delay feeds the flap

Delaying WAL replay is only free if nothing downstream measures replay position, and on this cluster the load balancer does exactly that. max_standby_streaming_delay is the second-most-popular answer, and on paper it is the polite one: instead of killing the query, hold back the apply. The docs describe it as “the maximum total time allowed to apply WAL data once it has been received from the primary server,” defaulting to 30 seconds (runtime-config-replication). The cost stays on the replica, which is what makes it look safe.

It isn’t safe here. HAProxy’s health check calls Patroni’s /replica?lag=, and Patroni computes lag from the replayed position. Held-back replay is, by definition, replay lag. Push the parameter from 30 s to 300 s and every conflict-heavy moment turns into a five-minute-wide window where the balancer can decide the replica is stale and pull it from rotation. The outcome would have been strictly worse than the problem: instead of a couple of dozen cancelled queries per day, fan-out outages across all reads.

The full set of candidates, and why each one died:

Candidate	Where the cost lands	Why it was rejected
`hot_standby_feedback=on`	Primary (bloat)	With slots, xmin survives replica downtime; single replica, no spare
`max_standby_streaming_delay` 30s → 300s	Replica (replay lag)	Replay lag is what the health check ejects on; amplifies flapping
Health-check threshold 10 MB → 512 MB	Nothing measurable	The requirement was stated in seconds; any byte value is a guess at time
HAProxy `agent-check` on wall-clock lag	New moving part	Correct, but a new component in the infrastructure; deferred until measured
Drop `on-marked-down shutdown-sessions`	Nothing	Shipped

The third row is the one I would have gotten wrong on my own. I proposed raising the byte threshold, and the customer rejected it with a better argument than mine: the requirement was written in time (“a second of staleness is fine, half a minute is not”), and a byte threshold answers a different question. On steady write load the two are nearly interchangeable. On batch write load they diverge by orders of magnitude, which is the whole story of this cluster.

Two problems that looked like one

Conflicts and replica ejections clustered in the same minutes because they share a cause, not because one causes the other; barely one conflict in twenty lines up with an ejection. There were two symptom streams. One in the PostgreSQL log — a couple of hundred cancelled queries over a week and a half:

ERROR: canceling statement due to conflict with recovery
DETAIL: User query might have needed to see row versions that must be removed.
FATAL: terminating connection due to conflict with recovery

And one in the HAProxy log (N here is the live session count at the moment of the ejection — dozens, in this cluster):

Server slave/pg_replica is DOWN, reason: Layer7 wrong status, code: 503,
info: "Service Unavailable", check duration: 4ms. 0 active and 0 backup
servers left. N sessions active, 0 requeued, 0 remaining in queue.

Both landed in the same handful of minutes each hour. The connection seemed too obvious to check, which is precisely why it deserved checking.

Before I got there I burned time on two hypotheses that a timestamp comparison would have killed in a minute each:

Checkpoints. With checkpoint_timeout=15min the primary logs a checkpoint every quarter hour. The ejections land on entirely different minutes. No overlap at all.
The backup window. The backup script raises max_standby_streaming_delay to 3600 s for the duration of pg_dump, which looked like an excellent suspect. Every single ejection lasted between 6 and 27 seconds. There is not a single hour-long window in the data.

Then I lined up the two event streams properly (the logs are in different timezones, so this needs converting to one scale first) and compared: about a third of the replica ejections and barely a twentieth of the conflict events fall in shared windows. Two independent symptoms driven by the same cron job, not a chain.

I should have run that correlation first. It cost one command, and it invalidated a morning’s worth of theories I’d been carefully stacking on top of each other.

What the lag actually was, in bytes and in seconds

The replica was never slow at applying WAL; it was slow at receiving it, and the difference decides which knob is even relevant. I sampled pg_stat_replication from the primary once per second:

1
2
3
4


SELECT now()::time(0),
 pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS replay_bytes,
 pg_wal_lsn_diff(pg_current_wal_lsn(), flush_lsn) AS flush_bytes
FROM pg_stat_replication;

Put on a relative time scale, with the figures rounded off, a single burst looks like this:

T+0 s replay= 0 MB flush= 0 MB
T+1 s replay=150 MB flush=150 MB ← cron batch starts
T+2 s replay=170 MB flush=170 MB
T+3 s replay= 90 MB flush= 90 MB
T+5 s replay= 0 MB flush= 0 MB ← caught up

flush tracks replay to within a fraction of a percent. Nothing is queued waiting to be applied, so the lag is purely transport: the primary generated WAL faster than the link carried it. At rest the replica sits tens of kilobytes behind. The health-check threshold is 10 MB. A burst of ~170 MB clears that threshold by a factor of 17 and is gone within a few seconds.

Then the measurement that actually settled the argument, using the *_lag interval columns, sampled once a second over twenty minutes:

Metric	`replay_lag`
Median	single-digit milliseconds
p95	tens of milliseconds
Worst sample seen	a couple of seconds
Samples above 5 s	none
Samples above 30 s	none

The stated requirement was “a second is acceptable, half a minute is not.” The replica never came within three orders of magnitude of unacceptable. And the nature of the lag is confirmed again by subtracting one column from the other at the peaks — the three largest samples in the window, figures rounded:

peak 1 ~160 MB | flush 1.0 s | replay 1.0 s | delta: fractions of a millisecond
peak 2 ~100 MB | flush 0.7 s | replay 0.7 s | delta: fractions of a millisecond
peak 3 ~95 MB | flush 1.7 s | replay 1.7 s | delta: fractions of a millisecond

Applying WAL adds fractions of a millisecond. The two cron runs caught in that window sit exactly a quarter of an hour apart, which is the cron job signing its work.

What lag= in the Patroni health check actually guards

The lag= parameter covers exactly one scenario, “replica alive but behind,” because role and state are checked independently of it. The source is more useful than the documentation here: the docs describe what the endpoint is for, the code describes what it does. From patroni/api.py in v3.0.2:

1
2
3
4
5
6
7


max_replica_lag = parse_int(self.path_query.get('lag', [sys.maxsize])[0], 'B')
if max_replica_lag is None:
 max_replica_lag = sys.maxsize
is_lagging = leader_optime and leader_optime > replayed_location + max_replica_lag

replica_status_code = 200 if not patroni.noloadbalance and not is_lagging and \
 response.get('role') == 'replica' and response.get('state') == 'running' else 503

Two things fall out of those few lines. First, parse_int(..., 'B'): the threshold is bytes, full stop. There’s no time-based variant of this endpoint, so the entire discussion about “what number should we put there” was unanswerable until somebody said the units out loud. Second, is_lagging sits in a chain with role and state, each able to produce a 503 on its own. A genuinely broken replica gets caught by state/role anyway: when streaming breaks, the row just vanishes from pg_stat_replication and there’s no byte figure left to compute.

There’s a third detail that makes the threshold shakier than it looks. leader_optime is read from etcd, refreshed once per loop_wait — 10 seconds by default (Patroni settings) — so the byte figure being compared can be up to a loop_wait stale before you even pick a number to compare it against. Against bursts that come and go in a few seconds, the measurement interval is longer than the event.

Prometheus history sizes the one scenario the threshold does cover (pg_replication_lag_seconds, 15 s scrape). Over a month, lag exceeded one minute exactly once: the single real failure in that window ran for hours, when the replica’s VM came back from a hypervisor reboot without network.

A couple of seconds in the worst normal case against hours for a real failure. Nearly four orders of magnitude between them, and the threshold was sitting right up against the lower bound. A threshold that far from the signal it’s supposed to catch isn’t “tuned strictly,” it’s tuned arbitrarily. The group_vars file even carries a comment from whoever raised it earlier, explaining that the bump was meant to avoid false DOWNs during checkpoint and autovacuum. Against a ~170 MB burst, no value in the tens of megabytes was ever going to work.

Why pg_replication_lag_seconds lies when nobody is writing

On a replica, postgres_exporter computes this metric as now() - pg_last_xact_replay_timestamp(), so during write silence it grows on its own without the replica being behind by anything. Its p99 over the month runs to tens of seconds, and reading that as “the replica served data tens of seconds stale” is wrong: it means the last transaction it replayed was that old, because no newer transaction existed. On the primary the same metric is flat zero. Only direct replay_lag sampling from the primary is trustworthy.

The second limitation is resolution. At a 15-second scrape interval, a burst that lasts a few seconds is invisible by construction. For short events the instrument is the HAProxy log, not the dashboard.

Reading burst length out of the HAProxy log

When monitoring resolution is too coarse, the health check itself is a measuring instrument with known timing. With inter 3s fall 3 rise 2:

DOWN is declared after fall 3 × inter 3s = 9 seconds above threshold.
UP returns after rise 2 × inter 3s = 6 seconds below it.
Therefore burst length ≈ DOWN duration + 3 s.

Applied to thirty-odd ejections over a bit more than a week:

DOWN lasted	Share of ejections	Burst was	Survives `fall 5`?
6 s	about two thirds	~9 s	no — eliminated
9 s	a couple of cases	~12 s	no — eliminated
18 s	a couple of cases	~21 s	yes
21 s	roughly a fifth	~24 s	yes
27 s	one case	~30 s	yes

Two thirds of them lasted the minimum possible DOWN, meaning the burst barely crossed the threshold and was already over. Raising fall 3 → fall 5 requires 15 consecutive seconds above threshold and removes around 70% of the ejections, with no staleness risk given the numbers above. The remaining third ran 21–30 seconds and would still eject.

To be clear about status: that change is argued for with the numbers above and agreed in principle, but it is not applied. It sits behind the same sign-off as anything else touching the balancer, and the residual third of the ejections needs either a higher byte threshold or the time-based check I keep coming back to.

That arithmetic required no new tooling, no sampling, and no agreement from anyone. The data had been in the balancer log the whole time.

What I shipped, and how the rollout went sideways

One line removed from the read backend, leaving the failover-critical backends untouched:

1
2
3
4
5


 backend slave
 option httpchk GET /replica?lag=10485760 ...
 http-check expect status 200
- default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
+ default-server inter 3s fall 3 rise 2

The HAProxy 2.4 manual describes the option plainly: with shutdown-sessions, “all connections to the server are immediately terminated when the server goes down” (configuration manual, §5.2). On a read backend fronting a replica that comes back within seconds, that is damage with no upside. In the logs it killed dozens of sessions at once. The application does not see 40001 for those connections, it sees a severed socket, which the developer’s interceptor never catches. The replica can still be ejected during a burst; live reads now finish, and new connections fall through to use_backend master if { nbsrv(slave) eq 0 }. The option stays in the master and long_queries backends, where killing sessions during failover is the point.

Three things about the rollout worth stealing:

--check --diff on the whole playbook, not just my part. Someone had added weight 80 / weight 20 by hand on the server and never put them in the template. A normal run would have silently erased them. I pulled the repository up to the server’s actual state first, then applied my change on top.
A group_vars file turned out to be a hardlink shared by several inventories. One edit, every environment changed.
I rolled it out with a restart instead of a reload. In master-worker mode kill -USR2 to the master reloads without dropping anything: the master holds the sockets, the port never closes, and the old worker drains its sessions. A restart dropped every connection at once — which is precisely the damage this change exists to prevent. Shipping a fix for connection drops by dropping every connection is the kind of thing you only do once.

Why Patroni ignores the log settings in your playbook

Anything under bootstrap.dcs is applied once, at cluster initialization, and edits to it afterwards do nothing. I found this while looking for something else entirely: pg_log had never rotated. Tens of gigabytes on the primary, single-digit gigabytes on the replica, files that had been piling up for years, on a cluster whose Ansible template configures rotation perfectly well.

Nobody had edited anything by hand this time. The Patroni documentation states the rule with unusual force: “Once Patroni has initialized the cluster for the first time and settings have been stored in the DCS, all future changes to the bootstrap.dcs section of the YAML configuration will not take any effect!” (YAML configuration). The section “will be written into /<namespace>/<scope>/config of the given configuration store after initializing the new cluster,” and from that moment the copy in etcd is the one that counts.

Changing those values later goes through patronictl edit-config or the REST API, not the file (dynamic configuration). Which means a Patroni cluster has a whole class of settings where the playbook is documentation rather than configuration, and --check --diff will never tell you, because the file on disk matches the template exactly. It is right and the cluster still ignores it.

What the application layer got wrong

The interceptor I was asked to review was correct; the defects were around it. Findings handed back to the developers:

ERROR and FATAL are different events. ERROR: canceling statement due to conflict with recovery cancels the statement and leaves the connection usable. FATAL: terminating connection due to conflict with recovery does not. Over the week and a half of logs, nearly all the events were ERROR and a handful were FATAL. One retry strategy cannot serve both.
The retry loop was unreachable past its first iteration, so it retried once regardless of configuration.
The application under-reports by half. Over two weeks the database logged roughly twice as many conflict events as the application reported through its single Log::warning. That gap is why nobody knew the size of the problem.
application_name is empty in every event, so a conflict cannot be attributed to a service from the database logs — only to a database user.
A pre-existing lazy-connection bug in the Laravel wrapper. bindValues() opens with $this->getPdo()->getAttribute(...), and getPdo() resolves the lazy closure and opens a connection to the primary. Every query with bindings calls it, read-only ones included, so the read/write split silently leaks connections to the primary on any parameterised read.

Half of my initial review comments on the interceptor did not survive my own re-checking. Separating “I would have written this differently” from “this is a defect” is most of what code review is, and I am not always good at it.

Step by step

Count the events before theorizing: split ERROR from FATAL in the PostgreSQL log and get the rate per day.
Correlate every symptom stream by timestamp on one timezone scale, first, before building any causal story.
Sample pg_stat_replication from the primary once per second; compare replay_lsn against flush_lsn to separate transport lag from apply lag.
Sample the *_lag interval columns to get lag in seconds, which is the unit your requirement is almost certainly written in.
Read your health-check endpoint’s source for the units it actually accepts before negotiating a threshold value.
Derive burst duration from the balancer’s own inter/fall/rise timing when scrape intervals are too coarse.
Check whether hot_standby_feedback is safe for your topology: with slots and one replica, it is not.
Run --check --diff across the whole playbook before applying, and reload rather than restart.
For anything Patroni owns, read the live values with patronictl show-config rather than trusting the YAML in your repository.

Bottom line

Neither of the two canonical answers to conflict with recovery was usable on this cluster, and both failures were topology-specific rather than wrong in general. hot_standby_feedback assumes you have no slots or a spare replica; raising the streaming delay assumes nothing downstream measures replay position. Check those assumptions against your own cluster before pasting either one into postgresql.conf.

The larger lesson is about units. The health check measured “how much WAL has not arrived” in bytes while the business requirement said “how stale is the data I am serving” in seconds. Every discussion about the right threshold was unanswerable until somebody wrote both sentences down next to each other. The measured answer was single-digit milliseconds at the median against a real failure measured in hours — and the deferred work, an agent-check agent reporting up/down from now() - pg_last_xact_replay_timestamp(), is now a decision that can be made with numbers instead of a guess.

Gmail bounced everything the relay sent: three broken auth mechanisms and a silent OpenDKIM

BitPage — Thu, 06 Aug 2026 07:08:22 +0000

Short answer: Three independent mechanisms were broken at once, and fixing any one of them changed nothing visible. The domain had no v=spf1 record (only a dead SenderID one), the PTR resolved forward to a different server, and OpenDKIM was signing nothing because KeyTable and SigningTable were attached as refile: while their contents were in plain two-column format. That last one cost the most time: when OpenDKIM can’t find a key it logs absolutely nothing, so the daemon reads active, the config parses, the key is readable, and mail still leaves unsigned.

The server is an outgoing relay for transactional mail from a couple of dozen application hosts: password resets, notifications, alerts. Inbound mail for the domain goes elsewhere, to Google Workspace as MX. Postfix on a legacy container with more than two years of uptime, OpenDKIM 2.10.3, no Ansible, edits by hand only. The daily log summary read a four-digit bounced against a single-digit sent: thousands of rejections, a handful of deliveries. Password reset mail was among the casualties, so people could not get back into their accounts. Nobody had changed anything on this box in years, which turned out to be the whole story.

TL;DR

Root cause (DKIM): KeyTable/SigningTable mounted as refile: while the entries were plain example.org mail._domainkey.example.org lines. Under refile: the key is a regex matched against the whole From: address, so example.org never matches noreply@example.org. Changing one word per line to file: fixed it.
Also broken: InternalHosts listed fewer than ten addresses against a Postfix mynetworks two to three times longer. Mail from the app hosts was being verified instead of signed.
Also broken: the only SPF-shaped TXT record was spf2.0/mfrom,pra SenderID. RFC 7208 discards anything that doesn’t begin with v=spf1, so the domain effectively had no SPF record.
Also broken: the PTR pointed at a hostname whose A record led to a different machine. Forward-confirmed reverse DNS failed.
Trap: grep -c DKIM-Signature /var/log/mail.log returns 0 whether signing works or not. Postfix does not write message headers to the log.
Trap: testing with sendmail from the mail server itself passes, because 127.0.0.1 was in InternalHosts. Real traffic from other hosts kept going out bare.

Why a server nobody touched stopped being deliverable

The requirements moved while the configuration stood still. Gmail began requiring authentication from bulk senders in February 2024 and has been ramping enforcement since, and a config that predated those rules simply stopped qualifying (Email sender guidelines). There was no change window to roll back to, no deploy to blame. The rejection came back verbatim as:

550-5.7.26 Your email has been blocked because the sender is unauthenticated.
550-5.7.26 Gmail requires all senders to authenticate with either SPF or DKIM.
550-5.7.26 Authentication results:
550-5.7.26 DKIM = did not pass
550-5.7.26 SPF [example.org] with ip: [203.0.113.51] = did not pass

Both named mechanisms failing at once should have been the tell that this was not one bug. I read it as one bug anyway and spent the next hour paying for it.

Three of the four facts in my own notes were wrong

I started from a ticket written a day earlier that already contained a diagnosis, and rechecking it by hand killed three of its four claims. The notes said: PTR is fine and forward and reverse match; DNS is hosted on Cloudflare; DMARC is absent.

The PTR pointed at legacy.example.org, whose A record led to 198.51.100.216, a different server entirely. dig NS returned ns-*.awsdns-*, so DNS was on Route 53, not Cloudflare. DMARC was present and had been all along, v=DMARC1; p=none. One claim out of four survived. These were my own notes, from the previous day.

The other rejection message had been telling me about the PTR the whole time:

550-5.7.25 [203.0.113.51] The IP address sending this message does not have a PTR
550-5.7.25 record setup, or the corresponding forward DNS entry does not match the sending IP

The operative half is the second clause. Not “you have no PTR record” but “the forward entry doesn’t match”. I had read to the first familiar phrase, seen a PTR existed, and moved on. RFC 1912 §2.1 states the rule plainly: “Make sure your PTR and A records match. For every IP address, there should be a matching PTR record in the in-addr.arpa domain” (RFC 1912). The same section warns that getting this wrong “can cause loss of Internet services similar to not being registered in the DNS at all”, which is a fair description of a bounce column four digits wide.

The fix had two possible shapes. Editing the A record of legacy.example.org to point here would have satisfied the check, but that hostname is live and fronts an unrelated service, so the blast radius covered somebody else’s system. Repointing the PTR to mail.example.org, whose A record already resolved to the sending IP, changed nothing outside this server. I also set Postfix myhostname to mail.example.org so the HELO name matches the PTR.

Why the SPF record that existed didn’t count

There was a TXT record on the domain, and it was worth exactly nothing, because it was a SenderID record rather than an SPF one. It began spf2.0/mfrom,pra. RFC 7208 §4.5 is unambiguous about what a verifier does with that: “Starting with the set of records that were returned by the lookup, discard records that do not begin with a version section of exactly ‘v=spf1’” (RFC 7208).

Discarded means the domain had no SPF record at all as far as Gmail was concerned, which is precisely what SPF [example.org] ... = did not pass was reporting. A record that looks like SPF to a human and is invisible to every verifier is worse than an empty zone, because it stops anyone from looking further. Published in its place:

v=spf1 ip4:203.0.113.51 include:_spf.google.com include:spf.example.net ~all

Why grepping the mail log for DKIM-Signature proves nothing

Postfix does not write message headers to its log, so the count is zero whether signing works or not. My first check was grep -c "DKIM-Signature" /var/log/mail.log, which returned 0, and I took that as evidence that nothing was being signed. It wasn’t evidence of anything. The method cannot distinguish a broken signer from a working one, and I used it as the basis for the next hour of work.

The check that does work is to send a message to a local mailbox and read the headers of the delivered file, /var/mail/nobody. That shows the actual DKIM-Signature header or its absence, with no inference in between. Building that first would have made every subsequent change a one-minute yes or no. I built it fourth.

OpenDKIM reports active, parses clean, and signs nothing

OpenDKIM writes no log line when it fails to find a signing key for a message, so systemctl is-active returning active alongside a clean startup is zero evidence that signing happens. I found and fixed two genuine defects on the strength of that assumption, and neither produced a signature.

The first was the config itself. /etc/opendkim.conf had been truncated to four lines, with no Mode, no KeyTable, no SigningTable. The fingerprints of a package upgrade were right there: /etc/opendkim/ had been touched recently while the files inside it were years older. I restored the missing directives and restarted. Clean start, no errors, no signature.

The second was key ownership. The private key belonged to _apt:uuidd with mode 0600, and the daemon runs as opendkim. Checking as the service user rather than as root is the only way to see this:

su -s /bin/sh opendkim -c "head -c 20 /etc/opendkim/default/mail.private"
head: cannot open '/etc/opendkim/default/mail.private' for reading: Permission denied

A real defect, and chown opendkim:opendkim is mandatory. The key became readable. Still no signature. Two genuine bugs repaired, both necessary, neither sufficient, and my supply of hypotheses was gone.

The actual root cause was one word, and it was in the man page

The tables were attached as refile: while their contents were in plain two-column format, and under refile: the lookup key is treated as a regex matched against the entire From: address. I stopped guessing and opened man opendkim.conf on the server itself, same 2.10.3 as the running binary, which is more authoritative than whatever version the web is serving. The answer sits in one paragraph about dataset types:

“If this table specifies a regular expression file (“refile”), then the keys are wildcard patterns that are matched against the address found in the From: header field. For all other database types, the full user@host is checked first, then simply host, then user@.domain (with all superdomains checked in sequence…)”

The tables held lines like this:

example.org mail._domainkey.example.org

Under refile:, the pattern example.org is matched against noreply@example.org and does not match, so no key is found and the message leaves unsigned without a single log line. Under file:, the lookup falls through to the second step, “then simply host”, where example.org matches exactly. The fix was one word in two lines:

1
2
3
4


-KeyTable refile:/etc/opendkim/KeyTable
-SigningTable refile:/etc/opendkim/SigningTable
+KeyTable file:/etc/opendkim/KeyTable
+SigningTable file:/etc/opendkim/SigningTable

The other direction works too: rewrite every table entry to *@example.org and keep refile:. I rejected it because the tables covered several domains, and rewriting every line of data is a wider edit than changing the dataset type to the one that already matches the format on disk.

The second half of the same bug: InternalHosts was a fraction of mynetworks

OpenDKIM only signs mail arriving from hosts in InternalHosts; everything else it verifies instead, so any app server missing from that list was sending bare mail by design. The man page states the whole behaviour in one line, describing the option as identifying “a set internal hosts whose mail should be signed rather than verified” (typo included, as printed in 2.10.3).

InternalHosts pointed at TrustedHosts, which listed fewer than ten addresses. Postfix mynetworks listed two to three times as many. Syncing the short list against the long one tripled it, finally covering the application hosts that had been relaying through a server which was, on paper, configured to sign their mail. The two files are not required to match exactly, since they answer different questions, but a host that can relay and cannot be signed is almost always an oversight rather than a decision.

This is also why my manual test had lied to me earlier. Mail sent with sendmail from the relay itself originates from 127.0.0.1, which was in the list, so a signature appeared and I briefly thought the job was done. The traffic that actually mattered came from the application hosts, none of them in the file. Test along the path production traffic takes, not the path that is convenient from an SSH session.

The four defects and the check that actually proves each one

Broken	Symptom you see	Check that settles it
No `v=spf1` record, SenderID only	`SPF … = did not pass`	`dig +short TXT example.org` — the string must begin `v=spf1`
PTR forward mismatch	`550-5.7.25 … forward DNS entry does not match`	`dig -x <ip>`, then resolve that name’s A record back — it must return `<ip>`
`refile:` on plain-format tables	Nothing. No log line at all.	Read the `DKIM-Signature` header of a delivered message
`InternalHosts` narrower than `mynetworks`	Signs locally, unsigned from apps	Send from a real application host, not from the relay

The third row is the one worth internalising. Three of these four failures announce themselves somewhere. The dataset-type bug produces no error, no warning, and a healthy-looking service, and the only way to observe it is to inspect the output rather than the process.

Step by step

Build the verification path before touching anything: send to a local mailbox and read the delivered headers. Every later change then gives an immediate yes or no.
Recheck every fact you were handed, including notes you wrote yourself yesterday.
Read the whole SMTP rejection. 5.7.25 and 5.7.26 name different failures, and in both the specificity lives in the second clause of the sentence.
Resolve the sending IP with dig -x, then resolve the answer forward. Both directions, or it doesn’t count.
Confirm the TXT record begins with v=spf1 rather than merely mentioning SPF.
For DKIM: confirm the dataset type matches the table format, confirm the service user can read the key with su -s /bin/sh opendkim -c "…", and confirm InternalHosts covers every host in mynetworks.
Send the final test from a host that is not the mail server.

The end state was DKIM-Signature: v=1; a=rsa-sha256; d=example.org; s=mail in the delivered headers, spf=pass, dmarc=pass, and 250 2.0.0 OK from Gmail. Total time was about an hour and a half, most of it spent on the silent one.

What authentication didn’t fix: six messages in ten still bounce

Passing SPF and DKIM gets mail accepted, not trusted, and this domain’s reputation problem survived every fix above untouched. A full day of counters after the fix still showed more rejections than deliveries — roughly six messages in ten going nowhere — and among them close to a hundred a day addressed to @iclaud.com, a typo of @icloud.com that the application has been faithfully retrying for years. Bounce rate is not the metric Google publishes a threshold for; that one is spam rate, which bulk senders are asked to keep under 0.30% in Postmaster Tools. They are different numbers, but they feed the same reputation ledger, and 0.30% is a useful sense of the scale a receiver considers acceptable. Mail with flawless authentication still lands in the spam folder if the domain spends years hammering addresses that don’t exist.

The reason nobody knew is that the feedback loop had been cut at both ends. The server delivered non-delivery reports to the sender address, whose domain it considered local via mydestination, and no local user by that name existed, so the reports were discarded on arrival. That was hundreds of destroyed reports a day. The application mails a dead address, the report about it is deleted by the same server, nobody finds out, and the application keeps mailing.

What I deliberately did not do was tighten DMARC past p=none. Moving to quarantine before the statistics are clean would route legitimate mail to spam by my own hand, and there is no rush. Still outstanding: the envelope sender equals the From: address, where a dedicated bounce address with VERP belongs; certbot 0.12.0 is dead on this box, still calling the retired acme-v01 endpoint, with a certificate that expired years ago. And every change here was made by hand on a container that lives outside Ansible, so none of it exists in any repository. The next person to open this server will find no trace of why it works.

Bottom line

When a daemon reports active and does not do its job while logging nothing, stop repairing hypotheses and go read the man page for the semantics of the specific option you are relying on. Silence is not the absence of a fault. It is its own class of behaviour, and it is always described somewhere in the documentation, usually in a sentence about how lookups are performed.

The process mistake was ordering. I fixed three genuine defects (truncated config, key ownership, then the host list) before finding the one that actually mattered, and each time I expected the next test to pass. A reliable check built at minute one, reading the headers of a delivered message, would have turned each of those into a one-minute negative result instead of an hour of accumulating false hope. Build the instrument before you start turning things.

How to migrate a stateful Docker app off a server with a dying disk

BitPage — Fri, 24 Jul 2026 11:35:00 +0000

Short answer: When the source server’s disk fails on writes but still reads fine, treat it as read-only hardware. Stream pg_dump -Fc over SSH straight to the target machine, pull Docker images from your registry instead of rebuilding end-of-life Dockerfiles, and anchor every rsync exclude with a leading slash. That moved an app of some fifteen services, with a PostgreSQL database on the order of ten gigabytes, onto a new node without a single write to the failing disk.

The source was an old OpenVZ box with a degraded RAID and a disk that had quietly stopped accepting writes. Reads were healthy. On it lived a Dockerized PHP application: some fifteen services (nginx, php-fpm, a pool of queue workers, PostgreSQL, Redis), a database on the order of ten gigabytes, a few gigabytes of media. The job was to stand up a production-like copy on a new Proxmox node for testing, with the real traffic cutover scheduled later.

TL;DR

Fix: run pg_dump -Fc on the source but pipe it through ssh -T to the target’s disk — the source only reads, the target does all the writing.
Fix: don’t rebuild images based on EOL distributions; pull the exact production images from the registry, reusing /root/.docker/config.json from the source host.
Trap: rsync --exclude=log/ without a leading slash matches log at every depth. It silently removed vendor/yiisoft/yii2/log/ from the copy and the app answered HTTP 500.
Trap: a dump to a dying disk hangs without an error. Watch the output file’s size delta, not the process list.

Why pg_dump hangs on a dying disk instead of failing

A dump writing to a dead disk doesn’t error out; it stops making progress while the process stays alive. My first attempt was the obvious one: pg_dump -Fc into a local backup/ directory on the source. The file grew to roughly a gigabyte and froze. Two size checks six seconds apart returned the same size, with the process still running. The disk had died on the write path specifically, and a hung dump was exactly how that looked from userspace.

The liveness check lied to me too. The watcher used pgrep -f pg_dump, and its own command line contained the string pg_dump, so it matched itself and kept reporting the dump as alive. ps -C pg_dump (exact process name match) or a size delta on the output file are the checks that don’t fool themselves.

Streaming the dump: keep every write off the source

The fix is to leave the source with the read half of the work and move the write half to another machine. pg_dump runs on the source, but its output goes over SSH to the target’s healthy disk:

1
2
3


ssh -T old-host \
 'docker exec app-postgres pg_dump -Fc -U app appdb' \
 > /srv/dumps/appdb.dump

Dumping from a live database is fine here. The PostgreSQL 11 documentation states it directly: “pg_dump does not block other users accessing the database (readers or writers)” (pg_dump reference).

Two details matter for the transport. ssh -T disables pseudo-terminal allocation, and docker exec must run without -t: a tty mangles the binary custom-format stream. With both in place the gigabyte-and-a-half dump completed with exit code 0, and on the target pg_restore --no-owner -j 4 -d appdb unpacked it back into the same ten gigabytes or so, roughly a hundred tables, with the rum extension intact.

Pull EOL images, don’t rebuild them

If production images already exist in a registry, pull them; rebuilding their Dockerfiles years later is a dead end. My first docker compose pull on the new host failed with Error response from daemon: error from registry: access forbidden — a private GitLab registry the new machine wasn’t authorized against. Instead of fixing auth, I detoured into rebuilding the images from the same Dockerfiles production once used. That went like this:

Base image	Debian release inside	What apt says in 2026
`nginx:1.17.4`	buster	`Release 404 Not Found` on deb.debian.org
`postgres:11`	stretch	PGDG packages for PostgreSQL 11 removed, 404

Buster I could route to archive.debian.org. Then the stretch-based postgres:11 build died on the PGDG repository, which no longer serves packages for an EOL major at all. I stopped there and admitted the detour was me solving the wrong problem.

The actual answer was sitting on the dying host the whole time: it pulls those images itself, which means working registry credentials live in its /root/.docker/config.json (that’s where docker login stores them). Copying that one file to the new host let docker compose pull fetch the whole set of images — the exact bytes production runs, no build step.

The rsync exclude that deleted framework files

--exclude=log/ without a leading slash matches every directory named log at any depth, not just the top-level one you meant. I added it to skip a couple of gigabytes of application logs during the code-and-media rsync. The copy finished clean, and the app came up with HTTP 500:

include(/app/vendor/yiisoft/yii2/log/Logger.php):
failed to open stream: No such file or directory

Nothing in that message points at rsync. A missing framework file reads like a corrupted vendor directory, and I chased that theory first. The real cause: the unanchored pattern had stripped vendor/yiisoft/yii2/log/ out of the transfer. The rsync man page is explicit about the rule: “if the pattern starts with a / then it is anchored to a particular spot in the hierarchy of files, otherwise it is matched against the end of the pathname.”

The fix is one character: --exclude=/log/ anchors the pattern to the transfer root. A second rsync run then sent only the log directories it had wrongly skipped inside the code, a few seconds of incremental transfer.

Testing against live integrations without side effects

Queue workers and cron are what produce outbound effects; a web tier on top of a copied database is safe to start. This app talks to a real payment provider, CRM, SMTP, and ClickHouse, and the test copy had to come up without firing any of that. The compose setup was already split into docker-compose.yml (web) and docker-compose.queue.yml (workers and scheduler), so the whole safety switch was one environment variable:

1

COMPOSE_FILE=docker/docker-compose.yml docker compose up -d

Web and API up, every queue worker and the scheduler down, database an isolated copy. Smoke test: API returned 200, the site returned its usual 302, the admin backend rendered the login page.

One integration still broke: ClickHouse refused connections from the new node’s IP (“Connection refused”), and backend pages that touch it answered 503. External allowlists (database, mail, analytics) are part of the migration checklist, not something to discover during cutover.

Step by step

Establish the disk’s failure mode first: “reads work, writes hang” changes every decision after it.
Copy /root/.docker/config.json from the source to the target; docker compose pull the production images.
Stream the database: ssh -T source 'docker exec db pg_dump -Fc -U app appdb' > dump on the target.
Restore with pg_restore --no-owner -j 4 -d appdb.
Rsync code and media with anchored excludes only (--exclude=/log/, not --exclude=log/).
Bring up the web tier alone via COMPOSE_FILE, leaving queues and cron down.
Smoke-test, and collect every external service that needs the new IP allowlisted.
Cut over later in a quiet window: fresh dump, DNS, allowlists.

Bottom line

Dying hardware changes the rules: the source is read-only whether you accept it or not, so structure the whole migration around that. Stream the dump instead of writing it locally, pull images instead of rebuilding EOL bases, and anchor rsync excludes before they cut into vendor/. The copy went from “disk is dying” to a responding production clone in a few hours, and most of the lost time was me fighting battles (EOL apt archives, a phantom vendor corruption) that the constraints had already decided.