BitPage

PHP 5.6 and PHP2Shell: is it vulnerable, what fixes it, and how to prove you fixed it

BitPage — Mon, 26 Oct 2026 05:25:00 +0000

Short answer: PHP 5.6 is vulnerable, and you can confirm it from the source rather than taking a vendor’s word for it: in vanilla 5.6.40 zend_mm_safe_error() ends in a zend_bailout(), which is the longjmp out of a failed allocation that CVE-2026-109618 is about, and session.upload_progress.enabled defaults to "1" with the callback registered unconditionally, which is the consumer path for CVE-2026-109619. The only free fix is Ondřej Surý’s sury patch 0261, applied to vanilla 5.6.40, and the piece it deliberately omitted shipped a day later in package revision 5.6.40-105.

PHP’s advisories cover 7.4 and newer. For 5.x there is no statement at all: not vulnerable, not unaffected, nothing. Version scanners fill the gap with “EOL, please upgrade”, which is not an answer when the code in question uses mysql_*, ereg and the /e modifier and the upgrade is a quarter of work rather than an afternoon.

The two CVEs and the fleet-wide picture are in the overview. This is the narrowest branch of the set and the one where proving the fix is hardest.

TL;DR

Vulnerable: yes. Both code paths are present in vanilla 5.6.40, verifiable in Zend/zend_alloc.c and ext/session/session.c.
Fix: sury patch 0261-Avoid-state-corruption-via-bailout.patch, by Arnaud Le Blanc, co-author of the upstream fix. It applies to vanilla 5.6.40 with line offsets only and no dependencies.
Take revision 5.6.40-105, not -104. -104 on 21 October is the bailout fix alone; -105 on 22 October adds the CVE-2023-0662 multipart limit that the bailout patch explicitly relies on.
Trap: with -104 alone there is no static marker whatsoever: not the version, not an ini value, not a new directive. Only a behaviour test.
Trap: the 2019 docker-library Dockerfile for 5.6 no longer builds. Everything it fetches has moved or shut down.

Is 5.6 actually vulnerable

Yes, and the useful part is that it is checkable without a PoC. Wiz and Sansec both put the start of the vulnerable code at 5.4.0. Rather than trust that, I read vanilla php-5.6.40 from php-src.

The 109618 root is zend_mm_safe_error() in Zend/zend_alloc.c. The shape of it is the same code the official EOL backports delete on 7.4 and 8.0:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


 heap->overflow = 1;
 zend_try {
 zend_error_noreturn(E_ERROR, format, limit, size);
 } zend_catch {
 ...
 } zend_end_try();
 } else {
 heap->overflow = 2;
 }
 zend_bailout();

That zend_bailout() is a longjmp out of a failed allocation, and if the allocation failed while the request body was being parsed, it unwinds out of the middle of the parse. A detail from the same function explains the log signature worth grepping for: when the error happens before any script is executing, it sets error_filename = "Unknown" and error_lineno = 0, which is why Allowed memory size of … exhausted … in Unknown on line 0 is the line that tells you an OOM happened before user code started.

The 109619 consumer is session upload progress, and on 5.6 it is on out of the box. In ext/session/session.c the directive is declared as STD_PHP_INI_BOOLEAN("session.upload_progress.enabled", "1", ZEND_INI_PERDIR, …), default "1", and php_rfc1867_callback = php_session_rfc1867_callback; is assigned unconditionally at module startup, with the only guard being if (!PS(rfc1867_enabled)) inside the callback. The application never has to call session_start() for that path to be live.

Both entry points exist, then, and the class of bug with them. Whether the specific exploitation technique from the public write-ups works against 5.6 is not something I tested, and I would not claim it either way. Supporting evidence from a direction I did not expect: Ubuntu’s own CVE record lists php5 on trusty as released at 5.5.9+dfsg-1ubuntu4.29+esm18. That is a vendor spending paid ESM effort on a branch two minor versions older than this one, which is not something you do for a bug you think is theoretical. Note the shape of that version string too: upstream stays 5.5.9 and only the Ubuntu revision moves, so php -v on a fixed trusty box is identical to php -v on an unfixed one.

Where to get a fix

One free source, and it is a good one.

The sury php5.6 package carries 0261-Avoid-state-corruption-via-bailout.patch. The author is Arnaud Le Blanc, co-author of the upstream fix, with Ilija Tovilo credited; the header says “Backported from the PHP 7.4 version” and “Minimal mitigation for EOL version.” It applies to a vanilla 5.6.40 tarball with line offsets only, no fuzz, and no prerequisite patches at all, which is unusual, since on 7.1 through 7.3 the equivalent patch needs the ZSTR_MAX_LEN backport first.

What I like about this patch is that it documents its own gaps in the header. Verbatim, from the list of deviations:

The max_multipart_body_parts hunk of the 7.4 patch is left out; the limit comes with the CVE-2023-0662 backport (Introduce max_multipart_body_parts INI).

A patch that tells you what it is not doing is a patch you can plan around. Which brings us to the interesting part.

The rejected options, briefly. A community port for 5.6.40 exists and does include a parts limit, but its authors tested it against the built-in CLI server, so it is at best a donor for a missing piece. CloudLinux alt-php56 is paid and awkward in Docker. Official PHP backports, Freexian ELTS and Zend have nothing for 5.x.

The gap, and the fact that it closed the next day

When I first worked through this, the omission above was an open problem. The sury maintainer had confirmed on the issue tracker that 5.6 has no multipart parts limit, because the CVE-2023-0662 fix was never backported to that branch, and an update was promised with no date. The conclusion was that the patch changes what happens when an OOM occurs without limiting the input that causes it, so a proxy-level body limit stays mandatory.

Then I pulled the current source package and read the changelog. The update had shipped:

php5.6 (5.6.40-105) unstable; urgency=medium
* Backported from 8.0.28
+ [CVE-2023-0662]: DOS vulnerability when parsing multipart request body.
-- Ondřej Surý <ondrej@debian.org> Thu, 22 Oct 2026 08:02:38 +0200
php5.6 (5.6.40-104) unstable; urgency=medium
* Backported from 8.2.35
+ [CVE-2026-109618, CVE-2026-109619]: Avoid state corruption via bailout.
-- Ondřej Surý <ondrej@debian.org> Wed, 21 Oct 2026 08:55:15 +0200

Revision -105 adds 0263-Introduce-max_multipart_body_parts-INI.patch, touching main/main.c and main/rfc1867.c, and the current series has 264 patches with both 0261 and 0263 in it. The missing half exists after all, one day behind the first revision, and the two patches together are the complete minimal fix for 5.6.

One thing to expect when you go looking for it: nothing in the pool is named a bare -105. The 5.6 packages ship as git-buildpackage snapshots, so the version you will actually see is of the form 5.6.40-105+0~20261022.135+debian11~1.gbp37a17e, and the top changelog entry in the source package is marked ** SNAPSHOT build ** and UNRELEASED. Compare on the -105 part and ignore the tail. If you compare on the whole string, or expect a clean Debian revision, you will conclude the fix is not published.

Three consequences, in order of how much they matter:

If you patched 5.6 on 21 October, you took -104 and you are not done. Same story on 7.1, where 7.1.33-80 carried the bailout fix and 7.1.33-81 added the prerequisite a day later. On EOL branches the fix can arrive in pieces, and patching early is a reason to re-check rather than to close the ticket.
The static marker problem goes away (see below).
The parts limit now has a number. max_multipart_body_parts is declared PHP_INI_ENTRY("max_multipart_body_parts", "-1", PHP_INI_SYSTEM|PHP_INI_PERDIR, NULL) and the default -1 falls back to max_input_vars + max_file_uploads, which on stock defaults is 1000 + 20. Beyond that you get “Multipart body parts limit exceeded %d. To increase the limit change max_multipart_body_parts in php.ini.” That fallback formula is identical in the 5.6 backport and in vanilla 8.2’s main/rfc1867.c, so a 5.6 build with both patches now has the same effective 1020-part ceiling as a current PHP.

The proxy-level body limit is still worth keeping. One reason is that a limit at the edge costs nothing; the better reason is that blocking multipart/* alone does not close the urlencoded variant of the amplification, so a body-size cap has to exist regardless of what PHP is doing.

Trap: with the bailout patch alone there is no marker at all

This is what makes 5.6 harder than every other branch. php -v prints 5.6.40 before and after. php -i prints session.upload_progress.enabled => On even though the patch disables the feature in code. On 7.1 through 7.4 you can at least look for max_multipart_body_parts, since it is absent from every vanilla 7.x tree. I counted zero occurrences in main/main.c for php-5.6.40, php-7.1.33, php-7.2.34, php-7.3.33 and php-7.4.33. With 0261 alone, 5.6 has no new directive, so that marker does not exist either.

Taking -105 fixes this as a side effect: with 0263 applied, max_multipart_body_parts does show up in php -i, and 5.6 gets the same static marker the 7.x branches have. That is a second reason to take the later revision even if you believe your edge limits make the parts limit redundant — it makes the build auditable.

If you are stuck on the bailout patch alone, the only proof is behavioural:

1
2
3


docker run --rm IMG php -r 'register_shutdown_function(function(){echo "SHUTDOWN\n";}); ini_set("memory_limit","8M"); $a=str_repeat("x", 64*1024*1024);'; echo "exit=$?"
# patched: Fatal error: Allowed memory size ..., exit=1, no SHUTDOWN
# unpatched: Fatal error ..., SHUTDOWN printed, exit=255

The patched build routes the fatal error to stderr and terminates with _exit(1), so shutdown functions never run. I confirmed that same signature independently on a sury-patched php8.0-cli build: exit 1, stderr, no shutdown function, against 255 and stdout on an unpatched one.

And in the build itself, assert on a symbol the patch introduces:

1

grep -c array_init_singleton Zend/zend_API.h # 0 in vanilla 5.6.40, >=1 after the patch

I checked that count against vanilla 5.6.40, 7.4.33 and 8.0.30: zero in all three, and every one of the six EOL patch sets adds it. Do not use zend_objects_store_mark_destructed for this: it exists in unpatched sources, so the assert passes unconditionally. That was my first choice and it was useless.

Trap: the 2019 Dockerfile no longer builds

The approach that keeps the blast radius small is the same as on the other branches: build a base image from the official docker-library Dockerfile for 5.6, add the patch, change one FROM line in the application image. The right commit is fab49d4cb1c6, the last one whose tree still carries 5.6/, directly before e9320fdb752e ‘Revert “Update to 5.6.40” (this removes the EOL 5.6 series)’. The variant to take is 5.6/stretch/fpm, with ENV PHP_VERSION 5.6.40 on line 64 and the sha256 1369a51eee3995d7… on line 66 — same OS and same layout as the running image.

Then nothing it references still exists. Three separate breakages, none of them about PHP:

Debian stretch is off the mirrors. Point at archive.debian.org and accept expired Release files, because the archive’s are.
The tarball URL is dead. secure.php.net/get/…/from/this/mirror no longer resolves; www.php.net/distributions/… does.
The keyserver shut down. ha.pool.sks-keyservers.net is gone; keyserver.ubuntu.com works. The signature still verifies — “Good signature”, with the signing key long expired, which was already true in 2019.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


FROM debian:stretch-slim
# stretch moved to archive.debian.org in its entirety
RUN printf '%s\n' \
 'deb http://archive.debian.org/debian stretch main' \
 'deb http://archive.debian.org/debian-security stretch/updates main' \
 > /etc/apt/sources.list \
 && rm -f /etc/apt/sources.list.d/* \
 && echo 'Acquire::Check-Valid-Until "false";' > /etc/apt/apt.conf.d/99archive
ENV PHP_URL="https://www.php.net/distributions/php-5.6.40.tar.xz" \
 PHP_ASC_URL="https://www.php.net/distributions/php-5.6.40.tar.xz.asc"
# gpg --keyserver keyserver.ubuntu.com --recv-keys "$key"
COPY php2shell-5.6-0261-avoid-state-corruption.patch \
 php2shell-5.6-0263-max_multipart_body_parts.patch /usr/src/
RUN set -eux; \
 docker-php-source extract; \
 cd /usr/src/php; \
 patch -p1 --forward < /usr/src/php2shell-5.6-0261-avoid-state-corruption.patch; \
 patch -p1 --forward < /usr/src/php2shell-5.6-0263-max_multipart_body_parts.patch; \
 grep -q array_init_singleton Zend/zend_API.h; \
# repack with the patch in: docker-php-ext-install re-extracts php.tar.xz later
 tar -cJf /usr/src/php.tar.xz -C /usr/src php; \
 ./configure ... # as in the original

Keep the edits confined to source delivery. The PHP build and every configure flag stay exactly as the official image had them, which is the entire reason to start from the official Dockerfile instead of writing one.

That repack of php.tar.xz deserves a sentence. The official Dockerfile ends with docker-php-source delete, which leaves the pristine tarball behind; an application image later running docker-php-ext-install re-extracts it and compiles extensions against unpatched sources. I have not seen this break anything in practice, so it is a precaution rather than a reported bug — but it costs one line.

I rejected debian:12 plus the sury php5.6-fpm package. The fix is there and the OS keeps getting updates, which is genuinely better long-term, but the layout is different (/etc/php/5.6/, php-fpm5.6, extensions via apt) and rewriting the Dockerfile for a set of landing pages is not where the emergency budget goes. Moving to 7.4 is pointless, since 7.4 is EOL too. Moving to 8.3 is a real project, with mysql_*, ereg, the /e modifier, mcrypt, each() and PHP 4 constructors all in the way.

What changes for the application

Take a single php:5.6-fpm container fronting a dozen or so landing-page domains, with nginx next to it and fastcgi_pass php56:9000. Rebuilding that base on 5.6.40 plus the two patches leaves php -m and php-fpm -tt identical, and the only differences in php -i are session.upload_progress.enabled going to Off from the added ini file, the new max_multipart_body_parts entry, and whatever OpenSSL the final stretch update carried.

The behavioural changes are the same as on every EOL backport:

OOM terminates the worker through _exit(1), so shutdown functions do not run, the error goes to stderr, and nginx answers 502 where it used to answer 500.
Session upload progress is off in code, whatever php.ini says.
Multipart parts beyond the new limit are dropped with a warning.
Only Content-Disposition and Content-Type are retained on multipart parts.

For static landing pages none of that is visible. For anything with a large form or a shutdown handler doing real work, it is, and that is the thing to check before swapping the container rather than after.

Worth comparing old and new before deploying, since it is three commands:

1
2
3
4
5
6


for i in OLD NEW; do
 docker run --rm $i php -m > m-$i.txt
 docker run --rm $i php -i > i-$i.txt
 docker run --rm $i php-fpm -tt 2>&1 | sed -E 's/^\[[^]]+\] NOTICE: //' > f-$i.txt
done
diff m-OLD.txt m-NEW.txt; diff f-OLD.txt f-NEW.txt; diff i-OLD.txt i-NEW.txt

One thing worth doing in parallel, which is outside the scope of patching: a genuinely static landing page does not need PHP in front of it at all, and for the ones that do need it, an allowlist of the specific .php files is cheaper than keeping a 2019 interpreter correct forever.

Step by step

Confirm what you have: php -v inside the running container, and the image’s build date.
Fetch the current sury php5.6 source package and check the changelog revision. You want -105 or later, not -104.
Extract 0261 and 0263 from debian/patches/.
Build a base image from docker-library commit fab49d4cb1c6, variant 5.6/stretch/fpm, with the archive and keyserver edits, both patches, the marker assert, and the php.tar.xz repack.
Diff php -m, php -i and php-fpm -tt against the old image.
Verify: max_multipart_body_parts present in php -i, plus the OOM behaviour test showing exit 1 with no shutdown function.
Change the FROM line in the application image and add session.upload_progress.enabled = 0.
Keep the body-size limit at the proxy. It is the layer that does not depend on a 5.6 patch being correct.

Bottom line

“End of life” does not mean “not vulnerable”, and it does not mean “nothing to apply”. For 5.6 the fix comes from the co-author of the upstream patch, applies to the last release of the branch with no dependencies, and ships in a package whose changelog names the CVEs.

The thing to get right is the revision. 5.6.40-104 is the bailout fix with a documented hole in it; 5.6.40-105, one day later, fills the hole and, as a side effect, gives 5.6 the max_multipart_body_parts marker that makes a patched build identifiable at all. Without it the only proof is an OOM exiting 1 with shutdown functions skipped, which is worth wiring into the build either way.

A minimal backport closes the mechanism, not every input to it. Whatever it leaves out stays your perimeter’s problem, and the patch header is where you find out what that is.

Companion pieces: the fleet-wide audit and verification matrix, PHP 8 and the frozen Docker tags, and PHP 7.1 to 7.4.

Primary sources: GHSA-w8hw-w9fx-j79w, GHSA-6wmf-4rv5-p3rp, php-src at php-5.6.40, packages.sury.org, docker-library/php, and GHSA-54hq-v5wp-fqgv for CVE-2023-0662.

PHP2Shell on PHP 7: patch applied is not patch compiled is not patch working

BitPage — Mon, 26 Oct 2026 05:20:00 +0000

Short answer: PHP 7 has no fixed release and still has a fix for every branch: the official minimal backport 712e60f9edfd for 7.4 (17 files, +240/−59) and Ondřej Surý’s sury patch sets for 7.3, 7.2 and 7.1. The trap is that a clean patch --dry-run proves nothing: the backported fix uses ZSTR_MAX_LEN, a macro that does not exist in vanilla 7.3 or earlier, so the patch applies with offsets only and then the build dies with error: 'ZSTR_MAX_LEN' undeclared. Verify three separate things: that the patch applies, that it compiles, and that it works.

PHP 7.4 went end of life in 2022 and 7.1 long before that. There is no release coming, docker-library stopped building php:7.* images in November 2022, remi for EL6 froze in 2020, and moving an old Laravel or Yii2 project to 8.x is a project in its own right rather than a weekend. None of that means there is no patch.

The two CVEs and the fleet-wide verification matrix are covered in the overview. This one is the 7.x mechanics.

TL;DR

Fix, 7.4: the official branch PHP-7.4-CVE-2026-109618-109619, one commit on top of php-7.4.33. Apply it to 7.4.33, not to whatever you are running.
Fix, 7.3 / 7.2 / 7.1: the sury patch sets (0139, 0159, 0186 respectively) plus their prerequisites from the same debian/patches/series.
Trap: --dry-run checks patch text, not symbol dependencies. On 7.3 and older you also need the ZSTR_MAX_LEN backport (0107, 0135, 0167), or the compile fails.
Trap: the sury 7.1 package needed two revisions. 7.1.33-80 on 21 October carried the bailout fix; 7.1.33-81 the next day added the CVE-2023-0662 prerequisite. If you took -80 you are not finished.
Trap: patching your current patch version leaves CVE-2019-11043 open on 7.1 and 7.2, and the official patch does not even apply to 7.4.5.
Marker: max_multipart_body_parts appearing in php -i is a valid static proof on 7.1 through 7.4: absent from every vanilla 7.x tree, added by the fix.

Where to get a fixed PHP 7.x

Every branch has a source. They differ in who maintains them and in how much work lands on you.

Branch	Source of the fix	What comes with it
7.4	Official minimal backport, PHP Security Team: branch `PHP-7.4-CVE-2026-109618-109619`, commit `712e60f9edfd`	17 files, +240/−59: `Zend/*`, `main/rfc1867.c`, `main/SAPI.c`, `main/main.c`, `main/php_variables.c`, `ext/session/session.c`. `fpm` and `apache2handler` untouched
7.3	sury `php7.3 7.3.33-35`, patch `0139-Avoid-state-corruption-via-bailout`	needs `0107` (`ZSTR_MAX_LEN`) and `0079` (`max_multipart_body_parts`)
7.2	sury `php7.2 7.2.34-66`, patch `0159`	needs `0135` (`ZSTR_MAX_LEN`) and `0105` (`max_multipart_body_parts`)
7.1	sury `php7.1 7.1.33-81`, patch `0186`	needs `0167` (`ZSTR_MAX_LEN`) and `0188` (`max_multipart_body_parts`)

If you intend to assert those versions in a config-management check, read the Version: field from the .dsc rather than the filename, because Debian strips the epoch out of filenames and the epochs here are not uniform:

Source package	Full version
`php7.1`	`7.1.33-81+0~20261022.128+debian12~1.gbpa88ccf`
`php7.2`	`7.2.34-66+0~20261021.122+debian12~1.gbpe0bcd4`
`php7.3`	`7.3.33-35+0~20261021.144+debian12~1.gbpecf76f`
`php7.4`	`1:7.4.33-31+0~20261021.121+debian12~1.gbpe9c428`
`php8.0`	`1:8.0.30-25+0~20261021.88+debian12~1.gbpabebea`

The 1: epoch is on 7.4 and 8.0 and on neither of the three older branches. A check written as a string comparison against 7.4.33-31 therefore fails on exactly the branch where the official fix exists, which is a satisfying way to waste an hour. Compare on the upstream-plus-revision part and let the distro suffix vary.

The sury patches say where they came from. 0139 for 7.3 is headed “Backported from 8.2.35”; the series for the older branches is attributed to Arnaud Le Blanc, co-author of the upstream fix, and the 5.6 version of the same patch describes itself as a “Minimal mitigation for EOL version.” Those are the patches that go into the Debian PHP packages, with the CVE numbers written into debian/changelog, which is why I chose them over the alternatives.

The alternatives, for completeness. A community port covering 5.4 to 7.3 touches 8 files against the official 17, with zend_hash.c, php_variables.c and the VM left alone, and its authors tested it against the built-in CLI server; I rejected it. An independent MIT-licensed port for 7.3 with ready-made images on ghcr.io looked more serious but still less authoritative than the Debian maintainer’s. Paid routes exist and work. TuxCare backported the fix to every alt-php branch from 5.2 to 8.5, and their own FAQ states the shape of it plainly: “alt-php74 stays at 7.4.33 and changes from release -79 to -81 on EL7.” That is the same version-number-does-not-move problem as the sury packages, from a different vendor, and it is why their documented verification route is a CVE tracker and the RPM release number rather than php -v. Freexian ELTS covers php7.4 on Debian 11.

Before paying anyone, check whether your distro already did the work. Ubuntu’s CVE record lists every 7.x branch it ships as released, with the upstream version untouched in each case: php7.0 on xenial at 7.0.33-0ubuntu0.16.04.16+esm22, php7.2 on bionic at 7.2.24-0ubuntu0.18.04.17+esm14, php7.4 on focal at 7.4.3-4ubuntu2.29+esm5. If your PHP came from the distro rather than a PPA, the fix may be an apt upgrade away under a subscription you already have.

One gap worth naming because it has no answer: sury php7.4 on Ubuntu 18.04. The ondrej PPA dropped bionic in 2023, and Ubuntu’s bionic fix is for the distro php7.2, not for a PPA-installed php7.4. A PPA package is not covered by ESM no matter what the subscription says. If you have that combination, the fix is to change something structural: the OS, the branch, or the exposure.

A side benefit of the packages over hand-patching the official commit: the sury EOL revisions carry much more than PHP2Shell. The 7.2.34-66 and 7.3.33-35 changelog entries lead with libgd, an SQL injection through E'...' backslash breakout and a recursive-symlink crash, all backported from 8.2.33; 7.4.33-31 leads with an IPv6 ACL bypass in FastCGI listen.allowed_clients and mysqlnd packet overreads. Across each of those changelogs there are between forty and sixty CVE lines. Hand-applying one commit closes one bug; taking the package closes the backlog.

Patch the last patch release of the branch

Two reasons, and the second one is the one people skip.

The patch often does not apply to older patch versions. The official 7.4 commit goes onto 7.4.33 and 7.4.29 cleanly and conflicts on 7.4.5 in zend_vm_*. Applied to 7.3.33 it leaves rejected hunks in six files; to 7.2.34 it rejects 46 hunks and the whole VM; on 7.1 the same.

Where it does apply, it can leave an older RCE untouched. The sury 0186 patch applies cleanly to 7.1.8, 7.1.11 and 7.1.17, every one of which predates the fix for CVE-2019-11043, an unauthenticated RCE in php-fpm behind nginx with fastcgi_split_path_info, fixed in 7.1.33, 7.2.24 and 7.3.11. Closing PHP2Shell on 7.1.8 and leaving that open is not a fix, it is a reshuffle. The only safe route is the last release of the branch plus the patch.

Trap 1: the dry run is clean and the build fails

Take 7.3.33 from a vanilla tarball with the sury pair 0079 and 0139. Both apply with line offsets only, no fuzz, exit status zero. The compile then stops:

zend_vm_execute.h:4971: error: 'ZSTR_MAX_LEN' undeclared

--dry-run checks whether the patch text matches the file, and nothing else. It has no opinion about whether the identifiers in the patch exist. The fix was backported from 8.2.35 and uses a macro that 7.3 never had. In the sury package it arrives as a separate, earlier patch: 0107-backport-ZSTR_MAX_LEN.patch, by Remi Collet, dated 22 November 2024, three lines in Zend/zend_string.h:

1
2


+#define ZSTR_MAX_OVERHEAD (ZEND_MM_ALIGNED_SIZE(_ZSTR_HEADER_SIZE + 1))
+#define ZSTR_MAX_LEN (SIZE_MAX - ZSTR_MAX_OVERHEAD)

I pinned the boundary so you do not have to guess which branches need it. ZSTR_MAX_LEN is present in vanilla Zend/zend_string.h on PHP 7.4 and 8.0 and absent on 7.3 and everything older. The 7.3 bailout patch references it 38 times and never touches zend_string.h itself. That is exactly the shape of failure that passes every pre-flight check and dies in the compiler: the text matches, the symbol does not exist.

Which is why 7.4 needs no prerequisite patch and 7.3, 7.2 and 7.1 all do: 0107, 0135 and 0167. The 7.x notes I started from had this confirmed only for 7.3, with 7.2 and 7.1 inferred from the same code; the vanilla boundary closes that inference.

The general rule is cheap to apply. Before building, read the patch for identifiers it uses but does not define, and find where they come from:

1
2
3


curl -sSfO https://packages.sury.org/php/pool/main/p/php7.3/<php7.3_7.3.33-35…debian.tar.xz>
tar -xJf php7.3_*.debian.tar.xz
grep -l -E 'state corruption via bailout|max_multipart_body_parts|define ZSTR_MAX_LEN' debian/patches/*.patch

There is a shortcut hidden in the numbering, and it also tells you something the changelog does not. Compare the prerequisite’s patch number to the bailout patch’s number. A lower number means the prerequisite was already in the package before PHP2Shell and comes along for free. A higher number means it was added after the fix, in a later revision — which means the first revision shipped incomplete.

Branch	Bailout patch	`max_multipart_body_parts` patch	Already present?
7.4	0164	0062	yes
7.3	0139	0079	yes
7.2	0159	0105	yes
7.1	0186	0188	no, added later

Trap 2: the 7.1 package needed a second revision the next day

The numbering above is not trivia. I read the debian/changelog out of the published source packages, and 7.1 has two revisions where the others have one:

php7.1 (7.1.33-80), 21 October 2026 — the bailout fix, CVE-2026-109618 and CVE-2026-109619.
php7.1 (7.1.33-81), 22 October 2026 — “Backported from 8.0.28: [CVE-2023-0662]: DOS vulnerability when parsing multipart request body.”

CVE-2023-0662 is the multipart-parts limit. It was fixed upstream in 8.0.28, which means 7.1 never received it in its supported life, which means the memory-amplification input that PHP2Shell depends on was unlimited on that branch until the second revision. For 7.2, 7.3 and 7.4 the limit had already been backported years earlier, so one revision was enough.

If you patched 7.1 within the first day, you took -80 and you are missing the prerequisite. The same pattern applies to 5.6, which I cover separately. The operational lesson is not about these two packages. It is that for EOL branches the fix can arrive in pieces over several days, and “I patched on day one” is a reason to check again rather than a reason to stop.

Trap 3: the build assert that asserts nothing

Put a marker check in the build so it fails loudly when the patch stops applying. Then make sure the marker is real.

Mine was not. I used grep -q zend_objects_store_mark_destructed after the patch step, the build passed, and I believed it. That symbol exists in unpatched sources too, so the assert passed unconditionally. It was a comment that happened to compile. I counted it afterwards across all six EOL patch sets: zero occurrences in any of them. It was never going to work.

The test for a candidate marker is one command: it must occur zero times in the vanilla tree and at least once after the patch.

1

grep -c array_init_singleton Zend/zend_API.h # 0 before, >=1 after

array_init_singleton is the one I would standardise on. It occurs zero times in vanilla 5.6.40, 7.4.33 and 8.0.30, and every one of the six EOL patch sets adds it, so a single marker covers 5.6 to 8.0 instead of a different grep per branch. zend_new_array_singleton in Zend/zend_hash.h also works, but only on 7.3 and newer, so it needs a per-branch table. Do not reach for zend_mm_safe_error_ex, which the EOL backports add and the supported branches do not have at all — useful for telling an EOL backport apart from upstream, useless as a general patch marker.

Trap 4: the patch is in and the version and ini say otherwise

php -v reports 7.4.33, 7.3.33, 7.2.34 or 7.1.33 identically with and without the patch, and php -i still prints session.upload_progress.enabled => On even though the patch disables the feature in code. A scanner reading version plus ini will call a patched build vulnerable, and an ini showing Off does not prove a patch is present.

On 7.x there is a static marker, and this is the branch range where it works. max_multipart_body_parts is absent from every vanilla PHP 7 tree and added by the fix. I counted the directive in main/main.c:

Vanilla tag	`max_multipart_body_parts`
php-5.6.40	0
php-7.1.33	0
php-7.2.34	0
php-7.3.33	0
php-7.4.33	0
php-8.0.28	1
php-8.0.30	1
php-8.1.34	1

That makes php -i | grep max_multipart_body_parts a usable answer on 7.1 through 7.4, and useless from 8.0 on, where the directive arrived in 8.0.28 independently of PHP2Shell.

The reliable check on any EOL branch is behavioural. The backports replace the bailout with a hard exit, which you can see from userland:

1
2
3
4


T='register_shutdown_function(function(){echo "SHUTDOWN-CALLED\n";}); $a=str_repeat("x", 20*1024*1024);'
php -d memory_limit=8M -r "$T"; echo "exit=$?"
# patched: Fatal error ..., exit=1, no SHUTDOWN-CALLED
# unpatched: Fatal error ..., exit=255, SHUTDOWN-CALLED

I confirmed that signature independently against a sury php8.0-cli 1:8.0.30-25 build: exit 1, the fatal error on stderr rather than stdout, shutdown function skipped. One warning carried over from the overview: this test does not work on supported branches. On 8.2 the vulnerable 8.2.29 and the fixed 8.2.35 both exit 255 and both run shutdown functions, because the upstream fix leaves ordinary memory_limit handling alone. It is an EOL test.

Docker: your own base image, from the last official Dockerfile

For the Docker installs the recipe is the same on every branch: take the last docker-library Dockerfile that still had the branch, add the patches, repack the source tarball.

Finding the right commit matters more than it sounds, because you want the Dockerfile as it was, not a reconstruction. Ask git for the file’s own history rather than eyeballing the log (git log --oneline -- 7.3/bullseye/fpm/Dockerfile), then take the commit directly below the removal:

Branch	Last commit with the branch	`PHP_VERSION`	Removed in
7.4	`b9f17156020c` “Update 7.4 to 7.4.33”	7.4.33	`f2999de80bd8` “Remove PHP 7.4 (EOL 28 November 2022)”
7.3	`b62fa33a7198` “Use gnu-libiconv for php iconv extension on alpine”	7.3.33	`f2754514dafb` “Remove 7.3 (#1270)”
7.2	`3dc9a6988d47` “Add "–with-pic" to configure flags”	7.2.34	`4908ef2763a9` “Remove 7.2 (EOL)”

The variants survive as you would hope: 7.4/bullseye/fpm and 7.3/bullseye/fpm plus 7.3/alpine3.15/fpm, and for 7.2 the OS is one generation back: 7.2/buster/fpm and 7.2/alpine3.12/fpm. Note that none of those three last commits is a version bump except on 7.4; on 7.3 and 7.2 the final touch was an unrelated build-flag change, which is exactly why guessing from the log’s subject lines gets you the wrong tree.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


FROM debian:bullseye-slim
RUN sed -i 's#http://deb.debian.org/debian-security#http://archive.debian.org/debian-security#' /etc/apt/sources.list \
 && grep -q 'archive.debian.org/debian-security bullseye-security' /etc/apt/sources.list
# ... unchanged; GPG_KEYS imported from keyserver.ubuntu.com ...
COPY php2shell-7.3-0079-max_multipart_body_parts.patch \
 php2shell-7.3-0107-zstr_max_len.patch \
 php2shell-7.3-0139-avoid-state-corruption.patch /usr/src/
RUN set -eux; \
 apt-get update; apt-get install -y --no-install-recommends patch <build-deps>; \
 docker-php-source extract; \
 cd /usr/src/php; \
 for p in 0079-max_multipart_body_parts 0107-zstr_max_len 0139-avoid-state-corruption; do \
 patch -p1 --forward < /usr/src/php2shell-7.3-$p.patch; done; \
 grep -q array_init_singleton Zend/zend_API.h; \
# repack with the patch in: docker-php-ext-install re-extracts php.tar.xz later
 tar -cJf /usr/src/php.tar.xz -C /usr/src php; \
 ./configure ... # as in the original

Because the layout stays identical to the official image (/usr/local/etc/php, docker-php-ext-install, pecl, zz-docker.conf), the application Dockerfile changes one line, and one base image per branch serves every project on that branch.

1
2


FROM php73-p2s/php-base:7.3.33-p2s-fpm-bullseye
COPY zz-php2shell.ini /usr/local/etc/php/conf.d/zz-php2shell.ini # session.upload_progress.enabled = 0

Keep the OS family you already had. An Alpine project stays on Alpine, because moving to Debian changes musl to glibc and swaps the ImageMagick build underneath an imagick extension that was built against the old one. A Debian project moving buster to bullseye is fine and sometimes necessary, since buster is entirely on archive.debian.org. For anything integrating with a payment gateway that still negotiates legacy ciphers, bullseye is also the last stop with OpenSSL 1.1.1 — going to bookworm means OpenSSL 3 and that handshake needs testing before, not after.

The zz-php2shell.ini is redundant with the patch, which disables upload progress in code. Set it anyway, because it makes the state visible to whoever audits the box next.

For 7.4, the whole patch step collapses to a single git format-patch 712e60f9edfd, and a successful run prints 17 patching file … lines with no offsets and no fuzz.

CentOS 6: your own RPM, and nothing to roll back to

For hosts where PHP 7.1 or 7.2 came from webtatic or remi on CentOS 6, the equivalent move is a rebuilt RPM from the same vendor’s spec, so dependencies and layout stay as the installed packages expect.

7.1 starts from the webtatic SRPM php71w-7.1.33-1.w6.src.rpm, still on repo.webtatic.com/yum/el6/SRPMS/ and verifiable against the webtatic key. Add the sury patches as Patch400–Patch402, bump Release: 1.1 so the result is newer than what is installed. Build inside centos:6 pointed at vault.centos.org and an EPEL 6 archive, deliberately without remi, so dependencies come only from base and EPEL exactly as the original did.
7.2 starts from remi’s php72.spec at the commit remi built 7.2.34-1.el6.remi from, plus the sury 7.2 patches and the same Release bump.

Apply the patches unconditionally, outside any %if block, with --fuzz=0, and put the marker grep in %prep so the build fails rather than silently producing an unpatched RPM:

1
2
3
4
5
6
7
8
9


Release: 1.1%{?dist}
Patch400: php2shell-7.1-0167-zstr_max_len.patch
Patch401: php2shell-7.1-0186-avoid-state-corruption.patch
Patch402: php2shell-7.1-0188-max_multipart_body_parts.patch
# in %prep, after the vendor patches:
%patch400 -p1 --fuzz=0
%patch401 -p1 --fuzz=0
%patch402 -p1 --fuzz=0
grep -q array_init_singleton Zend/zend_API.h

Four things that cost time in that environment, none of them PHP’s fault:

There is nothing to roll back to. The SRPMs and RPMs for the older 7.1 patch versions are gone from webtatic. Before rpm -Uvh, archive every file the installed packages own and every config, because yum downgrade has nowhere to go:

1
2
3
4
5
6


B=/root/php-backup; mkdir -p $B
tar -czf $B/etc.tar.gz /etc/php.ini /etc/php.d /etc/php-fpm.conf /etc/php-fpm.d
rpm -qa 'php71w*' | sort > $B/packages.txt
rpm -ql $(rpm -qa 'php71w*') | while read f; do [ -f "$f" ] || [ -L "$f" ] && echo "$f"; done \
 | tar -czf $B/files.tar.gz -T -
rpm -Uvh --test *.rpm && rpm -Uvh *.rpm

Restoring means unpacking those archives; the rpm database stays on the new version, which is ugly but recoverable.

PHP 7.2’s buildconf wants autoconf 2.64 or newer and EL6 ships 2.63. Install autoconf268 from EPEL and build with PHP_AUTOCONF=autoconf268 PHP_AUTOHEADER=autoheader268.
The oniguruma5php dependency (libonig.so.105) is missing on older remi installs; take it from the remi el6 archive.
PHP installed outside the rpm database happens on long-lived stages, and then rpm -Uvh needs --nodeps.

One pleasant surprise: rpm -Uvh runs the condrestart scriptlet itself, so php-fpm comes back without a separate restart and the gap is a second or two.

Traps in the old build environment

Reviving a Dockerfile from 2019 or 2020 breaks on infrastructure, not on code:

apt-get update 404s, because buster and bullseye-security moved to archive.debian.org. Rewrite sources.list and follow it with a grep -q assert so a future format change fails the build instead of quietly using the wrong mirror.
The tarball signing key import fails, because ha.pool.sks-keyservers.net shut down. keyserver.ubuntu.com works and the signature is still verified — do not drop the verification step, change the keyserver.
Alpine 3.12 and 3.15 packages are still on the mirrors, so those builds needed no repository edits at all.

Keep the edits confined to source delivery. The PHP build itself and every configure flag stay exactly as the official image had them, which is the whole reason for starting from the official Dockerfile.

What changes for the application

The behavioural changes are identical across all the EOL ports, because they are the same patch:

OOM terminates the worker through _exit(1). Shutdown functions do not run, the message goes to stderr, and nginx returns 502 where it returned 500. An application that regularly brushes memory_limit will start serving 502s, and monitoring that matches exit code 255 will not match 1.
Session upload progress is off in code, whatever php.ini reports.
max_multipart_body_parts appears, defaulting to -1, which falls back to max_input_vars + max_file_uploads, or 1020 parts on stock defaults. Past that, PHP emits “Multipart body parts limit exceeded %d. To increase the limit change max_multipart_body_parts in php.ini.” Forms with thousands of fields will notice.
Only Content-Disposition and Content-Type are retained on multipart parts.

What I did not test is whether the exploit worked before and stopped working after. No PoC was run. The claim here is that the official and maintainer patches are applied, compiled and behaving as their diffs say, which is a different and weaker claim than “verified not exploitable”. It is worth being precise about that.

Two more things I am not claiming. Whether memory_limit = -1 under OpenVZ runs into beancounters the way it runs into RLIMIT_AS is a guess — the shape of the failure is the same, but I have not tested it. And the docker-php-ext-install repack is a precaution; I never saw it cause a real failure.

Step by step

Identify the exact patch version and SAPI of every 7.x install. Not the branch, the patch version — the route depends on it.
Pick the source per branch: official commit for 7.4, sury patch set for 7.3, 7.2 and 7.1.
Read debian/patches/series and take the prerequisites, not just the bailout patch. Check whether the prerequisite’s number is higher than the bailout patch’s; if it is, make sure you have the later package revision.
Target the last patch release of the branch. Check what other CVEs you are leaving behind if you do not.
Build from the last docker-library Dockerfile for the branch, or the vendor’s RPM spec. Repack php.tar.xz after patching.
Put a marker assert in the build that returns zero on a vanilla tree.
Verify the result three ways: max_multipart_body_parts in php -i, the marker symbol in the source tree, and the OOM behaviour test.
On CentOS 6, archive the installed files before rpm -Uvh. There is no package to go back to.

Bottom line

For an EOL branch, “no release” is not “no fix”. The official backport for 7.4 and the sury patch sets for 7.1 through 7.3, applied on top of the last docker-library Dockerfile or the vendor’s RPM spec, produce a fixed PHP on the same branch. The application changes one FROM line or installs packages one Release newer.

The part that actually goes wrong is verification, and it has three independent layers. The patch’s own dependencies: --dry-run is clean on 7.3 and the build dies on ZSTR_MAX_LEN, which exists in 7.4 and 8.0 and nowhere below. The marker in the tree: it has to be a symbol the patch introduces, which zend_objects_store_mark_destructed is not and array_init_singleton is. And behaviour: exit code 1 with shutdown functions skipped, which is the signature of an EOL backport and not of a fixed supported branch.

Then patch the last release of the branch rather than the version you are on, because on 7.1 and 7.2 there is a second unauthenticated RCE underneath this one, and on 7.4.5 the official patch simply does not apply.

Companion pieces: the fleet-wide audit and verification matrix, PHP 8 and the frozen Docker tags, and PHP 5.6.

Primary sources: GHSA-w8hw-w9fx-j79w, GHSA-6wmf-4rv5-p3rp, php-src, packages.sury.org, docker-library/php, php.net ChangeLog-7.

PHP2Shell on PHP 8: docker build --pull succeeded and PHP is still 8.2.29

BitPage — Mon, 26 Oct 2026 05:15:00 +0000

Short answer: docker build --pull honestly downloads the newest image under your tag, which tells you nothing, because the tag may not have been rebuilt in over a year. php:8.2-fpm-bullseye has carried PHP 8.2.29 since 5 August 2025 and never received the PHP2Shell fix; the fixed 8.2.35 exists only under -bookworm and -trixie. Check PHP_VERSION in the base image config before you build, not the build log afterwards.

When PHP2Shell landed I expected the 8.x half of the work to be the easy half. Bump the tag, recreate the container, move on to the EOL branches where the actual problem lives. In several places the rebuild went green and the PHP inside was still vulnerable, and the check everybody runs (reading the build output) did not show it.

The background on the two CVEs and the fleet-wide audit is in the overview piece. This one is only about PHP 8 and Docker.

TL;DR

Fix: pin the OS suffix explicitly (php:8.2-fpm-bookworm), build with --pull --platform linux/amd64, and verify PHP_VERSION in the base image before the application build.
Fix, supported branches: php -r 'var_dump(ini_get("memory_hard_limit_percent") !== false);' returns true only on 8.2.35 / 8.3.36 / 8.4.27 / 8.5.12 and later. It is the one static check that survives a distro backport. The !== false is load-bearing: the directive defaults to NULL, so ini_get() returns an empty string and a truthiness test calls a patched build unpatched.
Trap: minor tags of third-party images freeze too. dunglas/frankenphp:1.8-php8.4-bookworm has held PHP 8.4.10 since 14 July 2025; 8.4.27 ships only in FrankenPHP 1.13.x.
Trap: wordpress:latest has moved to PHP 8.3.36. A blind pull changes your PHP minor version without saying so.
Trap: EOL 8.0 has no fixed release at all. The only path is 8.0.30 plus the official backport commit, which means 23 patch versions of forward movement before you even start.
Trap: the OOM behaviour test does not prove patching on 8.2. I measured 8.2.29 and 8.2.35 behaving identically.

Why the same tag with a fresh pull is not a fix

An official image is PHP plus an OS plus the docker-library tag policy, and the fixed PHP is not under every tag where you would expect it. I went through the registry API for every tag I had in a Dockerfile. Five distinct failure modes, all measured on 26 October 2026:

Tag	PHP inside	Tag last pushed	What it does to you
`php:8.2-fpm-bullseye`	8.2.29	2025-08-05	frozen; `--pull` fetches a vulnerable image and succeeds
`php:8.2-fpm-bookworm`	8.2.35	2026-10-21	fixed
`php:8.2-fpm`	8.2.35	2026-10-24	fixed, but same config digest as `-trixie`; the OS moved under you
`php:8.4-fpm-bookworm`	8.4.27	2026-10-21	fixed
`php:8.0-fpm`	8.0.30	2023-11-22	frozen; no fixed release exists for the branch
`php:8.1-cli-alpine`	8.1.34	2025-12-22	frozen; branch is EOL, no fixed release
`wordpress:latest`	8.3.36	2026-10-22	fixed, but the PHP minor version changed
`wordpress:php8.2-apache`	8.2.35	2026-10-22	fixed, branch pinned
`dunglas/frankenphp:1.8-php8.4-bookworm`	8.4.10	2025-07-14	frozen minor tag
`dunglas/frankenphp:1.13.2-php8.4.27-bookworm`	8.4.27	2026-10-21	fixed

One caveat on that third column, because I got it wrong myself at first and the two numbers are a few days apart. An image has two timestamps: created in its config blob, which is when it was built, and the tag’s own last push, which is when that name last moved. For php:8.2-fpm-bullseye they are 2025-07-31 and 2025-08-05. The column above is the push date, because the question being asked is “when did this tag last change”, not “when was this layer compiled”. If you script this, pick one field and label it; mixing them produces a table that nobody including you can reconcile later.

Reading down that list, the modes are:

Tags on older OS releases are frozen. -bullseye stopped at 8.2.29. The branch is still supported, the tag is not still built. This is the one that cost me time, because every signal except PHP_VERSION looks healthy: the pull succeeds, the layers for the application are new, the build date on the final image is today.
The suffix-less tag changed OS. php:8.2-fpm now resolves to the same image config digest as php:8.2-fpm-trixie (Debian 13), so the two names are the same image under different labels. Dockerfiles that call apt-key stop building, and OpenSSL, ICU and libpq all move at once.
Minor tags of third-party images freeze the same way. FrankenPHP’s 1.8-php8.4-bookworm is a tag that looks floating and has not moved since July 2025.
A latest tag can change the PHP branch. An image built when wordpress:latest carried 8.2 now gets 8.3.36 from the identical Dockerfile line.
Distribution packages backport without a version bump. For a PPA or distro install, php -v cannot answer the question at all; the package changelog can.

Alpine is not an escape hatch. Alpine minor tags freeze exactly like Debian ones: php:8.2-cli-alpine3.20 carries 8.2.28, alpine3.21 carries 8.2.29, alpine3.22 carries 8.2.31, and only the bare -alpine suffix carries 8.2.35.

The check that would have caught all of it takes one command and belongs before the application build, not after:

1
2
3
4


docker pull php:8.2-fpm-bullseye
docker image inspect php:8.2-fpm-bullseye \
 --format '{{range .Config.Env}}{{println .}}{{end}}' | grep PHP_VERSION
docker image inspect -f '{{.Created}}' php:8.2-fpm-bullseye

That .Created is the build date from the config blob, which is the only one of the two timestamps a local Docker can show you; the push date needs the registry API. For this purpose the build date is enough. If it is more than a few weeks old on a supported branch, the tag is not being rebuilt and no amount of --pull will change it.

Supported branches: what to actually pin

Take a setup with a Laravel app on php:8.2-fpm-bookworm, a couple of small services still on -bullseye, a WordPress site on wordpress:latest, an Octane service on the FrankenPHP 1.8 minor tag, and a CLI install from the ondrej PPA. Each needs a different change and only one of them is a version bump.

Move -bullseye to -bookworm. That is Debian 11 to 12, which moves more than PHP: OpenSSL 1.1.1w to 3.0.22, ICU 67 to 72, libpq 13 to 15, curl 7.74 to 7.88. OpenSSL 3 is the one that bites, because it drops legacy ciphers. Run a TLS handshake against every external API you talk to from both the old and the new image before deploying, not after.
Never use the suffix-less tag. php:8.2-fpm is a moving OS target by design.
WordPress: wordpress:php8.2-apache, not latest. Then only the PHP patch version and the OS move, and going to 8.3 becomes a decision rather than a side effect. Worth knowing separately: changing the WordPress image does not update the WordPress core in the volume, because the entrypoint only copies core into an empty directory.
FrankenPHP: pin both versions and the digest, as in dunglas/frankenphp:1.13.2-php8.4.27-bookworm@sha256:…. Going from 1.8 to 1.13 is a real upgrade of the server, not just of PHP: the header limit becomes 16 KiB, putenv() no longer writes to the process environment, and SCRIPT_NAME and PATH_INFO are set differently. Read the FrankenPHP release notes before the bump, not during the incident.
PPA or distro package: apt install --only-upgrade 'php8.2-*', then confirm from the changelog rather than the version. First run dpkg -l 'php8.2*' and check whether an fpm, cgi or mod_php package is installed that you had forgotten about. A CLI-only box that quietly has libapache2-mod-php8.2 on it is a web-facing install.
Add session.upload_progress.enabled = 0 to the image’s ini directory regardless of branch. It closes the confirmed FPM path for CVE-2026-109619 and breaks nothing except upload progress bars. It has to be in php.ini or the pool config: ini_set() and .user.ini are applied after the body is parsed, which is too late by construction.

Always build with --platform linux/amd64 if you build on an Apple Silicon laptop. An arm64 image that cannot start on the target is at least an obvious failure; the subtler version is a successful multi-arch build where you test the wrong architecture locally.

I rejected memory_limit = -1 as a standalone mitigation here. The advisory lists it for 109618, but on a pool with a few hundred children and no container memory cap it trades a high-complexity RCE for a reliable OOM of the whole VM. RLIMIT_AS and ulimit -v are worse: they trigger the vulnerable allocation failure themselves.

EOL 8.0: build your own base image

There is no fixed 8.0 release, no tarball and no Docker image. What exists is a branch in php-src: PHP-8.0-CVE-2026-109618-109619, which is the php-8.0.30 tag plus a single commit, 82f8f31b16a0, titled “Avoid state corruption via bailout” and described in its own message as a minimal mitigation for an EOL version. It touches 17 files for +229/−59: Zend/*, main/rfc1867.c, main/SAPI.c, main/main.c, main/php_variables.c and ext/session/session.c. The fpm and apache2handler SAPI directories are untouched.

The patch applies cleanly to 8.0.30 and conflicts on 8.0.7 in zend_vm_*. That settles the route: there is exactly one, and it is 8.0.30 plus the patch. For an application pinned to 8.0.7 from a 2021 image that is 23 patch versions of forward movement, which is work you have to scope honestly rather than wish away. The 8.0 NEWS entries between those two versions that actually change behaviour: 8.0.28 introduced max_multipart_body_parts and fixed password_verify; 8.0.24 and 8.0.27 stopped url-decoding cookie names; 8.0.30 stopped libxml loading external entities; 8.0.8 made FILTER_VALIDATE_URL stricter.

The base image comes from the last docker-library Dockerfile that still had the branch. For 8.0 that is commit 41c9ad0b43a5, variant 8.0/bullseye/fpm, the last commit to touch that file before 4d3ed4a9bcb0 “Remove PHP 8.0 (EOL 26 November 2023)”. It carries ENV PHP_VERSION 8.0.30 on line 63, which is the version the official patch wants anyway. Using the official Dockerfile rather than writing your own matters more than it looks: the layout stays identical, so /usr/local/etc/php, docker-php-ext-install and pecl all keep working, and the application Dockerfile changes one FROM line.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


FROM debian:bullseye-slim
# bullseye-security moved to archive.debian.org when Debian 11 left LTS
RUN sed -i 's#http://deb.debian.org/debian-security#http://archive.debian.org/debian-security#' /etc/apt/sources.list \
 && grep -q 'archive.debian.org/debian-security bullseye-security' /etc/apt/sources.list
# ... unchanged up to the PHP build ...
COPY php2shell-8.0.patch /usr/src/php2shell-8.0.patch # git format-patch 82f8f31b16a0
RUN set -eux; \
 apt-get update; apt-get install -y --no-install-recommends patch <build-deps>; \
 docker-php-source extract; \
 cd /usr/src/php; \
 patch -p1 --forward < /usr/src/php2shell-8.0.patch; \
 grep -c array_init_singleton Zend/zend_API.h; \
# repack the patched tree: docker-php-ext-install re-extracts php.tar.xz later
 tar -cJf /usr/src/php.tar.xz -C /usr/src php; \
 ./configure ... # as in the original

The grep -q after patch is an assert, and choosing the right symbol matters. My first attempt used zend_objects_store_mark_destructed, which exists in unpatched source as well, so the assert passed unconditionally and proved nothing. array_init_singleton in Zend/zend_API.h occurs zero times in vanilla 8.0.30 and is added by the patch, which is what an assert needs.

I rejected the sury php8.0 packages on debian:bookworm-slim for this one. The fix is there, in 0137-Avoid-state-corruption-via-bailout.patch, and the package also carries dozens of post-EOL CVE fixes the official backport does not, but the layout is different (/etc/php/8.0, php-fpm8.0, extensions via apt), so the application Dockerfile would have to be rewritten. For a fleet-wide emergency, one changed FROM line beats a better-maintained package.

Gotchas

The patch is in and the version says otherwise

php -v reports 8.0.30 with and without the patch. Worse, the obvious marker does not work on this branch: max_multipart_body_parts exists in 8.0.28 and later regardless of the patch — I counted it in vanilla main/main.c for php-8.0.28 and php-8.0.30, one occurrence each. And php -i still prints session.upload_progress.enabled => On even though the patch disables the feature in code with if (!PS(rfc1867_enabled) || true).

On 8.0, then, a version-and-ini scanner will call your patched image vulnerable, and an ini showing Off does not prove a patch is present. The proof is behavioural:

1
2
3
4


T='register_shutdown_function(function(){echo "SHUTDOWN-CALLED\n";}); $a=str_repeat("x", 20*1024*1024);'
docker run --rm IMG php -d memory_limit=8M -r "$T"; echo "exit=$?"
# patched: Fatal error ..., exit=1, no SHUTDOWN-CALLED
# unpatched: Fatal error ..., exit=255, SHUTDOWN-CALLED

Then tag the result so nobody has to rerun the test to identify the build: an image tag like 8.0.30-p2s-… plus a LABEL, and the behaviour test kept in the build checklist.

That same test proves nothing on 8.2

This is the correction I had to make to my own notes. I assumed the exit-code-1 signature was what “patched” looked like everywhere. It is not. I ran the identical script in a 512 MB container with memory_limit=64M against three builds:

Build	Exit	Fatal error to	Shutdown function
`php:8.2-cli-alpine3.21`, 8.2.29 (vulnerable)	255	stdout	ran
`php:8.2-cli-alpine`, 8.2.35 (fixed)	255	stdout	ran
sury `php8.0-cli` `1:8.0.30-25` (EOL backport)	1	stderr	did not run

The upstream fix for supported branches deliberately leaves ordinary memory_limit handling alone — the soft OOM is deferred to a safe point rather than turned into a process exit. Only the EOL backports convert it into _exit(1). Use the behaviour test on EOL branches and the new memory_hard_limit_percent directive on supported ones; swapping them gives you a confident wrong answer in both directions.

The image is updated and the running PHP is not

This one is delayed, which makes it worse. A new image in the registry is not a new PHP in production.

The stage deployment role in one place ran docker compose up -d --no-recreate with no pull, so the stages would never have picked the new image up on their own. I found that by reading the role before the rollout rather than after, which was luck as much as process. CI runners are the same shape of problem: a runner pulls php-cli:8.2 when its next job starts, so until then it is still running the vulnerable image, and your report needs a line that says “fixed later” rather than “fixed”.

1
2
3


docker inspect -f '{{.Image}}' <container> # must equal the new image ID
docker image inspect <image:tag> -f '{{.Id}}'
docker exec <container> php -r 'echo PHP_VERSION, " up=", ini_get("session.upload_progress.enabled"), PHP_EOL;'

`docker-php-ext-install` rebuilds from unpatched sources

The official Dockerfile ends with docker-php-source delete, leaving /usr/src/php.tar.xz in place. Later, an application image running docker-php-ext-install re-extracts that tarball (the pristine one, without your patch) and compiles extensions against it. Repacking the patched tree into the same php.tar.xz right after patch costs one line and removes the question.

I have not seen this cause a real failure, so treat it as a precaution rather than a reported bug. It is cheap enough that the distinction does not matter.

A 2021 Dockerfile no longer builds, and without a build there is no patch

Two separate breakages, both unrelated to PHP:

Debian 11 left LTS in August 2026. bullseye-security moved to archive.debian.org and returns 404 on deb.debian.org. Rewrite sources.list and put a grep -q after it so the build fails loudly if the file format changes under you rather than silently falling back.
The pecl.php.net REST channel no longer serves old releases for PHP 8.0, so pecl install redis fails. Download the exact version you were already running (redis-5.3.4.tgz in my case) and verify the sha256 instead.

Neither is interesting. Both are the reason an emergency rebuild of a five-year-old image takes a day instead of an hour, and both are worth discovering before the day you need the patch.

What changes for the application

On the EOL backport, three behavioural changes, all visible from the application side:

OOM terminates through _exit(1). Shutdown functions do not run, the fatal error goes to stderr, and FPM behind nginx answers 502 where it answered 500. An application that regularly brushes memory_limit will start serving 502s, and anything you do in a shutdown handler (flushing an error reporter, releasing a queue job) stops happening.
Session upload progress is off in code, whatever php.ini says.
max_multipart_body_parts caps form size. Default -1 falls back to max_input_vars + max_file_uploads; on stock defaults that is 1020 parts, after which PHP emits “Multipart body parts limit exceeded %d. To increase the limit change max_multipart_body_parts in php.ini.” Forms with thousands of fields will hit it.

On the supported branches the picture is different, and I checked this in the source because the advisory does not spell it out. Upstream keeps session upload progress and adds a guard instead of amputating the feature: if (PG(modules_activated)) { /* request_parse_body(). Do not run here as it corrupts the session. */ return retval; }. That guard is new in the fixed release: PG(modules_activated) appears twice in ext/session/session.c on both 8.2.29 and 8.2.34, and three times on 8.2.35. An 8.2 app with upload progress bars keeps them; an EOL app loses them the moment it is patched.

Step by step

For every Dockerfile, print PHP_VERSION and Created from the base image under the tag you currently reference. Decide from that, not from the tag name.
Replace suffix-less and OS-suffixed-but-frozen tags with a tag that is still being built, pinning the OS explicitly.
For third-party images, pin the full version triple plus the digest.
Build with --pull --platform linux/amd64.
Before deploying a Debian 11 to 12 move, test TLS handshakes to every external API from both images.
Verify, per branch: memory_hard_limit_percent on 8.2 and newer, the package changelog for distro installs, the OOM behaviour test for self-built EOL images.
Recreate every PHP container explicitly. Then re-verify inside the running container, not in the registry.
List the places where the fix will land later (CI runners, stages behind --no-recreate) as outstanding rather than done.

Bottom line

A Docker tag is not a promise of updates. --pull can leave PHP entirely unchanged, as php:8.2-fpm-bullseye does at 8.2.29, or change far more than PHP, as php:8.2-fpm does by moving to Debian 13. The only reliable inputs are PHP_VERSION in the base image config before the build and the running container’s own php -v after it.

How you prove a build is fixed depends on where it came from: official image by version, distro package by changelog, self-built backport by behaviour, and supported branch by the new memory_hard_limit_percent directive. For EOL 8.0 the official minimal backport on top of 8.0.30, built with the last docker-library Dockerfile for the branch, gives a fixed PHP without changing branch — and the application changes one line.

Branch-by-branch companions: the fleet-wide audit and the verification matrix, PHP 7.1 to 7.4, and PHP 5.6.

Primary sources: GHSA-w8hw-w9fx-j79w, GHSA-6wmf-4rv5-p3rp, php.net ChangeLog-8, docker-library/php, FrankenPHP releases. Tag data read from the Docker Hub and registry v2 APIs on 26 October 2026.

PHP2Shell on a mixed fleet: auditing 5.6 to 8.4 and proving each build is actually patched

BitPage — Mon, 26 Oct 2026 05:10:00 +0000

Short answer: for PHP 8.2 and newer, a patched build is provable statically. The fixed releases 8.2.35, 8.3.36, 8.4.27 and 8.5.12 introduce a new INI directive, memory_hard_limit_percent, which exists in none of their predecessors. For every branch from 5.6 through 8.1 there is no fixed release and no version change, so the only proof is behavioural: on a patched build an out-of-memory condition terminates the worker with exit code 1 and skips register_shutdown_function, where an unpatched build exits 255 and runs it.

PHP2Shell is two linked CVEs, published 21 October 2026 and fixed in releases dated the 22nd. The part that made it expensive was not the patching. It was answering, machine by machine, a question that normally takes one command: is this PHP fixed or not.

TL;DR

Fix: stay on the branch, move to the last patch release of that branch, then apply the patch. Never patch the patch version you happen to be running — on 7.1 and 7.2 that leaves CVE-2019-11043 open, and on 7.4.5 and 8.0.7 the official patch does not even apply.
Proof, supported branches: php -r 'var_dump(ini_get("memory_hard_limit_percent") !== false);'. True only on 8.2.35 / 8.3.36 / 8.4.27 / 8.5.12 and later.
Proof, EOL branches: the OOM behaviour test. Exit 1 and no shutdown function means the backport is compiled in; exit 255 and a shutdown function means it is not.
Trap: the behaviour test proves nothing on a supported branch. The upstream fix deliberately leaves normal memory_limit handling alone, so 8.2.29 and 8.2.35 behave identically under it. I measured both.
Trap: max_multipart_body_parts in php -i is a valid marker on 7.1 through 7.4 and useless on 8.0 and 8.1, where the directive has existed since 8.0.28.
Trap: the mitigations that look most obvious (RLIMIT_AS, ulimit -v, systemd LimitAS=) trigger the vulnerable allocation failure themselves.

What PHP2Shell actually is

Two bugs in request body parsing, both reachable before a single line of application code runs. The php-src advisories are the only authoritative description; several vendor write-ups have the two CVE numbers swapped, so take the GHSA pages as the reference.

GHSA-w8hw-w9fx-j79w (CVE-2026-109618): an out-of-memory condition during body parsing aborts the parse with a longjmp(). Half-initialised structures are later read as valid. Triggering it is a memory-amplification problem, not a size problem: a small multipart body with a large number of parts occupies orders of magnitude more in PHP’s heap than it does on the wire.
GHSA-6wmf-4rv5-p3rp (CVE-2026-109619): stale global state. A producer request that dies abnormally (fatal error, timeout, OOM) leaves residue, and a consumer request in the same worker reads it.

The two advisories are identical in their metadata, which is a hint about how closely linked the bugs are. Both are CVSS 4.0 score 9.2, critical, on the same vector CVSS:4.0/AV:N/AC:H/AT:P/PR:N/UI:N/VC:H/VI:H/VA:H/SC:N/SI:N/SA:N: high complexity and present attack requirements, but no privileges and no user interaction. Both list the same five weaknesses rather than one each: CWE-416, CWE-755, CWE-824, CWE-1325, CWE-1419. Both were published on 21 October 2026 within about twenty seconds of one another, 13:34 UTC. Both give the same four fixed versions. If you are reading a vendor summary that assigns distinct CWEs to the two CVEs, it is editorialising.

Two consequences decide the whole response. The body is parsed before routing, so authentication, CSRF tokens, maintenance mode and any framework-level filtering sit downstream of the bug and protect nothing. And the confirmed consumer path on FPM is session upload progress, which is on by default and runs without the application ever calling session_start(). I measured session.upload_progress.enabled at 1 on a stock php:8.2-cli-alpine. An application that never touches sessions is still exposed.

Fixed releases exist for four branches only: 8.5.12, 8.4.27, 8.3.36 and 8.2.35, all dated 22 October 2026 in ChangeLog-8.

The reason this could not wait for a maintenance window is the timeline. Amazon’s threat intelligence team reports a first exploitation attempt in its MadPot honeypot telemetry on 17 October, four days before disclosure and while the bugs were still under embargo, with more following from 02:00 UTC on 21 October; Sansec records Cloudflare seeing attacks at 07:43 UTC on 21 October. The advisories went public at 13:34 UTC that day. Whichever number you take, the grace period between “this is public” and “this is being attacked” was negative.

Why a mass-scan script gives you the wrong answer

A script that SSHes everywhere and runs php -v reports “no PHP” on machines where PHP is the only thing running. I had one written and threw it away.

The failure is structural. On a host whose PHP lives in a container, php -v on the host is not installed, so the script records a clean machine. On a Proxmox node the PHP is inside an LXC container that docker ps knows nothing about. On a legacy OpenVZ host you need vzlist -a. And docker exec against a container that has stopped answering hangs with no timeout at all, which parks the scan and leaves a stuck process on the host for someone to find later.

Take a registry of about two hundred entries (physical servers, VMs, LXC containers, legacy OpenVZ, cloud instances) where some are in an Ansible inventory and some exist only in a text file. Walking it by hand against a fixed checklist, one machine at a time, PHP turns up in roughly forty places across branches from 5.6 to 8.4, and the status “probably vulnerable” is never needed: wherever there is access, the exact version and SAPI are knowable.

Before any of that, reconcile the inventory against the registry. On a fleet that size expect a dozen or so entries to disagree, and each disagreement is a machine that silently drops out of the audit: containers listed in one source and absent from the other, decommissioned nodes still described as live, the same container ID apparently running on two different nodes, a “mail server” that turns out to be empty. None of those are exotic. They are what happens when two people maintain two lists for three years.

One checklist item that cost me real time: do not test SSH access with a plain ssh command. My ~/.ssh/config was substituting a personal key for every host, so “the shared key works here” was true of a key I was not testing. On one machine that made me roll back a key that had already been added correctly, then add it again. The only honest form is:

1

ssh -F /dev/null -o IdentitiesOnly=yes -i ~/.ssh/<key> -p <port> root@<host> true && echo OK

The per-machine checklist, read-only

Seven items, in this order, nothing that writes:

OS; PHP on the host; processes matching php, php-fpm, apache2|httpd, nginx, frankenphp. Container processes are visible from the host, which is often the first hint that the host-level php -v was lying.
Where PHP lives: host, docker ps, pct list for Proxmox, vzlist -a for OpenVZ.
The version of every SAPI that is actually serving, not the one on $PATH. php -v, php-fpm -v, and inside containers via docker exec.
For distribution packages, the CVE in the package changelog — a backport does not change the version number.
For Docker, whether the running container’s image matches the image currently under that tag, and when that image was built.
Whether it takes web requests, and what proxies it.
Mitigations in force: session.upload_progress.enabled, post_max_size, accounting for conf.d and pool-level php_admin_value.

1
2
3
4
5
6
7
8


# What PHP is really serving, host plus containers, with a timeout for hung containers
ps -eo user,comm | grep -E 'php|apache2|httpd|nginx|frankenphp' | sort | uniq -c
for c in $(docker ps --format '{{.Names}}'); do
 v=$(timeout 20 docker exec "$c" sh -c \
 'for b in php php-fpm frankenphp; do command -v $b >/dev/null && printf "%s: %s; " $b "$($b -v | head -1)"; done' \
 </dev/null 2>/dev/null)
 echo "$c | $(docker inspect -f '{{.Config.Image}}' "$c") | ${v:-no php}"
done

The timeout 20 is not decoration. Without it one unresponsive container stops the audit. When docker exec is hopeless you can still get the version from the image config without entering the container:

1

docker image inspect <image:tag> --format '{{range .Config.Env}}{{println .}}{{end}}' | grep PHP_VERSION

Which branches have a fix at all

Every branch from 5.6 upward has a path to a fix. Having a fix and having a package are different things.

Branch	Official	What is actually installable
8.5, 8.4, 8.3, 8.2	releases 8.5.12 / 8.4.27 / 8.3.36 / 8.2.35	official Docker images, sury packages
8.1, 8.0, 7.4	minimal source backports from the PHP Security Team: branches `PHP-{7.4,8.0,8.1}-CVE-2026-109618-109619`, one commit on top of the last tag	plus sury packages for Debian 11–13; 7.4 also via Freexian ELTS and Ubuntu Pro ESM
7.3, 7.2, 7.1, 5.6	none	sury patches by Ondřej Surý; Ubuntu Pro ESM for the distro `php7.2` and `php5`; TuxCare alt-php, paid

The official backports are a single commit each, and the commit stats are worth knowing before you plan the work: 712e60f9edfd on the 7.4 branch touches 17 files for +240/−59, 82f8f31b16a0 on 8.0 touches 17 files for +229/−59, and 1d7dec178a09 on 8.1 touches 23 files for +311/−73. None of them touch the fpm or apache2handler SAPI directories.

The sury patch for the oldest branch documents its own limits, which is why I trust it more than the community ports. The header of 0261-Avoid-state-corruption-via-bailout.patch, by Arnaud Le Blanc, co-author of the upstream fix, states plainly: “Minimal mitigation for EOL version.” It then lists what it deliberately leaves out, including that “the max_multipart_body_parts hunk of the 7.4 patch is left out; the limit comes with the CVE-2023-0662 backport”. A patch that tells you what it is not doing is a patch you can plan around.

A third-party “universal” port covering 5.4 to 7.3 was circulating. It modifies 8 files against the official 17, with zend_hash.c, php_variables.c and the VM untouched, and its own authors tested it on the built-in CLI server. I did not use it.

The ordering rule that matters

Patch the last patch release of the branch, not the version you are running. Two independent reasons, both verified on vanilla tarballs:

The patch frequently does not apply. The official 7.4 commit goes onto 7.4.33 and 7.4.29 cleanly and conflicts on 7.4.5 in zend_vm_*. The 8.0 commit is clean on 8.0.30 and conflicts on 8.0.7.
Where it does apply, it can leave an older hole open. The sury 7.1 patch applies cleanly to 7.1.8, 7.1.11 and 7.1.17 — all of which predate the CVE-2019-11043 fix that landed in 7.1.33. Closing PHP2Shell on 7.1.8 buys you an unauthenticated FPM RCE you already had.

How to prove a build is patched

This is where I spent the most time, and where the published advice is thinnest. Three layers, and they fail differently.

The version number is not a proof, in either direction. A self-built “7.4.33 plus patch” reports exactly 7.4.33, the same as an unpatched one. And distributions do not even agree with each other about whether the version should move. Ubuntu backports into a frozen upstream version; Debian rebases to the fixed upstream version. From the two vendors’ own trackers:

Vendor	Suite	Package	Fixed version	Upstream version moved?
Ubuntu	trusty (ESM)	`php5`	`5.5.9+dfsg-1ubuntu4.29+esm18`	no
Ubuntu	xenial (ESM)	`php7.0`	`7.0.33-0ubuntu0.16.04.16+esm22`	no
Ubuntu	bionic (ESM)	`php7.2`	`7.2.24-0ubuntu0.18.04.17+esm14`	no
Ubuntu	focal (ESM)	`php7.4`	`7.4.3-4ubuntu2.29+esm5`	no
Ubuntu	jammy	`php8.1`	`8.1.2-1ubuntu2.28`	no
Ubuntu	noble	`php8.3`	`8.3.6-0ubuntu0.24.04.13`	no
Debian	bookworm	`php8.2`	`8.2.35-1~deb12u1`, from `8.2.32-1~deb12u1`	yes
Debian	trixie	`php8.4`	`8.4.27-1~deb13u1`, from `8.4.24-1~deb13u1`	yes

Two rows there are worth a second look. Ubuntu’s php8.1 on jammy is fixed at 8.1.2, a branch upstream declared end of life with no fixed release. The distro is maintaining it anyway. Ubuntu’s php8.3 on noble stays at 8.3.6 while upstream’s fix is 8.3.36; a scanner comparing against “8.3.36 or newer” marks a fully patched machine as vulnerable. Meanwhile on Debian bookworm the version genuinely did move to 8.2.35, so there php -v works fine.

The practical consequence is that a version check needs to know which vendor built the PHP before it can interpret the number, which is a strange thing to need and a good argument for checking something else instead. Both tables above come from machine-readable sources you can query yourself: https://ubuntu.com/security/cves/CVE-2026-109618.json and the Debian security tracker’s JSON. A scanner working from php -v alone will call a patched image vulnerable and a backported package safe, and be wrong both times.

The INI output is a proof on supported branches only, and this is new. The fixed releases add a directive that no earlier release has. I checked main/main.c across tags:

Tag	`memory_hard_limit_percent` occurrences
php-8.2.29	0
php-8.2.34	0
php-8.2.35	4
php-8.3.36	4
php-8.4.27	4
php-8.5.12	4
php-8.1.34	0
`PHP-7.4-CVE-...` backport branch	0
`PHP-8.0-CVE-...` backport branch	0
`PHP-8.1-CVE-...` backport branch	0

The directive appears exactly in the four fixed releases and in none of their predecessors. It is declared as PHP_INI_ENTRY("memory_hard_limit_percent", NULL, PHP_INI_ALL, OnChangeMemoryHardLimitPercent) and rejects nonsense with “memory_hard_limit_percent must be >= 100, or -1”. That gives one check valid on any supported branch, including a distro package whose version string refuses to move:

1
2
3


php -r 'var_dump(ini_get("memory_hard_limit_percent") !== false);'
# true -> this build is 8.2.35 / 8.3.36 / 8.4.27 / 8.5.12 or newer
# false -> unpatched, or an EOL backport, or a partial distro backport

Two things about that check, both of which I got wrong before I got them right.

The first is the !== false, which is not defensive style but the whole mechanism. The directive’s default value is NULL, so on a patched build ini_get() returns an empty string, and an empty string is falsy in PHP. Written the obvious way, the check reports every build in the fleet as unpatched:

Image	`ini_get(...)`	`!== false`	`if (ini_get(...))`
`php:8.2-cli-alpine3.21` (8.2.29)	`false`	`false`	`false`
`php:8.2-cli-alpine` (8.2.35)	`''`	`true`	`false`
`php:8.1-cli-alpine` (8.1.34, EOL)	`false`	`false`	`false`

Measured on those three images. The middle row is the one that matters: a patched build and an unpatched build are indistinguishable under a truthiness test, and the failure is silent and uniform. A scanner written that way produces a clean-looking report in which nothing is patched, which is the kind of output people believe because it matches their fears. array_key_exists('memory_hard_limit_percent', ini_get_all()) works too and is harder to get wrong.

The second is the false case. The directive is absent from all three official EOL backport branches and from all six sury EOL patch sets I unpacked, so a false on 7.x or 8.0 tells you nothing at all. It is a positive marker, not a negative one.

The behaviour test is the proof on EOL branches, and only there. The EOL backports replace the bailout with a hard exit. In the 8.0 patch the diff reads, verbatim:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


- heap->overflow = 1;
- zend_try {
- zend_error_noreturn(E_ERROR, format, limit, size);
- } zend_catch {
- } zend_end_try();
- heap->overflow = 0;
- zend_bailout();
- exit(1);
+ (void) heap;
+ zend_mm_safe_error_ex(format, limit, size);

and the new function ends in _exit(1) after writing the fatal error to stderr directly. That is observable from userland. The whole test is this:

1
2
3
4
5


<?php
register_shutdown_function(function () { fwrite(STDERR, "SHUTDOWN-RAN\n"); });
ini_set('memory_limit', '64M');
$a = [];
while (true) { $a[] = str_repeat('x', 100000); }

1
2


docker run --rm --memory 512m -v "$PWD/oom.php:/oom.php:ro" <image> php /oom.php
echo "exit=$?"

Three builds, same script, memory_limit=64M in a 512 MB container:

Build	Exit code	Fatal error written to	Shutdown function
`php:8.2-cli-alpine3.21`, PHP 8.2.29 (unpatched)	255	stdout	ran
`php:8.2-cli-alpine`, PHP 8.2.35 (fixed)	255	stdout	ran
sury `php8.0-cli` `1:8.0.30-25+0~20261021.88+debian12~1.gbpabebea`, EOL backport	1	stderr	did not run

The destructor ran in none of the three.

The middle row is the one that surprised me, and it corrects what I had assumed from reading the patches. The supported-branch fix leaves ordinary memory_limit handling exactly as it was (soft OOM is deferred to a safe point rather than converted into a process exit), so 8.2.29 and 8.2.35 are indistinguishable under this test. Only the EOL backports turn a memory_limit breach into _exit(1). Anyone using the OOM test as a general “is it patched” check on 8.2 will get a false negative every time.

The supported branches do have a hard path, through the new directive: zend_mm_hard_limit_error prints “Hard memory limit of %zu bytes exhausted (tried to allocate %zu bytes) in %s on line %d” and calls _exit(137). I could not reach it in testing, because the soft limit fires first even with -d memory_hard_limit_percent=100, so I am describing that one from the source rather than from a measurement.

The marker table, per branch

Putting it together, because the answer differs by branch and getting it wrong wastes a day:

Branch	Static marker in `php -i`	Why
8.2–8.5	`memory_hard_limit_percent`	new in the fixed releases
8.1, 8.0	none	`max_multipart_body_parts` already exists since 8.0.28
7.4, 7.3, 7.2, 7.1	`max_multipart_body_parts`	absent from every vanilla 7.x, added by the fix
5.6	`max_multipart_body_parts`, but only with the second sury patch	the bailout patch alone adds no directive

I verified the second column by counting the directive in main/main.c across vanilla tags: zero in 5.6.40, 7.1.33, 7.2.34, 7.3.33 and 7.4.33, one in 8.0.28, 8.0.30 and 8.1.34.

The build assert that asserts nothing

If you compile your own, put a marker check in the build so it fails when the patch silently stops applying. My first choice was wrong: I used grep -q zend_objects_store_mark_destructed, which passes on unpatched source as well, so the assert was decorative. The rule is to check the candidate symbol against a vanilla tree first and require zero.

array_init_singleton in Zend/zend_API.h is the one I would use now. It occurs zero times in vanilla 5.6.40, 7.4.33 and 8.0.30, and is added by every one of the six EOL patch sets — one marker covering 5.6 through 8.0 rather than a different grep per branch.

1

grep -c array_init_singleton Zend/zend_API.h # 0 before the patch, >=1 after

One gotcha if you verify a package rather than a source tree inside a container: debian:*-slim images ship /etc/dpkg/dpkg.cfg.d/docker with path-exclude /usr/share/doc/* and path-include /usr/share/doc/*/copyright. The package changelog you were told to grep for the CVE is not in the image at all; only copyright survives. I spent a while convinced a sury package had shipped without a changelog entry.

Mitigations that do not work, and one that makes it worse

Most of the obvious knobs are downstream of the bug.

file_uploads = Off does not help. The multipart body is still parsed.
ini_set() and .user.ini are applied after the body is parsed. Too late by construction. Only php.ini, pool-level php_admin_value, or the Apache config, followed by a full worker restart — for Apache, not a graceful one.
Blocking multipart/* at the proxy does not close the urlencoded variant, so you need a body-size limit as well. Do not scope the block to POST either: since 8.4, request_parse_body() parses other methods too.
RLIMIT_AS, RLIMIT_DATA, ulimit -v and systemd LimitAS= are actively harmful. They cause the same allocation failure the bug lives in. I would expect OpenVZ beancounters to behave the same way, but that is a guess I have not tested.
memory_limit = -1, which the advisory offers as a mitigation for 109618, needs care. On a pool with a few hundred workers and no container memory limit it trades a hard-to-exploit RCE for a reliable OOM of the whole VM, and in a container sharing space with a database it hands the OOM killer the database.

The one mitigation worth applying everywhere is session.upload_progress.enabled = 0 in php.ini or the pool config. It closes the confirmed FPM path for 109619 and breaks almost nothing — upload progress bars, if the application has them.

What the patch changes in application behaviour

The EOL backports change three things you can notice from the application side, and all three are worth knowing before deploying rather than after.

Shutdown functions stop running on OOM. The worker goes through _exit(1), so register_shutdown_function never fires, the message goes to stderr instead of the output stream, and FPM behind nginx returns 502 where it used to return 500. An application that routinely brushes against memory_limit will start producing 502s, and whatever you do in a shutdown handler (flushing an error reporter, releasing a queue job, writing a last log line) stops happening. If your monitoring matches on exit code 255, it will not match on 1.
Session upload progress is disabled in code, not in configuration. The backports patch the guard to if (!PS(rfc1867_enabled) || true), so the feature is off while php -i still reports session.upload_progress.enabled => On. The supported branches took the other route: they keep the feature and add a guard, if (PG(modules_activated)) { /* request_parse_body(). Do not run here as it corrupts the session. */ return retval; }. That guard is new in the fixed release: PG(modules_activated) occurs twice in ext/session/session.c on 8.2.29 and 8.2.34, and three times on 8.2.35.
A new limit truncates very large forms. max_multipart_body_parts defaults to -1, which falls back to max_input_vars + max_file_uploads. On stock defaults that is 1000 + 20, so 1020 parts, after which you get: “Multipart body parts limit exceeded %d. To increase the limit change max_multipart_body_parts in php.ini.” I measured the stock values on php:8.2-cli-alpine: max_input_vars 1000, max_file_uploads 20, max_multipart_body_parts -1. The fallback formula is identical in main/rfc1867.c on vanilla 8.2 and in the 5.6 backport.

Looking for traces

Because exploitation preceded disclosure, retrospective hunting is part of the job. The signatures come from the mechanism, not from a particular payload:

1
2
3
4
5


# OOM before the script even starts: the parse never reached user code
grep -h 'Allowed memory size.*Unknown on line 0' /var/log/php*/*.log
# worker crashes
grep -h 'AH00052' /var/log/apache2/error.log # mod_php segfault
grep -h 'exited on signal 11' /var/log/php*-fpm.log # FPM

Beyond the logs: runs of large POSTs from one address in a short window, PHP_SESSION_UPLOAD_PROGRESS appearing in request bodies, an unusually long PHPSESSID in a Set-Cookie response, and shell processes such as sh -c, curl or wget parented to a PHP worker. Adding $content_type and $request_length to the nginx access log format makes the first of those searchable, and costs nothing to leave on.

Step by step

Reconcile the registry against the inventory before touching any machine. Every disagreement is a machine that would not have been checked.
Walk the fleet by hand against the seven-item checklist. Record the exact version and SAPI, never “probably”.
For each branch found, decide the source of the fix from the availability table — official release, official EOL backport, or sury patch set.
For every self-built target, apply the patch to the last patch release of the branch, and check for older CVEs in whatever you are moving off.
Verify in three layers: patch applies, patch compiles, patch works. The third is the marker table plus the behaviour test for EOL.
Tag self-built images and packages so the build is identifiable later — an image tag and a LABEL, or an RPM Release bump.
Apply session.upload_progress.enabled = 0 everywhere regardless, in php.ini or the pool config.
Run the retrospective log hunt back to 17 October before you close the incident.

Bottom line

“EOL means there is no fix” is false — every branch from 5.6 up has a usable patch, from the PHP Security Team for 7.4 through 8.1 and from the Debian maintainer for everything older. “There is a fix” is not the same as “there is a package”, and for the old branches it means a build you now maintain. Patch the last patch release of the branch, never the version you are on.

The part worth taking away is the verification. php -v cannot answer the question on any branch. On 8.2 and newer, ini_get('memory_hard_limit_percent') !== false can, and it works through a distro backport that never changes the version string. On 5.6 through 8.1 nothing static answers it and you need the OOM behaviour test — which, inverted, means that same test proves nothing on 8.2, where both the vulnerable and the fixed build exit 255 and run their shutdown functions. I measured that one the hard way.

The branch-by-branch work is written up separately: PHP 8 and the frozen Docker tags, PHP 7.1 to 7.4 with the official and sury backports, and PHP 5.6, where the only proof is behavioural.

Primary sources: GHSA-w8hw-w9fx-j79w, GHSA-6wmf-4rv5-p3rp, php.net ChangeLog-8, php-src for the backport branches, packages.sury.org for the EOL patch sets, and docker-library/php for the historical Dockerfiles.

Room 3 froze on a suspending @Transaction: the SQLite driver you pass picks your connection pool

BitPage — Sat, 10 Oct 2026 12:35:00 +0000

Short answer: Room 3 does not manage connections the same way for every driver. It reads the hasConnectionPool property off whatever SQLiteDriver you pass to setDriver(), and that single boolean picks the concurrency model. AndroidSQLiteDriver returns true, so Room wraps it in PassthroughConnectionPool, which holds one thread-confined connection and has no acquisition timeout at all. A suspending @Transaction DAO function on that pool can block forever: the main thread stays idle, so there is no ANR, no crash, and no log line. Swapping to BundledSQLiteDriver makes Room run its own pool instead, and Room configures PRAGMA synchronous on every connection it opens, which quietly discards anything you set in RoomDatabase.Callback.onOpen.

TL;DR

Fix: pass a driver that reports hasConnectionPool = false (BundledSQLiteDriver does, by inheriting the default) so Room manages its own pool. On WAL that is four readers and one writer, and it has a documented acquisition timeout (30 seconds by default) that throws SQLiteException instead of waiting.
Diagnosis: kill -3 on the process, then read /data/anr/trace_00. SIGQUIT dumps every thread without needing a crash, an ANR dialog, or a debugger attach.
Trap: the driver swap moves you from a single connection to several, and Room runs its per-connection PRAGMA setup on each of them. RoomDatabase.Callback.onOpen fires once per database, not once per connection, so a pragma you set there survives on exactly one connection.
Second trap: if you write a driver wrapper to fix that, delegate inTransaction(). The default implementation in androidx.sqlite throws NotImplementedError, and Room calls it before every write transaction.

Why does an Android app hang on the splash screen with no crash and no ANR?

Because the main thread is not stuck. Android raises an ANR when the main thread fails to service input or draw, and a main thread parked in epoll_pwait waiting for the next message is, as far as the system is concerned, perfectly healthy. If your splash gate is a boolean that some background work is supposed to flip, and that background work is blocked, the main thread will happily keep drawing the splash forever.

This is what I was looking at one morning in early autumn: reinstall the app with adb install -r, cold start, splash, nothing. Three cold starts, three hangs, every time. The process was alive, dumpsys activity activities showed it as topResumedActivity, and the splash condition came from the standard pattern:

1

splashScreen.setKeepOnScreenCondition { !viewModel.isReady }

I spent the first stretch of that session chasing the wrong things, and all three dead ends were my own.

Dead end one: looking for a crash that was not there. My first instinct was that something had thrown on a background thread and been swallowed.

$ adb logcat -d --pid=$(adb shell pidof com.example.tracker) | grep -c "FATAL\|AndroidRuntime"
0

Zero. There was no exception. A deadlock produces no log output at all, which is exactly what makes it expensive: every tool that reports failures had nothing to report.

Dead end two: blaming the network. The app opens a server-sent-events stream on startup, so I assumed the stream had stalled and the readiness flag was waiting on it. The OkHttp log said otherwise: the requests went out and came back with 200 OK, including the long-lived stream. The network layer had done its job and gone quiet.

Dead end three: blaming the splash API. I read setKeepOnScreenCondition three times looking for a lifecycle mistake. There was none. The condition was correct; the value it read never changed.

The lesson I keep relearning: when there is no error, stop reading your own code and ask the runtime what the threads are doing.

How do you get a stack trace out of a process that has not crashed?

Send it SIGQUIT. The Android runtime installs a signal handler that dumps every thread’s stack to a trace file, which is the same mechanism that produces an ANR trace, except you get to trigger it on demand:

$ adb root
$ adb shell kill -3 $(adb shell pidof com.example.tracker)
$ adb shell cat /data/anr/trace_00

That dump changed the problem from “the app hangs” to “I know which lock”. The main thread was where I expected it:

"main" prio=5 tid=1 Native
native: #00 pc ... /apex/com.android.runtime/lib64/bionic/libc.so (__epoll_pwait+8)
at android.os.MessageQueue.nativePollOnce(Native method)
at android.os.Looper.loopOnce(Looper.java:...)

Idle. The interesting thread was a coroutine worker. Trimmed, and with my own class and package names replaced by generic ones:

"DefaultDispatcher-worker-2" prio=5 tid=24 Waiting
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:...)
at androidx.room3.TriggerBasedInvalidationTracker$syncTriggers$2$1.invokeSuspend(InvalidationTracker.kt:634)
at androidx.room3.coroutines.PassthroughConnectionPool$useConnection$3.invokeSuspend(PassthroughConnectionPool.kt:90)
...
at androidx.room3.util.DBUtil__DBUtilKt.performInTransactionSuspending(DBUtil.kt:102)
at com.example.tracker.data.OutboxDao_Impl.takeNextForUpload(OutboxDao_Impl.kt:192)

Three names in that stack told the whole story: PassthroughConnectionPool, TriggerBasedInvalidationTracker, and performInTransactionSuspending. A DAO function of mine, marked @Transaction and suspend, had entered Room’s transaction machinery and was waiting on a lock held by something that was waiting on it.

The DAO function was unremarkable. It claims the next queued item by moving it to an in-flight state inside a transaction, so that two concurrent workers cannot pick up the same row:

1
2
3
4
5
6


@Transaction
open suspend fun takeNextForUpload(userId: String, nowMillis: Long): OutboxEntity? {
 val next = oldestUploadable(userId) ?: return null
 updateState(userId, next.clientId, "UPLOADING", nowMillis)
 return next.copy(state = "UPLOADING", updatedAtEpochMillis = nowMillis)
}

A @Transaction function with a body, rather than a single query, is what the Room team calls a transaction delegate. Room generates a wrapper that opens a transaction and invokes your code inside it; the generated entry point is performInTransactionSuspending, whose own documentation says it “is needed to support @Transaction delegate DAO functions”. That distinction matters here, and I will come back to it.

Why does the SQLite driver decide whether Room can deadlock?

Because Room asks the driver whether it already has a pool, and takes the answer at face value. SQLiteDriver carries a hasConnectionPool property whose default is false, and the Android framework driver overrides it. From AndroidSQLiteDriver.android.kt:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


/**
 * A [SQLiteDriver] implemented by [android.database] and that uses the Android's SDK SQLite APIs.
 *
 * This driver internally has a connection pool and a opened connection from it is safe to be used
 * in a multi-thread and concurrent environment.
 */
public class AndroidSQLiteDriver : SQLiteDriver {

 @get:JvmName("hasConnectionPool")
 override val hasConnectionPool: Boolean
 get() = true

BundledSQLiteDriver does not override the property at all, so it inherits false. The consequence runs opposite to the intuition: the driver that advertises a connection pool is the one that leaves Room without a pool of its own. RoomConnectionManager.android.kt branches on it directly, and the comment names the driver:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


this.connectionPool =
 if (config.sqliteDriver.hasConnectionPool) {
 // If the driver already has a connection pool then use a pass-through pool to
 // support drivers such as the Android since internally it already has a
 // thread-confined connection pool.
 PassthroughConnectionPool(...)
 } else if (config.name == null) {
 newSingleConnectionPool(...)
 } else {
 ...
 }

“Thread-confined” is the load-bearing word. The framework’s pool hands out a connection tied to the calling thread, and coroutines change threads whenever the dispatcher feels like it. PassthroughConnectionPool is candid about what it is:

1
2
3
4
5
6
7
8


/**
 * An implementation of a connection pool that doesn't do any connection management.
 *
 * Used in two scenarios:
 * * When Room is in compatibility mode (no driver is provided to Room).
 * * When a driver that reports having a connection pool (returns `true` from
 * [SQLiteDriver.hasConnectionPool]) is provided to Room.
 */

It holds a single private lateinit var connection: SQLiteConnection and finds reentrant access by looking for a ConnectionElement in the current coroutine context. Lose the thread, and you still have the context element; lose the context element, and you queue behind yourself.

	`AndroidSQLiteDriver`	`BundledSQLiteDriver`
`hasConnectionPool`	`true` (overridden)	`false` (inherited default)
Pool Room installs	`PassthroughConnectionPool`	`newSingleConnectionPool` or `newConnectionPool`
Connections Room sees	one, thread-confined	one (in-memory) or four readers + one writer on WAL
Acquisition timeout	none	30 s by default, then `SQLiteException`
Native code in your APK	none	`libsqliteJni.so`, four ABIs
SQLite version	whatever the OS ships	whatever the artifact ships

The reader and writer counts are not folklore; they are constants in ConnectionPoolConfiguration.kt:

1
2


internal const val WAL_DEFAULT_NUMBER_OF_READERS = 4
internal const val WAL_DEFAULT_NUMBER_OF_WRITERS = 1

Where exactly does a suspending @Transaction wait on itself?

In the invalidation tracker sync that Room runs immediately before opening a write transaction. DBUtil.kt acquires the connection first, then calls invalidationTracker.sync() inside that connection’s scope:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


internal suspend inline fun <R> RoomDatabase.internalPerform(
 isReadOnly: Boolean,
 inTransaction: Boolean,
 crossinline block: suspend (PooledConnection) -> R,
): R =
 useConnection(isReadOnly) { transactor ->
 if (inTransaction) {
 ...
 if (!isReadOnly && !transactor.inTransaction()) {
 invalidationTracker.sync()
 }
 val result = transactor.withTransaction(type) { block.invoke(this) }

sync() reaches TriggerBasedInvalidationTracker.syncTriggers(), which asks for a writer connection of its own (useConnection(isReadOnly = false)) and then takes a plain ReentrantLock guarding the observed-table state. That state is guarded by two different kinds of lock at once, and InvalidationTracker.kt documents the ordering rule in the source:

1
2
3
4
5


/**
 * Mutex to serialize [onSync] calls. Should never be acquired within a [lock] region, the order
 * must always be `[onSyncMutex]? -> [lock]` to avoid deadlocks.
 */
private val onSyncMutex = Mutex()

A kotlinx.coroutines.sync.Mutex is owned by a coroutine. A ReentrantLock is owned by a thread. Holding one and then taking the other, in a code path that is allowed to change threads partway through, is the hazard, and it is the hazard the Room team named out loud. The commit behind the fix in 3.0.2, authored by Daniel Santiago Rivera on 7 August 2026, says it plainly:

A @Transaction delegate in a DAO has user-code that might switch dispatchers and cause databases with a passthrough pool (AndroidSQLiteDriver) to deadlock due to a change of thread. There an optimization to not hop to the transaction thread for ‘internal’ Room operations but that is only safe knowing no dispatcher change occurs, something that can’t be guaranteed with transaction delegates in DAOs.

The remedy in that commit is a context marker, and the comment it left in DBUtil.kt is now the second line of performInTransactionSuspending:

1
2
3
4
5


public suspend fun <R> performInTransactionSuspending(db: RoomDatabase, block: suspend () -> R): R =
 // The external operation marker is used since this wraps user code that might change
 // dispatchers and on a passthrough pool with thread local connections (AndroidSQLiteDriver)
 // the confined transaction thread behavior must be used at the cost of some thread hops.
 withContext(RoomExternalOperationElement + db.getCoroutineContext(inTransaction = true)) {

In the current source that withContext sits at DBUtil.kt:102, which is the exact frame my thread dump pointed at.

The reason the symptom was an eternal splash rather than an error is the timeout column in the table above. PassthroughConnectionPool.useConnection takes no timeout parameter and has nowhere to put one. Room’s own pool does, and useReaderConnection documents the consequence: “If a caller has to wait too long to acquire a connection a SQLiteException will be thrown due to a timeout.” Room 3.1.0-alpha01 made the value settable, and its setConnectionPoolTimeout documentation puts the default at 30.seconds. Same class of bug, two completely different failure modes. One throws after half a minute and lands in your crash reporter. The other holds the only connection in the process, so every subsequent read queues behind it, the readiness flag never flips, and you get a splash screen that outlives your patience.

Is this already fixed in a newer Room 3?

Partly, and the version history is worth reading before you blame your own code. All three of these entries live in the Room 3 release notes:

Version	Date	Relevant entry
3.0.2	26 Aug 2026	“Fix a deadlock that can occur on @Transaction DAO wrapper suspend functions that would do a context switch and had the AndroidSQLiteDriver configured.” (`b/543285472`)
3.0.3	9 Sep 2026	“Fix a regression that would cause Room to throw an IllegalMonitorStateException when using @Transaction wrapper DAO functions.” (`b/543285472`, `b/553140228`)
3.1.0-alpha01	9 Sep 2026	Adds `setConnectionPoolTimeout(timeout: Duration)` to `RoomDatabase.Builder`

I was running 3.0.2 when this bit me, which is to say I was running the version whose notes claim to fix this exact configuration, and the DBUtil.kt:102 frame in my dump is the fix itself. Then 3.0.3 had to repair a regression introduced by that fix, under the same bug id. I draw one conclusion from that sequence, and it is not that Room is broken: a thread-confined connection behind a coroutine API is a hard problem that took several attempts to get right, and the configuration that needs those attempts is the passthrough pool. If you can avoid being on that code path at all, avoid it.

Which is why the first thing I changed was one line in the builder:

1
2
3
4


Room.databaseBuilder<AppDatabase>(context, DB_NAME)
 .setDriver(BundledSQLiteDriver()) // was AndroidSQLiteDriver()
 .setJournalMode(JournalMode.WRITE_AHEAD_LOGGING)
 .build()

The splash cleared on the next install. Upgrade to current Room as well, of course. But the driver choice is the one that moves you off the pool implementation that “doesn’t do any connection management”.

What does switching to BundledSQLiteDriver actually cost?

Native code in your package, and you can measure that from Google’s Maven repository without building anything. I pulled both artifacts at version 2.7.1 and read the archives:

$ curl -sO https://dl.google.com/android/maven2/androidx/sqlite/sqlite-framework-android/2.7.1/sqlite-framework-android-2.7.1.aar
$ curl -sO https://dl.google.com/android/maven2/androidx/sqlite/sqlite-bundled-android/2.7.1/sqlite-bundled-android-2.7.1.aar
$ stat -c '%10s %n' *.aar
2551019 sqlite-bundled-android-2.7.1.aar
60286 sqlite-framework-android-2.7.1.aar

60,286 bytes against 2,551,019. The difference is four copies of a compiled SQLite:

$ unzip -l sqlite-bundled-android-2.7.1.aar 'jni/*'
Length Date Time Name
--------- ---------- ----- ----
1281456 1980-02-01 00:00 jni/arm64-v8a/libsqliteJni.so
1254892 1980-02-01 00:00 jni/armeabi-v7a/libsqliteJni.so
1140764 1980-02-01 00:00 jni/x86/libsqliteJni.so
1189688 1980-02-01 00:00 jni/x86_64/libsqliteJni.so
--------- -------
4866800 9 files

Deflated inside the archive each one costs a little under half that:

ABI	`libsqliteJni.so` uncompressed	stored in the AAR
`arm64-v8a`	1,281,456 B	645,789 B
`armeabi-v7a`	1,254,892 B	625,184 B
`x86_64`	1,189,688 B	626,525 B
`x86`	1,140,764 B	632,389 B

A universal APK carries all four, roughly 2.5 MB compressed. An ABI split or an App Bundle delivers one, which is 620 to 650 KB compressed and about 1.2 MB unpacked. That is the honest number to put in a decision, rather than the “it adds several megabytes” figure I had been repeating. The four ABIs are not an accident either; they are the Android entries in the requiredNativeTargets list in sqlite-bundled/build.gradle, where the Android targets are also the ones compiled with -Oz for size while every other platform gets -O3.

You get something for those bytes: a SQLite version you control, identical across every device and every OS release, instead of whatever the vendor shipped. For an app that uses FTS5, JSON functions, or PRAGMA behaviour that changed between SQLite releases, that is the actual reason to bundle, and the deadlock fix is a bonus.

One caveat from the bundled driver’s own documentation: the amalgamation is compiled in multi-thread mode, so its connections are not individually thread-safe. That is fine precisely because Room puts a real pool in front of them. It is not fine if you hand those connections around yourself.

Why did the driver swap silently change PRAGMA synchronous?

Because Room sets PRAGMA synchronous itself, per connection, based on the journal mode, and it does so without telling you. From RoomConnectionManager.kt:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


private suspend fun configureSynchronousFlag(connection: SQLiteConnection) {
 // Use NORMAL in WAL mode and FULL for non-WAL as recommended in
 // https://www.sqlite.org/pragma.html#pragma_synchronous
 val wal = configuration.journalMode == WRITE_AHEAD_LOGGING
 if (wal) {
 connection.executeSQL("PRAGMA synchronous = NORMAL")
 } else {
 connection.executeSQL("PRAGMA synchronous = FULL")
 }
}

On AndroidSQLiteDriver this was invisible to me, because I had set PRAGMA synchronous = FULL in a RoomDatabase.Callback.onOpen and there was only ever one connection for it to apply to. Once Room was running its own WAL pool, the setting survived on the first connection and nowhere else, and the asymmetry is explicit in the same file. The two configuration paths are deliberately different:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


/**
 * Performs initial database connection configuration and opening procedure, such as running
 * migrations if necessary, validating schema and invoking configured callbacks if any.
 */
private suspend fun configureDatabase(connection: SQLiteConnection) {
 configureBusyTimeout(connection)
 configureJournalMode(connection)
 configureSynchronousFlag(connection)
 // ... migrations, identity check ...
 onOpen(connection) // -> callbacks.forEach { it.onOpen(connection) }
}

/**
 * Performs non-initial database connection configuration, specifically executing any
 * per-connection PRAGMA.
 */
protected suspend fun configurationConnection(connection: SQLiteConnection) {
 configureBusyTimeout(connection)
 configureSynchronousFlag(connection)
 openDelegate.onOpen(connection) // note: user callbacks are NOT invoked here
}

The public reference for RoomDatabase.Callback.onOpen documents the behaviour in one sentence: “Called when the database has been opened.” Nothing there says once per database rather than once per connection, and with a single-connection pool the difference never shows up. Note also that the Room 3 signature is open suspend fun onOpen(connection: SQLiteConnection): you are handed a connection, which reads like a per-connection hook and is not one.

Worth being precise about what the downgrade costs, because “Room turned off fsync” would be wrong. SQLite’s own documentation is unusually clear here:

WAL mode is always consistent with synchronous=NORMAL, but WAL mode does lose durability. A transaction committed in WAL mode with synchronous=NORMAL might roll back following a power loss or system crash. Transactions are durable across application crashes regardless of the synchronous setting or journal mode. The synchronous=NORMAL setting provides the best balance between performance and safety for most applications running in WAL mode.

The database never corrupts, and an app crash loses nothing. What NORMAL gives up is the last committed transaction across a power loss or a kernel panic. Room’s default is defensible, and the SQLite documentation Room links to recommends it. It was wrong for my case because the table in question is an outbox: a row written there is the only record that a user’s work exists before it reaches the server, and losing the most recent commit on a device whose battery died is exactly the case the outbox exists to survive. FULL is 2 in SQLite’s encoding, and SQLite calls it “atomic, consistent, isolated, and durable (ACID) in WAL mode”.

Pick deliberately. If your WAL database is a cache or a materialised read model, leave Room’s NORMAL alone.

How do you catch a pragma that only applies to some connections?

With an instrumented test that reads the pragma back on a writer connection. I only found the regression because I had written one, and I had written it for a different reason: after the driver change I wanted proof that WAL was still on. Room exposes the connection directly for this:

1
2
3
4
5
6
7


@Test
fun journalModeIsWalAndSynchronousIsFullOnTheWriterConnection() = runTest {
 db.useWriterConnection { connection ->
 assertThat(connection.readSingleText("PRAGMA journal_mode")).isEqualTo("wal")
 assertThat(connection.readSingleInt("PRAGMA synchronous")).isEqualTo(2)
 }
}

The failure was two integers:

expected: 2
but was : 1

2 is FULL, 1 is NORMAL. A couple of dozen other instrumented database tests passed, because none of them cared about durability settings. This is the kind of assertion that looks like test-suite padding right up until the day it earns its place. If you configure pragmas on an Android database, assert them on a connection rather than trusting the callback that set them.

useWriterConnection and useReaderConnection are suspending functions on RoomDatabase and are the honest way to ask; a raw query through a DAO may well land on a different connection than the one you mean to inspect.

How do you force a pragma on every connection Room opens?

Wrap the driver, not the database. Room calls SQLiteDriver.open() for each connection it creates, which is the only hook that sees all of them. Intercepting the pragma statement itself is the smallest change that works:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


class FullSyncSQLiteDriver : SQLiteDriver {
 private val delegate = BundledSQLiteDriver()

 override fun open(fileName: String): SQLiteConnection =
 FullSyncConnection(delegate.open(fileName))

 private class FullSyncConnection(
 private val delegate: SQLiteConnection
 ) : SQLiteConnection {

 override fun prepare(sql: String): SQLiteStatement =
 delegate.prepare(
 if (isSynchronousPragma(sql)) "PRAGMA synchronous = FULL" else sql
 )

 // Required. The interface default throws, and Room calls this before every
 // write transaction.
 override fun inTransaction(): Boolean = delegate.inTransaction()

 override fun close() = delegate.close()

 private fun isSynchronousPragma(sql: String): Boolean =
 sql.trim()
 .replace(Regex("\\s+"), " ")
 .startsWith("PRAGMA synchronous =", ignoreCase = true)
 }
}

The inTransaction() override is the part I got wrong first, and my initial explanation of why was also wrong. I assumed the interface’s default implementation answered from the wrapper’s own state and therefore returned a stale false. It does not answer at all. From SQLiteConnection.nonWeb.kt:

1
2
3
4
5
6


public actual interface SQLiteConnection : AutoCloseable {

 /** Returns true if the connection has an active transaction, false otherwise. */
 public actual fun inTransaction(): Boolean {
 throw NotImplementedError("$this does not implement inTransaction().")
 }

Without the delegation, every DAO function that writes inside a transaction fails, because internalPerform calls transactor.inTransaction() on the way in. In my case that was every test in one DAO class at once, which at least made the cause obvious. The reason the omission compiles cleanly is that both of these are default interface members, and they have been since the version I was on. The frozen signature file sqlite/api/2.7.0-rc01.txt records it:

public interface SQLiteConnection extends java.lang.AutoCloseable {
method public default boolean inTransaction();
public interface SQLiteDriver {
method @InaccessibleFromKotlin public default boolean hasConnectionPool();
property public default boolean hasConnectionPool;

A Kotlin class implementing SQLiteConnection compiles without inTransaction() and crashes the first time Room needs it. Treat every default member of a driver or connection interface you wrap as something you must forward on purpose.

What I would check first next time

No crash, no ANR, process alive: stop reading code, send SIGQUIT, read /data/anr/trace_00.
A Room frame in a blocked stack: check which driver is in the builder before anything else. PassthroughConnectionPool in the trace means one connection for the whole process.
After changing the driver: re-assert every pragma you rely on, on a writer connection, not in the callback that sets it.
After wrapping a driver or connection: list the interface’s default members and forward all of them.
Before blaming your code: read the release notes for the exact library version you are on. The answer was in a one-line entry about @Transaction DAO wrapper suspend functions, and it took me longer to find than the thread dump did.

Versions this was reproduced and fixed on

Component	Version
`androidx.room3:room3-runtime`	3.0.2 (current stable at time of writing: 3.0.3)
`androidx.sqlite:sqlite`	2.7.0 (current: 2.7.1)
Kotlin	2.4.10
`minSdk` / `compileSdk`	26 / 37
Emulator	Android 16

How to never pay a generative API twice: a durable task table, not a retrying job queue

BitPage — Sat, 26 Sep 2026 07:40:00 +0000

Short answer: the moment you might have been charged is not the provider’s response, it is the instant before the request leaves your process. Write that instant to a database row before the HTTP call, and a worker that dies mid-request becomes recoverable without guessing: nothing on the row means nobody paid and the task can go back to the queue; a saved request id means poll and never resubmit; the boundary written with no id means a third terminal state that no automation is allowed to retry.

TL;DR

Fix: own task table with a lease, and a submit_started_at column written before the paid call. A queue retries by design; pg-boss says so in its own docs: “the job can run twice and handlers should be idempotent.”
Alternative: a provider-side idempotency key, if your provider has one. Stripe does. The OpenAI and Anthropic clients ship maxRetries: 2 and send no idempotency header at all.
Trap: treating an expired deadline as a failure. A poll that ran out of patience is “I don’t know”, and the difference costs money every time you get it wrong.

Why a job queue is the wrong unit of recovery for a paid call

Because retrying is the entire point of a job queue, and for a paid POST a retry is a second charge. I started by running generation steps as ordinary jobs on the queue the rest of the system already used, pg-boss, because that is where every other background task lived. That lasted until I read the queue’s own documentation on what happens when a worker looks dead:

A worker that only looked dead (a network partition, a stalled event loop) keeps running its handler after its job is failed this way, so the job can run twice and handlers should be idempotent.

That is the pg-boss queue reference, and it is being honest rather than apologetic. Every durable queue has the same sentence somewhere, because a queue cannot tell a dead worker from a slow one. The contract it offers is at-least-once, and it tells you to make the handler idempotent. For sending an email, fine. For a handler whose first line spends money at an external provider, “make it idempotent” is the hard part being handed back to you.

The defaults make it concrete. retryLimit is 2, so a failing job runs three times. expireInSeconds defaults to 15 minutes, after which an active job is retried or failed regardless of what the worker is doing. Image generation lands in tens of seconds and video generation in minutes, so a long clip can sail past the expiration while it is perfectly healthy, get failed by the monitor, and get retried into a second charge. You can raise the expiration to 24 hours, the documented maximum, but then a genuinely dead worker sits undetected for a day.

There is a second problem that is not about money. A job that spends twenty minutes polling a remote id occupies a worker slot for twenty minutes. Concurrency on the queue is shared with everything else in the system, and a handful of slow generations will starve the short jobs that the queue exists for.

The unit of recovery had to stop being a job and become a row.

What RFC 9110 actually says about retrying a POST

It says do not, unless you can prove one of two things, and the second one is the design brief for the whole task table. RFC 9110, section 9.2.2:

A client SHOULD NOT automatically retry a request with a non-idempotent method unless it has some means to know that the request semantics are actually idempotent, regardless of the method, or some means to detect that the original request was never applied.

Two escape hatches. The first one, knowing the semantics are idempotent, is what a provider gives you with an idempotency key. Most generative APIs do not give you one, which I will get to. The second, “some means to detect that the original request was never applied,” is not a property of the protocol at all. It is a property of what you wrote down locally before you made the call.

The RFC also has a line about clients that skip both:

Some clients take a riskier approach and attempt to guess when an automatic retry is possible. For example, a client might automatically retry a POST request if the underlying transport connection closed before any part of a response is received, particularly if an idle persistent connection was used.

That paragraph describes the official clients of the largest paid APIs in the world, which is the subject of the next section.

Do the generative APIs give you an idempotency key?

Mostly no, and the clients retry your POSTs anyway. I went looking for this expecting a boring answer and got a much sharper one by reading the SDK sources instead of the marketing pages.

API	Idempotency key on a POST	Official client retries a POST by default
Stripe	`Idempotency-Key`, first result stored and replayed	not examined
OpenAI	none sent by `openai-node`, see below	yes, `maxRetries: 2` on 408, 409, 429, ≥500, timeouts
Anthropic	no occurrence of the word in `anthropic-sdk-typescript`	yes, `maxRetries: 2` on 408, 409, 429, ≥500, timeouts
Replicate	no occurrence in the published OpenAPI document	not examined

Stripe is the benchmark for how this is supposed to work, and its wording is worth quoting because it names the property that matters:

Stripe’s idempotency works by saving the resulting status code and body of the first request made for any given idempotency key, regardless of whether it succeeds or fails. Subsequent requests with the same key return the same result, including 500 errors.

That is the Stripe API reference on idempotent requests. Note the retention rule in the same page: keys can be pruned once they are at least 24 hours old, and a key reused after pruning generates a new request. Even the gold standard has an expiry after which your safety net is gone, which is a good reminder that the client side still has to keep its own record.

The OpenAI side is where it gets strange. The OpenAI Node client carries the full apparatus for idempotency keys. It declares the field, it has a key generator whose default value announces where the machinery came from, and it builds the header:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


protected idempotencyHeader?: string;

protected defaultIdempotencyKey(): string {
 return `stainless-node-retry-${uuid4()}`;
}

// ...later, while assembling headers:
let idempotencyHeaders: HeadersLike = {};
if (this.idempotencyHeader && method !== 'get') {
 if (!options.idempotencyKey) options.idempotencyKey = this.defaultIdempotencyKey();
 idempotencyHeaders[this.idempotencyHeader] = options.idempotencyKey;
}

idempotencyHeader is declared and never assigned anywhere in the file. The condition is therefore always false and the header is never sent. RequestOptions even exposes idempotencyKey with the comment “A unique key for this request to enable idempotency”, and setting it accomplishes nothing, because there is no header name to put it in. I spent a while convinced I had missed an assignment in a subclass. There is none.

Meanwhile the retry predicate in the same file does not look at the method at all:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


private async shouldRetry(response: Response): Promise<boolean> {
 const shouldRetryHeader = response.headers.get('x-should-retry');
 if (shouldRetryHeader === 'true') return true;
 if (shouldRetryHeader === 'false') return false;

 if (response.status === 408) return true; // request timeouts
 if (response.status === 409) return true; // lock timeouts
 if (response.status === 429) return true; // rate limits
 if (response.status >= 500) return true; // internal errors
 return false;
}

DEFAULT_TIMEOUT in that client is 600000, ten minutes, and a timeout goes down the same retry path. The Anthropic TypeScript client has a byte-for-byte equivalent predicate, the same maxRetries: 2 and the same ten-minute default, and zero occurrences of the string idempot in the entire file.

The default behaviour of both clients, then: send a paid POST, and if the server returns 500 or the socket goes quiet for ten minutes, send it again, up to two more times, with no way for the server to recognise it as the same request. Whether the first attempt was billed is between you and the provider’s meter. The client has no way to ask, and neither do you.

This is not a scandal. It is the reasonable default for an API where most calls are cheap and a dropped connection is more likely than a silently successful one. It stops being reasonable the moment a single call costs real money and the user can see the balance.

The boundary is before the request, not after the response

The instant that matters is the last moment before the send, not the first moment after the answer. This is the one idea the whole design rests on, and it took me an embarrassing amount of staring at a state machine to see it.

I had been drawing the boundary in the obvious place: the task is safe until the provider answers, and becomes committed when it answers. That is wrong, and the reason is that the answer is exactly the thing you do not have when the worker dies. A crash between “socket write completed” and “response parsed” leaves no evidence at all on the client side, and it is the most likely crash window there is, because that window is where all the waiting happens.

Flip it. The task is safe until you begin the request, and becomes unknown the instant you do. That boundary is knowable in advance, so it can be written down in advance:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


async function runImageTask(task: Task, ports: Ports): Promise<void> {
 if (task.status === "SUBMITTED" && task.remoteRequestId) {
 return pollUntilDone(task, ports); // already paid for, never resubmit
 }

 await ports.repo.markSubmitStarted(task.id); // the boundary
 const { requestId } = await ports.provider.submit(task.input);
 await ports.repo.recordSubmitted(task.id, requestId); // now it is addressable

 return pollUntilDone(task, ports);
}

Three writes, and every crash window between them maps to exactly one recovery action. markSubmitStarted is a separate round trip to the database before a call that is about to take thirty seconds, so the cost of the extra write is not worth discussing.

The one rule this imposes on the rest of the code is that nothing may sit between markSubmitStarted and the actual send. No input validation, no prompt assembly, no lazy credential fetch. Anything that can fail there produces a task parked in the unknown state for no reason, and parking a task costs a human’s attention later. Build the request fully, then mark, then send.

Three recovery branches after an expired lease

A supervisor picking up a task whose lease has expired has exactly three cases, and the row tells it which one without any guessing:

What is on the row	What happened	Recovery	Money
`status = RUNNING`, no `submit_started_at`	Died before the request went out	Back to `PENDING`	Nothing spent, safe to redo
`status = SUBMITTED`, `remote_request_id` set	Died after the provider acknowledged	Poll the saved id, never submit	Already spent, result is retrievable
`status = RUNNING`, `submit_started_at` set, no id	Died inside the request	`OUTCOME_UNKNOWN`, stop	Might be spent, nobody can tell

In code it is short enough to fit in a screenshot, which is the point:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


type Recovery =
 | { action: "requeue" }
 | { action: "poll" }
 | { action: "park"; status: "OUTCOME_UNKNOWN" };

function recoveryFor(task: Task): Recovery {
 if (task.status === "SUBMITTED" && task.remoteRequestId) return { action: "poll" };
 if (task.submitStartedAt === null) return { action: "requeue" };
 return { action: "park", status: "OUTCOME_UNKNOWN" };
}

Order matters in that function and it is not the order I wrote first. My first version checked submitStartedAt before the request id, which sends an already-submitted task back through the submit path whenever both columns are set, which is always for a healthy submitted task. The request id is the strongest evidence on the row, so it gets tested first.

Why OUTCOME_UNKNOWN has to be terminal

Because every automatic way out of it is a bet placed with someone else’s money. This was the design decision I argued with myself about longest, since a state that requires a human is an operational cost forever.

The two tempting exits both fail. Retrying assumes the request never landed, and if it did land you have paid twice and the user sees one result. Failing assumes the request never landed too, just with a nicer label: the task disappears from the pipeline, the charge does not disappear from the invoice, and the reconciliation happens weeks later when someone asks why the numbers do not match. Both are the same guess wearing different clothes.

The state is therefore terminal, the UI says the outcome could not be determined and suggests checking the provider’s dashboard, and a human presses the button. In exchange, the system never spends money on its own initiative. For a paid API that is the correct side of the error to be on, and the volume is low enough that it stays correct: the branch only fires when a process dies inside the send window, which is a few seconds out of a generation that takes minutes.

The same logic applies to deadlines, and this is where I got it wrong first. Put a cap on polling, because a remote id that never resolves cannot hold a slot forever. My original code marked the task FAILED when the cap was reached. That is the same lie as before: the provider might still be working, or might have finished and billed while my poll loop gave up. An exhausted deadline is not a failure, it is an unknown outcome that happens to be slow. It goes to the same terminal state, with a different reason string.

Claiming work with FOR UPDATE SKIP LOCKED

The claim is a single statement, and PostgreSQL documents this exact use case. From the SELECT reference:

With SKIP LOCKED, any selected rows that cannot be immediately locked are skipped. Skipping locked rows provides an inconsistent view of the data, so this is not suitable for general purpose work, but can be used to avoid lock contention with multiple consumers accessing a queue-like table.

“Inconsistent view of the data” is the point, not a caveat. Two supervisors claiming work must not see the same row, and neither needs an accurate count of what is pending.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


update generation_task t
 set lease_owner = $1,
 lease_expires_at = now() + $2::interval,
 status = case when t.status = 'PENDING' then 'RUNNING' else t.status end
 where t.id in (
 select id
 from generation_task
 where kind = $3
 and (status = 'PENDING'
 or (status in ('RUNNING', 'SUBMITTED') and lease_expires_at < now()))
 order by created_at
 limit $4
 for update skip locked
 )
returning t.*;

The case matters, and this is the part I got wrong. An earlier version of this statement set status = 'RUNNING' unconditionally, which erases the one signal that tells the executor to poll instead of submit. I rebuilt both versions on a scratch PostgreSQL 17.11 database to be sure I was not misremembering, and the buggy claim does exactly what you would fear:

 status | remote_request_id
---------+-------------------
RUNNING | req-abc

The remote_request_id survives, the status does not. A recovered task that had already been paid for comes back looking like a fresh one. The guard at the top of runImageTask still catches it, because that guard checks the id as well, but at that point the query had quietly become the only thing standing between a restart and a double charge, and I would rather two things stood there.

On the same scratch database, SKIP LOCKED behaves as advertised: with three eligible rows and one of them held by a concurrent SELECT ... FOR UPDATE in another session, the claim returned 2 rows immediately instead of blocking on the third. That is the whole reason the clause exists, and it is worth confirming with your own eyes once, because the alternative failure mode, a supervisor tick that silently waits on a lock, looks identical to a supervisor tick that has nothing to do.

Concurrency is per kind, not global, because the kinds have nothing in common. Take four concurrent image slots, three for clips, two for speech and exactly one for the final local render: the remote ones are limited by the provider’s rate limits and the local one is limited by CPU. Those particular numbers are arbitrary and yours will differ. The structure is not arbitrary: one global limit for all four would either starve the cheap fast work or oversubscribe the machine.

Heartbeat, because not every step polls

A lease needs its own renewal, separate from whatever the work is doing. I learned this from a step that had no polling at all.

Remote generation renews the lease for free as a side effect: the poll loop runs every few seconds, and touching the row at the same time costs nothing. Local work has no such loop. The final render is one long call into ffmpeg that returns when it returns, and speech synthesis for a short passage is a single request that either answers or does not. Both of them held a lease that nobody was extending, so the supervisor watched the lease expire on a task that was working perfectly, decided the worker was dead, and started recovery on a row that was still being written to.

The fix is a timer independent of the work: renew at one third of the lease duration, so two consecutive misses still leave a margin before expiry. The ratio is what matters, not the absolute value. pg-boss reaches the same conclusion from the other direction and documents it as a separate mechanism from expiration, with a floor of 10 seconds on heartbeatSeconds and a monitor that only notices staleness on its own interval, 60 seconds by default. Detection time is the sum of the two, which is worth internalising before you tune the lease down to something clever.

Pinning the order with a test

The order of those three writes is load-bearing and invisible, so it gets asserted directly rather than implied by behaviour. The executor takes its ports as an interface, the test passes recording fakes, and the assertion is on the call log:

1
2
3
4
5
6
7


expect(calls).toEqual([
 "markSubmitStarted",
 "submitImage",
 "recordSubmitted",
 "poll",
 "complete",
]);

Then the recovery case, which is the one that actually costs money if it regresses:

1
2
3


// a task restored in SUBMITTED state with a saved request id
expect(calls).not.toContain("submitImage");
expect(calls).toEqual(["poll", "complete"]);

A test that asserts on a sequence of call names is usually a smell, because it pins implementation detail rather than behaviour. Here the sequence is the behaviour. Any refactor that moves markSubmitStarted after submitImage still passes every functional test, produces identical output on the happy path, and reintroduces the exact bug the design exists to prevent. The only way to catch it is to say out loud that the write comes first.

The price in the docs is not the price on the invoice

Store two numbers per task, estimated and actual, because the first one will be wrong. I built the cost display off a model catalogue populated from provider documentation, showed users an estimate before they pressed the button, and felt good about it. The first real run billed noticeably more than the catalogue said. Take a step documented at four cents and invoiced at six: that is half again as much, multiplied by every step in a job. An estimate that far off is worse than no estimate, because people plan around it.

The catalogue was not lying, it was quoting a headline number that did not include what the actual call resolved to. The correction is structural rather than a better catalogue: cost_estimated comes from the provider’s quote endpoint at submit time, cost_actual comes from the billing figure that arrives with the result, and the user-facing total is the sum of whichever is available per task. When the two disagree the actual value wins and the catalogue entry gets fixed. Documentation prices are for capacity planning. Billing figures are for arithmetic.

Bottom line

If a background task spends money at an external provider, the queue’s retry is no longer a reliability feature and the row, not the job, has to be the unit of recovery. Write the “I am about to spend money” boundary to the database before the request goes out, save the provider’s request id the moment it exists, and accept a third terminal state for the narrow window in between, where neither retrying nor failing is honest. The RFC has been describing this for years under “some means to detect that the original request was never applied”, and the means is a column.

Primary sources worth reading in full: RFC 9110 §9.2.2 on idempotent methods, the PostgreSQL SELECT reference on SKIP LOCKED, the pg-boss queue documentation on heartbeats versus expiration, Stripe on idempotent requests, and the retry predicates in openai-node and anthropic-sdk-typescript, which are worth twenty minutes of anyone’s time who is sending paid POSTs through them.

curl error 60: the message text fingerprints the TLS backend, not your certificate

BitPage — Mon, 21 Sep 2026 05:00:00 +0000

Short answer: curl: (60) means one thing only: the peer’s certificate did not verify. Everything after the colon is written by whichever TLS library curl was compiled against, so Peer certificate cannot be authenticated with known CA certificates, SSL certificate problem: self-signed certificate in certificate chain and SSL certificate verification failed: certificate signer not trusted can all be the same broken trust store on three different boxes. Run curl -V before you search for the message, because the message identifies the build and not the fault.

TL;DR

Fix: read curl -V first. The backend name in that line decides how every TLS failure will be phrased, and searching the phrase without it sends you to answers written for a different library.
Detail: error 60 is CURLE_PEER_FAILED_VERIFICATION in curl.h, and the old name CURLE_SSL_CACERT is now a #define onto it. The two wordings you find online are the same number in different eras of the same file.
Trap: the number is not stable either. wolfSSL returns CURLE_SSL_CACERT_BADFILE, error 77, for a missing signer, where OpenSSL returns 60.
Trap: refreshing a CA bundle removes roots as well as adding them. Between the March 2018 and August 2026 Mozilla extracts published by curl, 80 roots left and 68 arrived, and only 53 of the original 133 survived.
Trap: if the application sets CURLOPT_SSL_VERIFYPEER to 0, none of this appears in its logs, because the CA certificates are never loaded at all.

What curl error 60 actually means

Error 60 is a single enum value with a single meaning, and it carries no detail. In include/curl/curl.h it reads:

1
2


 CURLE_PEER_FAILED_VERIFICATION, /* 60 - peer's certificate or fingerprint
 was not verified fine */

Further down the same header there is a compatibility line that explains half the confusion in every search result you will ever read on this:

1

#define CURLE_SSL_CACERT CURLE_PEER_FAILED_VERIFICATION

CURLE_SSL_CACERT used to be its own code. It is now an alias. So when one Stack Overflow answer talks about CURLE_SSL_CACERT and another talks about CURLE_PEER_FAILED_VERIFICATION, they are discussing the same integer, seen from different decades. Nothing about the actual chain is encoded in it.

The same failure, six different sentences

One untrusted root produces a different sentence in every backend, because each backend writes its own failf() string. These are lifted from lib/vtls/ in curl’s tree:

Backend	What curl prints	Exit code
OpenSSL 3.x	`SSL certificate problem: self-signed certificate in certificate chain`	60
OpenSSL 1.1.1	`SSL certificate problem: self signed certificate in certificate chain`	60
GnuTLS	`SSL certificate verification failed: certificate signer not trusted. (CAfile: ... CRLfile: ...)`	60
mbedTLS	`peer certificate could not be verified`	60
wolfSSL	`CA signer not available for verification`	77
Schannel	`schannel: SNI or certificate check failed: ...`	60
NSS, curl 7.61.1 and older	`Peer certificate cannot be authenticated with known CA certificates`	60

Three things in that table deserve a second look.

The wolfSSL row is not a typo. In lib/vtls/wolfssl.c, the branch for ASN_NO_SIGNER_E does this:

1
2
3
4
5


 else if(detail == ASN_NO_SIGNER_E) {
 if(conn_config->verifypeer) {
 failf(data, " CA signer not available for verification");
 return CURLE_SSL_CACERT_BADFILE;
 }

CURLE_SSL_CACERT_BADFILE is 77, documented in the header as “could not load CACERT file, missing or wrong format”. On a wolfSSL build, a trust store that loads perfectly but lacks the right root reports itself as a problem with the file. If your monitoring alerts on curl exit codes, that single branch is enough to make a fleet look like it has two unrelated incidents.

The GnuTLS row is the only one that tells you where it looked. It prints CAfile and CRLfile in the message itself. On OpenSSL you have to ask for that separately with -v, and most people never do, which is how you end up with an hour of arguing about a certificate when the actual question was which file curl opened.

The NSS row is the odd one out for a different reason, covered below: that sentence is not written by a backend at all.

Where the OpenSSL wording comes from, and why it changed

curl does not write the OpenSSL message. It writes a prefix and then asks OpenSSL to fill in the rest. From lib/vtls/openssl.c:

1
2
3
4
5
6


 lerr = SSL_get_verify_result(octx->ssl);
 if(lerr != X509_V_OK) {
 ssl_config->certverifyresult = lerr;
 failf(data, "SSL certificate problem: %s",
 X509_verify_cert_error_string(lerr));
 }

That %s comes straight out of OpenSSL’s crypto/x509/x509_txt.c, which is a plain switch over the X509_V_ERR_* codes. Which means the tail of the message follows OpenSSL’s source tree, not curl’s. In the 1.1.1 branch the entry reads:

1
2


 case X509_V_ERR_SELF_SIGNED_CERT_IN_CHAIN:
 return "self signed certificate in certificate chain";

In current OpenSSL the same case returns "self-signed certificate in certificate chain". One hyphen. The X509_V_ERR_* code is identical, the condition is identical, the string is different. Search engines treat those as two different phrases, so quoting the message verbatim silently filters your results by OpenSSL major version, and the filter is invisible because nothing in either sentence mentions a version.

The neighbouring entry in that file is the one worth memorising, because it is what openssl verify gives you when a chain cannot be completed:

1
2


 case X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT_LOCALLY:
 return "unable to get local issuer certificate";

Where the NSS wording came from, and why you will not see it again

Peer certificate cannot be authenticated with known CA certificates is not an NSS string at all, and it is not written by any backend. It is curl’s generic fallback text, and you get it precisely because the NSS backend declined to describe the failure.

Look at what the backend does in the 7.19.7 tree, in lib/nss.c:

1
2
3
4
5
6
7
8
9


 if(SSL_ForceHandshakeWithTimeout(connssl->handle,
 PR_SecondsToInterval(HANDSHAKE_TIMEOUT))
 != SECSuccess) {
 if(conn->data->set.ssl.certverifyresult == SSL_ERROR_BAD_CERT_DOMAIN)
 curlerr = CURLE_PEER_FAILED_VERIFICATION;
 else if(conn->data->set.ssl.certverifyresult!=0)
 curlerr = CURLE_SSL_CACERT;
 goto error;
 }

No failf(). It picks an error code and jumps. The label it jumps to hides the one useful fact behind the verbose flag:

1
2
3


error:
 err = PR_GetError();
 infof(data, "NSS error %d\n", err);

infof only prints under -v. So with no error buffer set, the curl tool falls back to curl_easy_strerror(), which in lib/strerror.c of that era said:

1
2


 case CURLE_SSL_CACERT:
 return "Peer certificate cannot be authenticated with known CA certificates";

That sentence is therefore a description of an enum constant, printed by a tool that was never told what went wrong. The actual reason was sitting one -v away as a bare negative integer.

Those integers come from NSS’s lib/util/SECerrs.h, where every entry is an offset from SEC_ERROR_BASE, defined as -0x2000 in secerr.h. The one you usually want is

1
2


ER3(SEC_ERROR_UNKNOWN_ISSUER, SEC_ERROR_BASE + 13,
 "Peer's Certificate issuer is not recognized.")

which arrives on the wire as NSS error -8179. The one that derails people is SEC_ERROR_UNRECOGNIZED_OID at SEC_ERROR_BASE + 143, or -8049, phrased "Unrecognized Object Identifier." It reads like a parsing bug and usually means an old NSS met an algorithm identifier that postdates it. Neither sentence resembles the curl wording they end up wrapped in, which is the entire problem.

The wording has a hard version boundary. CURLE_SSL_CACERT stopped being a distinct code in curl 7.62.0, released 31 October 2018, and the header still says so in a comment above the alias:

1
2


/* added in 7.62.0 */
#define CURLE_SSL_CACERT CURLE_PEER_FAILED_VERIFICATION

The strerror.c case went with it. Current curl has only CURLE_PEER_FAILED_VERIFICATION: "SSL peer certificate or SSH remote key was not OK". So a box printing the old sentence is running curl 7.61.1 or older, full stop, and that is a much more useful thing to learn from an error message than anything it says about certificates. curl 7.19.7, the build that shipped in enterprise Linux images of that generation, was released on 4 November 2009 and carries 72 published vulnerabilities in curl’s own release table.

NSS itself is gone from curl too. The deprecation list records it under past removals:

- NSS (removed in 8.3.0)

and the 8.3.0 changelog, dated 13 September 2023, carries the one-liner nss: remove support for this TLS library. Two lines further down the same list sits OpenSSL 1.1.1 and older (removed in 8.18.0), which means the unhyphenated self signed wording has already stopped being reachable on any curl built after January 2026. Give it a few years and that phrase becomes a version fingerprint exactly the way the NSS one is now.

Why openssl s_client said the certificate was fine

Because openssl s_client and curl were not reading the same store. That is the whole trick, and it is documented rather than folklore. curl’s SSL library comparison puts it in the glossary:

For engines that use a database and do not also support files, the CURLOPT_CAINFO option is ignored.

A curl linked against a database-backed library looks in that database. openssl s_client looks at OPENSSLDIR or whatever you pass to -CAfile. Populate one and not the other and you get exactly the standoff I walked into: Verify return code: 0 (ok) from s_client, error 60 from curl, same host, same second, same machine.

Even on a modern box with both tools on OpenSSL, the defaults differ in shape. On the machine I am writing this on:

$ curl -V
curl 8.14.1 (x86_64-pc-linux-gnu) libcurl/8.14.1 OpenSSL/3.5.7 ...
$ curl -sSv https://untrusted-root.badssl.com/ -o /dev/null 2>&1 | grep -i ca
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs

One is a single concatenated file of 224,449 bytes, the other a directory holding 301 hash-named symlinks. Both are generated from the same package, and they do drift apart the moment somebody edits one of them by hand.

The reproduction, for what it is worth, is two lines:

$ curl -sS https://untrusted-root.badssl.com/ -o /dev/null
curl: (60) SSL certificate problem: self-signed certificate in certificate chain
$ echo | openssl s_client -connect untrusted-root.badssl.com:443 \
-servername untrusted-root.badssl.com 2>/dev/null | grep Verify
Verify return code: 19 (self-signed certificate in certificate chain)

Note what each tool chose to show. OpenSSL gives you the X.509 reason code, 19. curl gives you 60, which is the same for every reason there is, and pushes the actual reason into prose. If you only ever copy one habit out of this article, make it running s_client alongside curl so you get the number.

The first command is curl -V

curl -V answers the question the error message is pretending to answer. It names the backend, the version, and therefore the entire vocabulary of failures you are about to see:

curl 7.19.7 (x86_64-...) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3

Two seconds of reading and the rest of the incident is bounded. That line says the box will phrase trust failures in curl’s own pre-7.62.0 vocabulary, that it consults an NSS database rather than a PEM file for its roots, and that CURLOPT_CAINFO may therefore be ignored outright. It also sets an upper bound on the protocol side of the conversation, which matters when a partner turns off older TLS: for the OpenSSL half of the world, TLS 1.3 did not exist before OpenSSL 1.1.1 on 11 September 2018, so any libcurl/x OpenSSL/1.0.x line is a box that cannot negotiate it no matter what you put in the trust store.

I did not run curl -V first. I spent a while comparing the certificate chain the browser showed against the one s_client printed, found them identical, and concluded the server was fine. Which it was. The wasted time came from treating the error text as a description of the situation rather than as a property of the binary printing it.

How stale is a stale trust store: 2018 against 2026

Stale is not a feeling, so I measured it. curl publishes dated Mozilla extracts at curl.se/docs/caextract.html, which makes this a two-command experiment anyone can repeat:

1
2
3


curl -sO https://curl.se/ca/cacert-2018-03-07.pem
curl -sO https://curl.se/ca/cacert.pem
grep -c 'BEGIN CERTIFICATE' cacert-2018-03-07.pem cacert.pem

Counting is not enough, because a root can be replaced by a re-issued root under the same common name. So compare SHA-256 fingerprints:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


import re, subprocess

def fingerprints(path):
 blocks = re.findall(
 r'-----BEGIN CERTIFICATE-----.*?-----END CERTIFICATE-----',
 open(path).read(), re.S)
 out = set()
 for b in blocks:
 p = subprocess.run(
 ['openssl', 'x509', '-noout', '-fingerprint', '-sha256'],
 input=b.encode(), capture_output=True)
 out.add(p.stdout.decode().strip())
 return out

old = fingerprints('cacert-2018-03-07.pem')
new = fingerprints('cacert.pem')
print(len(old), len(new), len(old & new), len(old - new), len(new - old))

The two bundles I pulled while writing this carry these headers:

## Certificate data from Mozilla as of: Wed Mar 7 04:12:06 2018 GMT
## Certificate data from Mozilla as of: Thu Aug 13 03:12:01 2026 GMT

And the result:

Measure	March 2018 extract	August 2026 extract
Roots in the file	133	121
File size	215,556 bytes	188,900 bytes
Fingerprints present in both	53	53
Fingerprints unique to this file	80	68

Eight and a half years, and the headline count moved by twelve. Underneath, 148 of the 201 distinct roots changed sides. DST Root CA X3 is in the 2018 file and not the 2026 one. ISRG Root X2, GTS Root R1, GTS Root R3, GTS Root R4, GlobalSign Root CA - R6 and Sectigo Public Server Authentication Root R46 are in the 2026 file and not the 2018 one. If you have ever diffed a trust store by line count and concluded nothing much happened, that table is why the method does not work.

Cross-signing is why a frozen bundle keeps working

Here is the part that makes a stale trust store so hard to spot: it keeps working almost everywhere. I pointed the 2018 bundle at a spread of large sites in 2026 and every one of them verified:

1
2
3
4
5


for h in www.google.com github.com www.cloudflare.com letsencrypt.org \
 www.microsoft.com www.gov.uk www.sectigo.com; do
 curl -sS --cacert cacert-2018-03-07.pem "https://$h/" -o /dev/null
 echo "$h -> $?"
done

Seven hosts, seven zeroes. A bundle whose root data predates the pandemic still validates the mainstream web, because CAs cross-sign their new roots against their old ones for exactly this reason. The new hierarchy ships a chain that terminates at something your ancient store already trusts.

That is also why the failure mode looks arbitrary when it finally arrives. It does not arrive when the roots change. It arrives when one particular operator retires one particular cross-signature, and then a single endpoint out of many starts refusing while everything else on the box is fine. Take eleven outbound integrations on a host and two of them failing: that is not a network problem and it does not look like a trust problem either. It looks like those two partners broke something.

My first theory was in fact the opposite of the truth. I assumed the two failing endpoints had moved to a new root hierarchy that my store did not know, and that the cross-signed path was the thing missing. The test for that is openssl verify with the intermediates supplied by hand. The manual is precise about the flag:

-untrusted filename | uri A file or URI of untrusted certificates to use for chain building.

1
2


openssl verify -CAfile old_root.pem \
 -untrusted inter1.pem -untrusted inter2.pem leaf.pem

That returned OK, which killed the theory. The control run, the same command with the cross-certificate left out, gave:

error 20 at 1 depth lookup: unable to get local issuer certificate

Error 20 is X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT_LOCALLY, the string quoted earlier from x509_txt.c. Two runs, a positive and a negative, and a theory I had spent half an hour on was gone. Do the negative control. Without it, verify returning OK proves nothing except that you handed it enough certificates.

Updating a bundle removes roots too

update-ca-trust and apt install ca-certificates are not additive operations, and the 80-roots-removed column above is the size of the risk. Replacing a trust store on a machine that talks to eleven partners can fix two endpoints and break a third, and the third failure will surface hours later when some cron job runs.

Before swapping anything, test the candidate store side by side against the old one:

1
2
3
4
5
6
7


while read -r host; do
 old=$(curl -sS -m 10 --cacert /etc/pki/tls/certs/ca-bundle.crt \
 "https://$host/" -o /dev/null; echo $?)
 new=$(curl -sS -m 10 --cacert /tmp/candidate-bundle.pem \
 "https://$host/" -o /dev/null; echo $?)
 printf '%-40s old=%s new=%s\n' "$host" "$old" "$new"
done < endpoints.txt

You want the new column to be a superset of the successes in the old column. A host that went from 0 to 60 is a regression, and finding it in a loop that takes a minute is better than finding it in a support ticket.

Build endpoints.txt from whatever the application actually dials, not from what the documentation says it dials. Grep the config, grep the code, grep an hour of egress logs. On the box in front of me the config listed fewer hosts than the code did, which is the normal state of affairs: somebody hardcoded a status endpoint years ago and nobody wrote it down.

The root that no public bundle will ever ship

Refreshing ca-certificates fixes the roots Mozilla ships, and does nothing at all for the roots it does not. Those are more common than the conversation around trust stores suggests:

an internal corporate CA, for services that never had a public certificate
a partner’s private PKI, where the integration contract specifies their root
a TLS-inspecting proxy sitting between you and the internet, minting certificates from its own CA
a national or sector CA that is outside the browser root programs by design

None of these will ever arrive through a package update, no matter how many times you run the update. If one of your failing endpoints is signed by such a root, the bundle refresh will not touch it, and you will conclude the refresh did not work. It worked. It was aimed at a different layer.

This is worth separating in your head before you start, because the two layers have different fixes and different blast radii. Refreshing the public set is a bulk operation with regression risk. Adding one private anchor is surgical and affects exactly the chains that terminate at it.

Where to put an anchor so the legacy path sees it

Put the PEM in the anchors directory and run the extract, then check that the file your application actually reads is a symlink. The update-ca-trust(8) manual gives the first half:

QUICK HELP 1: To add a certificate in the simple PEM or DER file formats to the list of CAs trusted on the system: add it as a new file to directory /etc/pki/ca-trust/source/anchors/ run update-ca-trust extract

1
2


cp partner-root.pem /etc/pki/ca-trust/source/anchors/
update-ca-trust extract

The second half is the trap, and the same manual spells it out under “Legacy Files”:

The following file paths were used in legacy versions of the utility and have since been replaced. Scripts and configurations referencing these old paths should be updated.

/etc/pki/tls/certs/ca-bundle.crt This file has been replaced by /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem.

The mechanism that keeps old software working is described a few paragraphs earlier: “the classic filenames have been changed to symbolic links”. On a healthy system /etc/pki/tls/certs/ca-bundle.crt points into /etc/pki/ca-trust/extracted/, so the extract updates it for free. On a system where someone once fixed a certificate problem by writing a bundle straight to that path, it is a regular file, the symlink is gone, and update-ca-trust extract now writes to a location nothing reads.

1

ls -l /etc/pki/tls/certs/ca-bundle.crt

One l at the start of that output or the absence of it decides whether your last twenty minutes accomplished anything. Check it before the extract and after.

The same manual carries the other half of the warning, about the extracted directory:

Please never manually edit the files stored in this directory, because your changes will be lost and the files automatically overwritten, each time the update-ca-trust extract command gets executed.

Which is, of course, exactly how the symlink got replaced by a regular file in the first place. Somebody hit a trust error, wrote the bundle where the error said to write it, and moved on.

The application logged none of this

The reason nobody noticed for so long is that the application had certificate verification turned off, so it never produced the error in the first place. The two options were sitting in the HTTP client’s setup:

1
2


CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_SSL_VERIFYHOST => false,

The documentation for CURLOPT_SSL_VERIFYPEER states what that does to the diagnostics, not just to the security:

When this option is disabled (set to zero), the CA certificates are not loaded and the peer certificate verification is skipped.

The store is not consulted. There is nothing to fail, so nothing is logged, so the trust store can rot for years without producing a single line anywhere. Every failure that got reported came from somewhere else: operational curl calls typed by hand, a cron script that used the system defaults, one integration whose client had been written by someone who left verification on. That is why a couple of endpoints looked broken and the rest looked healthy. The rest were not healthy. They were unchecked.

Sitting right next to those two lines was a third:

1

CURLOPT_CAINFO => $this->basePath . '/cert/ca-bundle.crt',

behind an if ($this->useLocalCerts) flag that was false. The directory it pointed at did not exist. It had not existed for years and nothing had ever complained, which the docs also explain, in CURLOPT_CAINFO:

If CURLOPT_SSL_VERIFYPEER is zero and you avoid verifying the server’s certificate, CURLOPT_CAINFO need not even indicate an accessible file.

A dead path guarded by a false flag feeding an option that is ignored anyway. Three layers of nothing, all of it looking like configuration. I read that block twice before working out it could not possibly matter.

The order the fix has to happen in

Fix the store first, turn verification on second, and never the other way round. The sequence matters because each step has to be observable before the next one lands:

Run curl -V on the affected host and write down the backend. Everything after this depends on it.
Build the endpoint list from config, code and egress logs. Not from documentation.
Run the whole list against the current store and record exit codes. This is your baseline and you will need it.
Fetch a candidate store and run the whole list against it with --cacert. Compare to the baseline. Nothing regresses, or you stop.
Install the candidate store. Verify that the legacy path is still a symlink into the extracted directory.
Add any private anchors to /etc/pki/ca-trust/source/anchors/ and run update-ca-trust extract. Re-run the list.
Only now flip CURLOPT_SSL_VERIFYPEER and CURLOPT_SSL_VERIFYHOST back to true, in a deployment you can roll back.

Doing step 7 before step 5 turns a background problem into an outage. Every integration that was quietly unverified starts verifying against a store that is still wrong, and instead of two failing endpoints you have all of them. The temptation is real, because step 7 is the one that actually fixes the security hole and it is a one-line change. Resist it for the length of one deploy cycle.

Keep the model from earlier: eleven outbound endpoints, two of them failing. Steps 1 through 6 take that to eleven verified and none failing, against a store you have diffed against the one it replaced. On the host that prompted this article the store work was done inside a single sitting and needed no service restart, because nothing in that path caches the bundle across requests. Step 7 shipped separately, the following day, with the baseline from step 3 as the rollback criterion.

How I verified it, endpoint by endpoint

Verification is the loop from earlier, run once more after the change, with the baseline pinned next to it. There is no clever tooling involved and that is the point: a shell loop over a text file of hostnames, exit codes in two columns, and a diff.

What makes it trustworthy is the negative control. Run the same loop with --cacert /dev/null and confirm every line reports 77. If they do not, something in the path is ignoring your --cacert, which on a database-backed backend is the documented behaviour rather than a bug, and you have just learned that your verification loop was proving nothing.

1
2


$ curl -sS --cacert /dev/null https://curl.se/ -o /dev/null
curl: (77) error setting certificate file: /dev/null

That is the third distinct exit code in this article for what a casual reading would call “certificate problem”: 60 for a chain that will not verify, 77 for a store that will not load, and 77 again on wolfSSL for a chain that will not verify. The numbers are not a taxonomy. They are an artefact of which branch in which backend happened to run.

Bottom line

Treat the text of a TLS error as a build identifier. curl -V first, the message second, and the trust store third, because the message is written by the library named in the first command and tells you nothing about the certificate that a chain dump would not tell you better. When you do get to the store, diff it by fingerprint rather than by line count, keep a baseline of exit codes across every endpoint you actually call, and remember that a refresh takes roots away as readily as it adds them: 80 out of 133 between two published Mozilla extracts, eight years apart.

PostgreSQL index bloat: why VACUUM never shrinks an index, and how to measure it with avg_leaf_density

BitPage — Wed, 16 Sep 2026 13:50:00 +0000

Short answer: If a PostgreSQL table’s heap is healthy and its indexes are six times larger than the heap, autovacuum is not broken and never was. VACUUM marks dead index entries reusable and reclaims index pages that become completely empty, but a page holding one surviving key out of several hundred stays allocated forever, because a b-tree does not merge sparse neighbours. Measure it with pgstatindex(...).avg_leaf_density, compare against the b-tree default fillfactor of 90, and rebuild with REINDEX INDEX CONCURRENTLY. VACUUM FULL is the wrong tool: it rewrites the entire table to fix a problem that lives in the indexes.

TL;DR

Fix: REINDEX INDEX CONCURRENTLY <name>, one index at a time. Peak extra space is one new index, not one new table, and the disk comes back after each step.
Diagnose: avg_leaf_density from pgstattuple. A freshly built b-tree sits near 90 because that is its default fillfactor. Anything far below is bloat you can quantify.
Predict: recovered size is roughly current_size x density / 90. It is arithmetic off two numbers you already have.
Don’t: VACUUM FULL, unless the heap is the thing that is bloated. It takes ACCESS EXCLUSIVE and needs free space the size of the whole table.
Trap: an index with idx_scan = 0 is not automatically dead. Uniqueness checks on INSERT do not increment that counter, so unique indexes look unused while they are holding your constraints up.
Trap: pg_stat_* counters are per node. A zero on the primary says nothing about what the replica is serving.

A note on numbers before anything else. The absolute volumes below are a model, because someone else’s disk is not mine to publish. What is real is the shape: the ratio of index size to heap, the densities rounded to whole percent, and every PostgreSQL default and doc quote, which you can check against the links yourself.

The disk alert was about the indexes, not the tables

The page that came in said the root volume was over 80% full, and the fix turned out to be in a place the alert could not see. Take a box with a 125 GB root volume, 101 GB used, 24 GB free, and a threshold that fires under 20% free. It had just crossed.

The space breakdown looked ordinary at first:

What	Size
PostgreSQL data directory	62 GB
Docker images	14 GB (6 GB unreferenced)
Application uploads	9 GB
Application logs	5 GB, no rotation
journald	tens of MB

Logs and stale images are the usual suspects and they are worth about 11 GB here. The database is worth 62. So I went to look at the largest tables, fully intending to VACUUM FULL the worst one.

 relname | total | heap | idx | n_live_tup | n_dead_tup
-----------------+---------+---------+---------+------------+------------
batch_result | 35 GB | 4710 MB | 30 GB | 19840000 | 2600000
batch_cache | 4504 MB | 30 MB | 4474 MB | 0 | 512000

That stopped the plan. A table of roughly 20 million rows carrying 4.6 GB of heap is fine. Its indexes are 30 GB, more than six times the data they point at. The second row is stranger still: zero live rows, 30 MB of heap, and 4.4 GB of indexes on top of nothing.

My first half hour went into a hypothesis one query would have killed. The lesson is cheap to state and I keep relearning it: split pg_relation_size from pg_indexes_size before choosing a tool. Heap bloat and index bloat are different diseases with different cures, and the combined pg_total_relation_size hides which one you have.

Autovacuum was not broken, it was working itself to death

The counters ruled out every standard explanation for “space is not coming back”. Autovacuum was on with default settings, there were no replication slots holding an old xmin, and no long-running transaction was pinning the horizon.

 relname | ins | upd | del | hot | hot_pct | av_cnt
----------------+-------------+-------------+-------------+-----------+---------+--------
batch_staging | 9105000000 | 20300000000 | 0 | 1810000 | 0.0 | 42100
batch_cache | 9160000000 | 0 | 9160000000 | 0 | | 51800
batch_result | 9140000000 | 0 | 9110000000 | 0 | | 7900

Autovacuum had run tens of thousands of times on each of the hot tables. It was not asleep, misconfigured or starved. It was running flat out and the space still was not coming back, which meant the problem was something autovacuum does not do at all.

This is the counterintuitive part and it sends people down a long road of tuning autovacuum_naptime and autovacuum_vacuum_cost_limit. None of those knobs touch the mechanism below. The instinct “space is not returning, so vacuum must be failing” is wrong here in a specific and useful way.

What VACUUM actually does with freed space

VACUUM makes space reusable inside the file; it does not hand it back to the operating system. The routine vacuuming docs are explicit:

The standard form of VACUUM removes dead row versions in tables and indexes and marks the space available for future reuse. However, it will not return the space to the operating system, except in the special case where one or more pages at the end of a table become entirely free and an exclusive table lock can be easily obtained.

That exception almost never fires under a churn workload, because the free pages end up scattered rather than parked at the tail. This is a deliberate trade: VACUUM runs without locking the table, and the price is that df does not move.

VACUUM FULL does return space, and the VACUUM reference spells out what it costs:

This method also requires extra disk space, since it writes a new copy of the table and doesn’t release the old copy until the operation is complete.

On the numbers above that means rewriting 35 GB with 24 GB free, while holding ACCESS EXCLUSIVE, in order to reclaim space that is sitting in the indexes. It would have run the volume to zero and taken the application down on the way.

Why a b-tree never compacts itself

A b-tree reclaims index pages that go completely empty and leaves partially empty ones exactly where they are. That single sentence is the root of the whole incident, and the reindexing docs say it plainly:

B-tree index pages that have become completely empty are reclaimed for re-use. However, there is still a possibility of inefficient use of space: if all but a few index keys on a page have been deleted, the page remains allocated. Therefore, a usage pattern in which most, but not all, keys in each range are eventually deleted will see poor use of space. For such usage patterns, periodic reindexing is recommended.

There is no page merging and no rebalancing on delete. PostgreSQL will not walk two neighbouring leaf pages that are 5% full each and fold them into one. Once density drops it stays dropped, and the only operation that raises it is a full rebuild of the index.

Worth being precise about the failure pattern, because it explains why this bites some workloads and not others. Deleting a contiguous range is survivable, since whole pages empty out and get reclaimed. Deleting most of every range is the bad case, and a batch job that rewrites a table by key is exactly that: it scatters a few survivors across every page it touches.

Why it accumulated so fast: HOT updates at 0.0%

The accelerant was heap-only tuple updates not happening. When an UPDATE leaves every indexed column unchanged and the page has room, PostgreSQL writes the new row version in place and touches no index at all. When it cannot, every index on the table gets a new entry.

On the staging table above, tens of billions of updates produced 1.8 million HOT updates. Rounded to one decimal that is 0.0%. Every one of those updates wrote a fresh entry into each of the table’s indexes, and every superseded entry became a dead key sitting on a page that would never be reclaimed.

The two mechanisms stack. The b-tree not merging pages explains why the bloat is permanent; HOT at zero explains why it arrived within months instead of years. Looking at either one alone makes the numbers seem impossible.

The load profile behind it: roughly nine billion inserts and almost exactly nine billion deletes against a table holding 20 million rows. That is the entire contents of the table rewritten several hundred times over. The cache table is the same story with the ending removed, since it is emptied with DELETE instead of TRUNCATE, which leaves zero live rows and 4.4 GB of index behind.

The table autovacuum had never touched once

Separately, a table can be permanently below the threshold and never get vacuumed at all, which looks identical from the outside. The autovacuum settings docs describe autovacuum_vacuum_scale_factor as “a fraction of the table size to add to autovacuum_vacuum_threshold”, and note that “The default is 0.2 (20% of table size)”.

Take a reporting table of 10 million live rows. The threshold is autovacuum_vacuum_threshold plus that fraction of the table, so 50 plus 2 million dead rows before autovacuum will look at it. This one had accumulated about 240,000, roughly a tenth of what it needed, so its autovacuum_count was 0 and had always been 0. Its HOT ratio was near 70%, so it was not bloating fast, but its dead rows were never being reclaimed and nothing was going to change that on its own.

ANALYZE is the part that still happens, because autovacuum_analyze_scale_factor defaults to 0.1 rather than 0.2 and counts inserts as well as updates and deletes. That is a useful asymmetry to remember when a table looks healthy in pg_stats and has never been vacuumed in its life.

For a big, slowly-churning table, 0.2 is the wrong default. A per-table autovacuum_vacuum_scale_factor = 0.01 is the standard remedy. One caveat if you run Patroni: setting it through ALTER SYSTEM survives until the next restart or failover, because Patroni rewrites postgresql.conf from its own configuration. Cluster-wide values go through patronictl edit-config, and per-table ones through ALTER TABLE ... SET, which lives in the catalog and is safe.

How to measure bloat instead of guessing at it

pgstatindex from the pgstattuple extension reports avg_leaf_density, and the number is directly interpretable because you know what a healthy value is. The CREATE INDEX docs state it: “B-trees use a default fillfactor of 90”. A rebuilt index packs its leaves to 90% and starts drifting down from there.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


CREATE EXTENSION IF NOT EXISTS pgstattuple;

SELECT i.indexrelname,
 pg_size_pretty(pg_relation_size(i.indexrelid)) AS sz,
 round(s.avg_leaf_density::numeric, 1) AS density
FROM (SELECT indexrelid, indexrelname
 FROM pg_stat_user_indexes
 ORDER BY pg_relation_size(indexrelid) DESC
 LIMIT 20) i
JOIN pg_class c ON c.oid = i.indexrelid
JOIN pg_am am ON am.oid = c.relam AND am.amname = 'btree',
LATERAL pgstatindex(i.indexrelid) s
ORDER BY s.avg_leaf_density ASC;

Two things in that query are load-bearing. The amname = 'btree' filter is mandatory, because pgstatindex is documented as returning “information about a B-tree index” and gives you an error or nonsense on anything else. The LIMIT 20 before the join matters because pgstatindex reads the index in full, so this is a scan of every byte of the twenty largest indexes and not a catalog lookup. Run it on a replica where you can. The files are identical and the primary does not pay for it.

What came back:

 indexrelname | sz | density
------------------------------+---------+---------
batch_cache_period_idx | 4416 MB | 0.6
batch_result_ref_idx | 9528 MB | 11.2
batch_result_owner_idx | 7810 MB | 11.3
batch_result_uniq | 6912 MB | 12.8
batch_result_pkey | 3140 MB | 42.0
batch_result_link_idx | 980 MB | 88.6

The bottom row is the control. One index on the same table sits at 88.6, which is where a b-tree lives when nothing pathological is happening to it, and it confirms the other five are not a measurement artifact. The top row is 4.4 GB of index structure at 0.6% density, pointing at a table with no rows in it.

An index at 11% density is a filing cabinet with one sheet of paper in each drawer. The filing is correct and the lookups return the right answer. You just walk nine drawers to find one page, and the cabinet takes nine times the floor space it needs.

Predicting the payoff before you spend the maintenance window

Multiply current size by current density and divide by 90. Since a rebuild targets fillfactor 90 by definition, that ratio is the compaction you should expect, give or take fragmentation and the fixed overhead of a small index.

Index	Size	Density	Predicted	After rebuild
`batch_cache_period_idx`	4416 MB	0.6%	29 MB	54 MB
`batch_result_ref_idx`	9528 MB	11.2%	1186 MB	1204 MB
`batch_result_owner_idx`	7810 MB	11.3%	981 MB	994 MB
`batch_result_uniq`	6912 MB	12.8%	983 MB	1002 MB
`batch_result_pkey`	3140 MB	42.0%	1465 MB	1480 MB

The formula lands within a couple of percent everywhere except the first row, and the first row is instructive. At 0.6% density the prediction is 29 MB and the result is 54 MB, because once the payload approaches nothing you are measuring the metapage, the root and a handful of leaves rather than data. The model is good in the range where the decision is actually hard.

That is enough to walk into a change window with a number instead of a hope. Summed over both tables it predicted around 28 GB of recovery, which is the difference between a ticket for next quarter and a maintenance slot tonight.

REINDEX, not VACUUM FULL

REINDEX rebuilds one index at a time, so peak extra space is the size of one new index rather than one new table.

1
2


REINDEX INDEX <name>; -- fast, takes ACCESS EXCLUSIVE
REINDEX INDEX CONCURRENTLY <name>; -- no write lock, slower, more WAL

REINDEX CONCURRENTLY has been available since PostgreSQL 12, which is why pg_repack was not worth installing here. An external tool for a one-off operation that the server can do natively is extra supply chain for no gain.

Approach	Lock	Extra space needed	Fixes
`VACUUM`	None	None	Nothing on disk, marks space reusable
`VACUUM (FULL, ANALYZE)`	`ACCESS EXCLUSIVE`	Size of the whole table	Heap and all its indexes
`REINDEX INDEX`	`ACCESS EXCLUSIVE` on that index	Size of one index	One index
`REINDEX INDEX CONCURRENTLY`	Reads and writes continue	Size of one index	One index

The trade-off is narrow and worth stating outright. Plain REINDEX is faster and has no failure debris, and on a staging box that is the obvious choice. On a primary serving traffic only CONCURRENTLY is defensible, and you pay for it in runtime, in WAL volume, and in cleanup when it fails. If the heap is the bloated part, VACUUM FULL is right and REINDEX is the wrong answer. Check which one you have first.

The zero-scan index that was holding up the constraints

While the rebuild plan was being written, the tempting side quest appeared: the hot table carried eight indexes and five of them showed idx_scan = 0. Dropping five unused indexes would have solved the disk problem outright.

Two of those five were unique. The monitoring docs define idx_scan as the “Number of index scans initiated on this index”, and a uniqueness check on INSERT is not an index scan in that sense. It does not move the counter. Those two indexes were being consulted on every single insert into the table and their statistics said they had never been touched since the counters were last reset.

Dropping them would have let duplicates into the results of a financial recalculation, silently, with no error at the time of the mistake. This was caught by a rule rather than by a metric: verify the business meaning of every index before dropping it. The metric did not know what the index was for.

The query, with the filter that makes it safe:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


SELECT i.relname AS tbl, i.indexrelname AS idx,
 pg_size_pretty(pg_relation_size(i.indexrelid)) AS sz, i.idx_scan
FROM pg_stat_user_indexes i
JOIN pg_index x ON x.indexrelid = i.indexrelid
WHERE i.idx_scan = 0
 AND NOT x.indisunique
 AND pg_relation_size(i.indexrelid) > 5 * 1024 * 1024
ORDER BY pg_relation_size(i.indexrelid) DESC;

-- a zero means nothing without knowing the window it was counted over
SELECT stats_reset, now() - stats_reset AS window
FROM pg_stat_database WHERE datname = current_database();

Run the second query first. An idx_scan = 0 over a statistics window that was reset last Tuesday is not evidence of anything, and here the window was over half a year, which is what made the zeros worth investigating in the first place.

Statistics counters live on each node separately

A pg_stat_* counter describes the node you read it from, not the cluster. Replicas keep their own, so an index that the primary has never scanned may be serving every read on the standby.

I checked the primary and one standby taken from the inventory file, then had to redo the reasoning properly: the list of nodes has to come from pg_stat_replication on the primary, not from an inventory that may be out of date. The conclusion held, but it had not been proved until the database itself confirmed how many subscribers there were. On a two-node cluster that distinction is easy to wave away, and waving it away is exactly how a busy index gets dropped.

What bites you after the decision is made

Five things caught me on the way through, in rough order of how much time each cost.

_ccnew and _ccold mean opposite things. A failed REINDEX CONCURRENTLY leaves an invalid index behind, and the REINDEX docs distinguish the two cases carefully. A _ccnew suffix is the transient index from a run that failed, so drop it and retry. A _ccold suffix is the original, which means the rebuild succeeded and only the cleanup failed, so drop it and you are done. The docs also note that “A nonzero number may be appended to the suffix of the invalid index names to keep them unique, like _ccnew1, _ccold2”. Any automation should check for leftovers as its first step:

1
2
3


SELECT c.relname
FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid
WHERE NOT i.indisvalid;

A synchronous replica turns REINDEX into application latency. A rebuild generates WAL on the order of the size of the index it produces. With synchronous_commit = on, every commit waits for the standby to confirm, and with wal_compression off that whole volume crosses the network uncompressed. Rebuild one index at a time, off-peak, checking replication lag between steps.

Laravel wraps migrations in a transaction, and only on PostgreSQL. DROP INDEX CONCURRENTLY inside a migration fails with cannot run inside a transaction block, and nothing in the migration file explains why. The reason is three files deep in the framework. Migration.php declares public $withinTransaction = true;, Migrator.php wraps the call when supportsSchemaTransactions() && $migration->withinTransaction, and PostgresGrammar.php sets protected $transactions = true;. The MySQL grammar does not override it, so it inherits false from the base class. The same migration passes on MySQL and fails on PostgreSQL. The fix is one line in the migration class:

1

public $withinTransaction = false;

That property is not in the Laravel documentation. I found it by reading the source, which is generally the faster route once a framework behaviour has no documented name.

Indexes dropped by hand come back. If the drop happened as ad-hoc SQL while the declaration is still sitting in a migration or a test schema, the next deploy to a fresh environment restores it. Grep the repository for the index name before dropping anything, and do the drop as a migration.

Two indexes with different names can be the same index. Matching column lists is not enough to prove it, and differing column order in a UNIQUE constraint does not prove the opposite, because for uniqueness purposes the order of columns means nothing. Compare pg_index properly: indclass, indcollation, indoption, indnullsnotdistinct, indnkeyatts to catch a hidden INCLUDE, and indpred and indexprs for partial and expression indexes. When every one of those matches, it is a duplicate, and you can drop the one with a billion scans on it: the planner moves to the twin without noticing.

What it bought

	Before	After
Disk used	101 GB (81%)	63 GB (50%)
Free space	24 GB	62 GB
PostgreSQL data directory	62 GB	33 GB
`batch_result` total	35 GB	11 GB
Its indexes	30 GB	6.4 GB
Density of the five worst	0.6-42%	89-93%

One detour on the way, for anyone who hits the same thing: after the rebuilds, pg_wal had grown and df had not moved as much as the arithmetic promised. Forcing a CHECKPOINT did nothing, and it should not have. The directory had grown to the configured max_wal_size ceiling, which is the setting doing its job rather than a leak, and PostgreSQL keeps recycled segments preallocated and shrinks back toward min_wal_size gradually. Chasing those gigabytes is wasted time.

The more interesting result was on the production cluster, where the same profile showed up on indexes that are actually hot:

Index	Size	Density	Scans
Tree-walk index on the hierarchy table	1.4 GB	14%	hundreds of millions
Unique on (period, user, sequence)	10 GB	15%	a couple hundred thousand
Primary key of the main table	2.9 GB	42%	hundreds of millions
Composite index on order links	12 GB	71%	hundreds of millions

The first row is the one that changed how I think about this. It is a 1.4 GB index at 14% density taking hundreds of millions of scans, and every one of those scans reads pages that are one seventh full. Rebuilt, it lands near 220 MB and fits in cache whole. The disk saving is real but secondary. The alert said “disk”, and the actual finding was a read path doing seven times the I/O it needed to.

Bottom line

Monitor index density, not free disk space. By the time a disk alert fires, the indexes have been degrading for months and the query latency has been degrading with them, quietly, in a way no threshold is watching. A weekly avg_leaf_density check over the twenty largest b-trees costs one scan on a replica and tells you months in advance.

Then treat the rebuild as maintenance rather than an incident response. B-tree pages not merging is not a bug and not a missing feature: it is a documented trade that buys you a VACUUM which never locks the table. Workloads that delete most of each key range pay for that trade, and the payment is a periodic REINDEX. Fix the workload if you can, because TRUNCATE instead of DELETE on a cache table removes a whole class of this in one line, and schedule the rebuild if you cannot.

Mass segfaults on a ZFS host: not bad RAM, an unsigned underflow in zfs_fillpage()

BitPage — Sun, 06 Sep 2026 04:15:00 +0000

Short answer: unrelated processes segfaulting all at once on a ZFS host, plus systemd[1]: Freezing execution in the journal, is kernel arithmetic rather than failing memory. In OpenZFS before 2.4.4 and 2.3.9, zfs_fillpage() computed the read length as io_len = i_size - io_off on unsigned types. Truncate a file in place at the moment an mmap page is faulted in, i_size drops below io_off, the difference underflows to near 2⁶⁴, and dmu_read() zero-fills physical memory far past the page it was asked for. The machine survives that. What finishes it off is systemd, whose CrashAction defaults to freeze.

From outside it looked like a powered-off box: no SSH, no HTTP, no hypervisor UI. A hard reset from the provider’s panel brought it back, which left the question nobody enjoys: what that was, and whether it happens again tomorrow.

An hour of digging produced two answers instead of one. The ZFS bug is the trigger. The price of the failure was set by a systemd default.

TL;DR

Cause: unsigned underflow in zfs_fillpage() when an mmap read races an in-place truncate. The kernel zero-fills physical pages belonging to whatever else was there, so the victims are random processes.
Fixed in: OpenZFS 2.4.4 and 2.3.9, both published 21 August 2026. On Proxmox, zfsutils-linux 2.4.4-pve1.
Not fixed in: 2.2.11, published the same minute as the other two. Verified by reading the code in the tag, not the release notes.
Why the outage runs for hours: CrashAction=freeze is the systemd default. A crashed PID 1 hangs forever, nothing restarts services, systemctl blocks, sshd dies later with nobody to bring it back.
One-minute test against bad RAM: ECC counters in /sys/devices/system/edac/mc/mc0/ at zero and no Machine Check in the journal means the DIMMs are not your problem.
Tell in the dump: the length register of memset is not garbage, it is a tidy negative number, exactly minus another register. That is a size computation gone wrong, not failing hardware.
One-line mitigation: CrashAction=reboot in a drop-in turns a repeat from an overnight outage into a 10-second one.

Why the machine looks powered off when it is not

It looks identical to a powered-off machine from outside, which is why the first hypothesis is always wrong. No port answers, the provider’s panel reports power as normal, and support has nothing to add. The obvious theory is that the box went down, so you start looking for the cause of a shutdown: an ACPI event, a watchdog, a power dip in the rack.

The journal kills that theory. journalctl --list-boots shows the previous boot, and its entries run right up to a point a couple of minutes before the new boot began, which is the board’s POST. The last line before the reset was an nginx worker dying:

nginx[…]: [alert] …: worker process … exited on signal 11

The machine never powered off. It ran the whole time and kept writing to the journal that it was in trouble. Nobody outside could hear it.

That yields a second, nastier fact. The moment things actually broke sits much earlier than the “crash time” you get from last. last -x will show crash without saying when it started. The real start is the first segfault in the avalanche, and it can be a whole night before the reset.

Why it is not bad RAM, even though it looks exactly like bad RAM

Simultaneous segfaults across unrelated processes is the classic signature of a failing DIMM, and the reflex it triggers is the wrong one. Before you schedule downtime for memtest86+, spend a minute on two counters:

1
2
3


cat /sys/devices/system/edac/mc/mc0/ce_count # corrected errors
cat /sys/devices/system/edac/mc/mc0/ue_count # uncorrected
journalctl -b -1 -k | grep -i "machine check"

Zeros in all three places on an ECC system mean memory as a subsystem is healthy. A single flipped bit would have been caught and logged. While you are there, confirm ECC is actually active rather than merely printed on the invoice:

1

dmidecode -t memory | grep -E "Error Correction Type|Total Width|Data Width"

Total Width needs to be 8 bits wider than Data Width, 72 against 64, because those are the physical bits doing the correction. Equal widths mean you have no ECC, whatever the spec sheet claims.

That minute saves hours of downtime spent diagnosing the wrong subsystem. ECC protects against flipped bits, not against code that walks the ordinary execution path and writes zeros into the wrong place.

Four dead ends, all through the same function

Every hypothesis that came from searching the symptom was wrong, and they cost more time than the answer did. Writing them out because each one closes a direction: if your stack looks like this, you can skip these branches.

Dead end 1: a Direct IO race. The first sensible search on the call stack landed on a real race in OpenZFS, where a concurrent O_DIRECT write can set db->db_data = NULL and dmu_read_impl() then faults on the dereference. The function names matched mine exactly. Checking the tunable made it look even better:

# cat /sys/module/zfs/parameters/zfs_dio_enabled
1

I said out loud that this was the leading theory. The registers disagreed: what I had was not a NULL dereference but a memset with a sensible destination and an absurd length. Different mechanism. I should have checked that in the first minute, since the registers were on screen the whole time.

Dead end 2: SLUB corruption on encrypted ZFS. Issue #18427 describes my symptom nearly word for word — mass userspace segfaults, reproducible across several releases. One command ended it:

# zfs get -r encryption rpool
NAME PROPERTY VALUE SOURCE
rpool encryption off default

Dead end 3: an already-fixed regression in dmu_read_impl. Issue #17886 is an oops in the same function, also with memory clobbered by a bad length, closed by PR #17915. It read like “known bug, fix exists, go upgrade.” Then I read the PR: it restores lost parentheses in the BRT_RANGESIZE_TO_NBLOCKS() macro, and the author states the blast radius plainly — “this could cause small memory corruptions for vdevs bigger than 64TB+”. A web-workload pool of a few terabytes is an order of magnitude short of that threshold. The stack in that bug also runs through brt_load() during pool import rather than through mmap under load. Same crash site, different bug.

Dead end 4: “the server was off for hours.” Covered above. It died against a continuous journal from the previous boot.

The unpleasant summary: all four trails ran through dmu_read_impl. It is a popular function, “memory corruption” is far too general a symptom, and searching by symptom in that situation returns an endless feed of other people’s misfortunes. The break came from arithmetic, not from searching.

The giveaway: the memset length is exactly minus another register

The whole answer sits in the oops dump, and it only needs counting. Trimmed to what matters:

BUG: unable to handle page fault for address: ffff8983e8400000
#PF: supervisor write access in kernel mode
#PF: error_code(0x0003) - permissions violation
Comm: php-fpm
RIP: 0010:memset+0xb/0x20
RAX: 0000000000020000 RSI: 0000000000000000 RDI: ffff8983e8400000
RDX: ffffffffff7a4000 R15: 000000000085c000
Call Trace:
? dmu_read_impl+0x21e/0x230 [zfs]
dmu_read [zfs]
zfs_fillpage [zfs]
zfs_getpage [zfs]
zpl_read_folio [zfs]
filemap_read_folio
filemap_fault
__do_fault
handle_mm_fault

Read the memset arguments. Under the x86-64 SysV ABI those are RDI for destination, RSI for the fill byte, RDX for the count.

RSI = 0, so it is zero-filling. RDX = 0xffffffffff7a4000 looks like garbage at a glance. Now take R15 = 0x85c000 = 8,765,440. This is the part worth slowing down for: 0xffffffffff7a4000 is exactly −8,765,440 in two’s complement. Not a random value, not a clobbered register. A tidy negative number.

A negative length does not appear from nowhere. It means something subtracted a larger number from a smaller one in unsigned arithmetic. The culprit is not whoever corrupted memory, it is whoever computed the size.

RAX = 0x20000 fits the picture too: 128 KiB, the default ZFS recordsize. An ordinary record read whose length went backwards.

With that hypothesis the search stopped being a search by symptom and became a search by function name plus exact version. The right issue was the first result.

What is broken in ZFS: an invariant that does not survive the release build

Two lines in zfs_fillpage() do the damage, and the guard in front of them does not exist in release builds. Here is the code from the 2.4.3 tag:

1
2
3
4
5
6
7
8


loff_t i_size = i_size_read(ip);
u_offset_t io_off = page_offset(pp);
size_t io_len = PAGE_SIZE;

ASSERT3U(io_off, <, i_size);

if (io_off + io_len > i_size)
 io_len = i_size - io_off;

The invariant is stated in plain sight: the page offset must be smaller than the file size. One line down, a subtraction is built on it. All honest, right up until you open the definition of ASSERT3U for a non-debug build in include/os/linux/spl/sys/debug.h:

1
2
3
4
5
6
7


/*
 * Debugging disabled (--disable-debug)
 */
#ifdef NDEBUG

#define ASSERT3U(x, y, z) \
 ((void) sizeof ((uintptr_t)(x)), (void) sizeof ((uintptr_t)(z)))

Two sizeof expressions that check nothing. The header says so directly: assertions are “compiled out when NDEBUG is defined, this is the default behavior for the SPL.” Distributions, Proxmox included, build ZFS that way. So the invariant is documented, and unenforced exactly where enforcement would have mattered.

Truncating the file in place while a page is being faulted in breaks that invariant. i_size drops below io_off, the subtraction underflows, and io_len becomes a number close to 2⁶⁴. dmu_read() takes it at face value and fills.

The fix adds the runtime check the assertion pretended to be. From the 2.4.4 tag:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


 /*
 * The page may be faulted in after the file has been truncated.
 * There is no data to read; just zero-fill the page.
 */
 if (io_off >= i_size) {
 void *zva = kmap(pp);
 memset(zva, 0, PAGE_SIZE);
 kunmap(pp);
 ClearPageError(pp);
 SetPageUptodate(pp);
 return (0);
 }

The commit is 223b8bc446, “linux: handle mmap read beyond file size”, merged 30 June 2026 by Brian Behlendorf. Its trailers are worth reading: Reported-by: Iliya Polihronov (@vnsavage) (Automattic), Reviewed-by: Alexander Motin. WordPress.com hit this independently, which fits — it is a PHP-shaped bug. readfile() on a cache file that another request is rewriting in place is the exact race.

The crashing process in my dump is php-fpm, running under an unprivileged LXC. Its uid maps to 82 inside the container, which is www-data in Alpine, so it was PHP from an Alpine image in Docker inside LXC, on a Debian host. Four layers down from the kernel, and none of them at fault.

Which versions are actually fixed, and which release from the same day is not

Three OpenZFS releases went out on 21 August 2026, and the fix is in two of them. Checking release notes is not enough here, so I read zfs_vnops_os.c in each tag:

Branch	Release	Published	`zfs_fillpage()` guarded
2.4	zfs-2.4.4	2026-08-21	yes
2.3	zfs-2.3.9	2026-08-21	yes
2.2	zfs-2.2.11	2026-08-21	no

The three tags were published within about a minute of each other. Release notes for 2.4.4 and 2.3.9 both carry the line linux: handle mmap read beyond file size #18715; the notes for 2.2.11 do not. In the 2.2.11 tree, ASSERT3U(io_off, <, i_size); and the bare subtraction are still there. The same holds in the 2.1 branch, so this is an old bug rather than a fresh regression, and downgrading to an older kernel does not help.

Check your own build rather than trusting a version number:

1
2
3
4


zfs version
# then, for the tag you are on:
curl -sf "https://raw.githubusercontent.com/openzfs/zfs/zfs-$(zfs version | head -1 | cut -d- -f2 | cut -d_ -f1)/module/os/linux/zfs/zfs_vnops_os.c" \
 | grep -c 'io_off >= i_size'

Nonzero means the guard is in. Use curl -sf: without -f, a nonexistent tag returns a 404 page that greps to “0 matches”, which is indistinguishable from “the fix is missing.” I nearly recorded that as a result once.

On Proxmox the fixed package is already in the ordinary repository. In pve-no-subscription for trixie, zfsutils-linux is at 2.4.4-pve1, so this is a plain apt upgrade plus a reboot rather than anything exotic.

Upstream the story is not closed. Issue #18787, opened 12 July 2026 by the Incus maintainer, who was losing hosts full of LAMP containers, is still open today even though the fix landed two weeks before it was filed. Track the PR number, not the ticket state. A regression test followed on 18 July.

Why the outage lasts hours: CrashAction defaults to freeze

The bug corrupts memory in milliseconds; the outage lasts all night because of a systemd default. Milliseconds after the corruption started, PID 1 caught its own crash:

systemd[1]: Caught <ABRT>, from our own process.
systemd[1]: Caught <ABRT>, dumped core as pid …
systemd[1]: Freezing execution.

CrashAction= arrived in systemd 256, alongside the systemd.crash_action= kernel command line option. The manual is unambiguous about the default:

Takes one of “freeze”, “reboot” or “poweroff”. Defaults to “freeze”. If set to “freeze”, the system will hang indefinitely when the system manager (PID 1) crashes. If set to “reboot”, the system manager (PID 1) will reboot the machine automatically when it crashes, after a 10s delay.

Freezing makes sense on a desktop, or anywhere an operator is standing next to the console and wants the corpse preserved. On rented hardware in someone else’s data centre it means “hang forever and wait for a human to notice.”

With PID 1 frozen the system loses the ability to repair itself. Nothing restarts crashed services, systemctl blocks, a clean shutdown is impossible. sshd went next, and after its last line there is not one further mention of it in the journal: no login attempts, no refusals. Port 22 simply had nobody listening. nginx meanwhile kept crashing at the same instruction pointer, tens of thousands of times, roughly twice a second, all night. The master was alive and forking workers; each worker inherited poisoned memory and died at once. A restart would have fixed it, and there was no one left to issue one.

The mitigation is one drop-in file:

1
2
3


# /etc/systemd/system.conf.d/10-crash-reboot.conf
[Manager]
CrashAction=reboot

1
2


systemd-analyze cat-config systemd/system.conf # inspect before applying
systemctl daemon-reexec # apply to the running PID 1

A repeat now costs ten seconds instead of a night. That is the whole trade: I cannot remove the bug from a release that does not contain the fix, so I bought recovery time instead of prevention.

Side gotchas worth stealing

Two smaller findings came out of the same morning, and both are worth checking on any Docker host.

A restart: on-failure policy survives a crash but may not survive a clean reboot. The containers came back after the hard reset because their processes were killed by SIGKILL, so the exit code was nonzero and the policy fired. On a graceful reboot dockerd sends SIGTERM, and anything that exits 0 (nginx does) is not a failure, so on-failure leaves it down. The configuration survives the disaster and can fail the maintenance window. Switching everything to unless-stopped with docker update fixes it live, no recreation and no downtime.

A container with no restart: line at all defaults to no. One monitoring container was in that state, so after every reboot it stayed down, and the proxy that depended on it spun through dozens of restarts inside half an hour with host not found in upstream. depends_on does not help here: it orders docker compose up and has no effect when the daemon starts containers on boot.

What to do right now

Check whether your ZFS build carries the guard: zfs version, then look for io_off >= i_size in zfs_fillpage() for that tag. Anything on 2.2.x or 2.1.x is exposed, including 2.2.11 from 21 August 2026.
Upgrade to 2.4.4 or 2.3.9. On Proxmox trixie that is zfsutils-linux 2.4.4-pve1 from pve-no-subscription, then a reboot into the new module.
If you cannot upgrade yet, cut the recovery time: drop in CrashAction=reboot and run systemctl daemon-reexec. Ten seconds against an open-ended hang.
Add an availability check from outside the host. Mine was found by a person arriving at work, which is a bigger hole than the kernel bug.
Audit Docker restart policies: docker inspect -f '{{.Name}} {{.HostConfig.RestartPolicy.Name}}' $(docker ps -aq). Anything showing no or on-failure is a machine that may not come back from a planned reboot.

Bottom line

Three layers of blame, and you can only fix two of them. The ZFS underflow is upstream’s to fix, and as of today it is fixed in 2.4.4 and 2.3.9 while 2.2.11 from the same day still carries it. The systemd default turned a memory-corruption event into an overnight outage, and that costs one line of configuration to change. The missing availability check is what stretched the outage until morning, and that one is nobody’s fault but mine.

The transferable part is the diagnostic move. When the kernel faults inside memset or memcpy, look at the length register before you look at anything else, and check whether it is a tidy negative number. If it is, stop investigating the hardware: someone computed a size wrong, and the fastest path to the answer is the function names in the stack plus your exact version, not the symptom.

Primary sources: commit 223b8bc446 and PR #18715 in OpenZFS, issue #18787, the zfs-2.4.4 release, and systemd(1) for systemd.crash_action=.

PostgreSQL conflict with recovery: why both standard fixes make it worse under Patroni

BitPage — Thu, 06 Aug 2026 07:31:00 +0000

Short answer: Both top search results for canceling statement due to conflict with recovery are wrong on a Patroni cluster with replication slots and one replica. hot_standby_feedback=on parks the xmin horizon inside the slot, where it survives the replica’s death and bloats the primary; raising max_standby_streaming_delay converts held-back replay into the exact lag your load balancer ejects the replica for. What I shipped instead was removing on-marked-down shutdown-sessions from the read backend, after measuring that replica lag runs in single-digit milliseconds at the median against a health-check threshold of 10 MB.

The cluster: one primary and one replica under Patroni 3.0.2 with etcd, PostgreSQL 15, synchronous replication, use_slots: true. Web requests from PHP-FPM read through HAProxy 2.4.30 against the replica; CLI work (queues, scheduler) goes to the primary. Once every quarter hour a cron job wipes and rebuilds a large table in full, and it belongs to someone else’s business logic, so it is not negotiable. The application was catching SQLSTATE[40001], and a developer had already written an interceptor at the PostgresConnection level that retries the cancelled read on the primary. The question I was handed was whether he’d done it right. That took a day to answer and turned out to be about something else entirely.

TL;DR

Fix: drop on-marked-down shutdown-sessions from the read backend — it was killing dozens of live sessions per ejection, and the application does not see 40001 for those, it sees a dead socket.
Don’t: hot_standby_feedback=on with replication slots and a single replica — the xmin outlives the replica, so the failure mode moves from “a report dies” to “the primary fills its disk.”
Don’t: raise max_standby_streaming_delay when the load balancer health-checks replayed position — you pay for it with fan-out read outages instead of a couple of dozen cancelled queries a day.
Measure: replay_lag sampled from the primary once a second for twenty minutes. Median in single-digit milliseconds, p95 in tens, worst case a couple of seconds — against a byte threshold that a ~170 MB burst clears in a few seconds.
Trap: Patroni’s lag= parameter is bytes-only. A requirement stated in seconds cannot be expressed in it at all.

Why hot_standby_feedback is the wrong first move with replication slots

Replication slots change what hot_standby_feedback=on costs you, because the xmin stops being tied to a live connection. The PostgreSQL 15 documentation is upfront about the general risk: the parameter “can be used to eliminate query cancels caused by cleanup records, but can cause database bloat on the primary for some workloads” (runtime-config-replication). That sentence is what every answer on the internet is implicitly waving away when it recommends the setting.

The part that applies specifically here is one section over, in the description of slots: “Replication slots provide an automated way to ensure that the primary does not remove WAL segments until they have been received by all standbys, and that the primary does not remove rows which could cause a recovery conflict even when the standby is disconnected” (warm-standby).

Read that as an operator rather than as a feature description. Without slots, feedback xmin lives as long as the session does; kill the replica and the primary is free again. With slots, the xmin settles into pg_replication_slots.xmin and stays there while the replica is gone. There’s one replica in this cluster and no spare. Turning the setting on would mean any replica outage becomes unbounded bloat on the primary, and I’d have traded a couple of dozen cancelled read queries a day for a way to take down the whole database. That one didn’t need a benchmark to reject.

Why raising max_standby_streaming_delay feeds the flap

Delaying WAL replay is only free if nothing downstream measures replay position, and on this cluster the load balancer does exactly that. max_standby_streaming_delay is the second-most-popular answer, and on paper it is the polite one: instead of killing the query, hold back the apply. The docs describe it as “the maximum total time allowed to apply WAL data once it has been received from the primary server,” defaulting to 30 seconds (runtime-config-replication). The cost stays on the replica, which is what makes it look safe.

It isn’t safe here. HAProxy’s health check calls Patroni’s /replica?lag=, and Patroni computes lag from the replayed position. Held-back replay is, by definition, replay lag. Push the parameter from 30 s to 300 s and every conflict-heavy moment turns into a five-minute-wide window where the balancer can decide the replica is stale and pull it from rotation. The outcome would have been strictly worse than the problem: instead of a couple of dozen cancelled queries per day, fan-out outages across all reads.

The full set of candidates, and why each one died:

Candidate	Where the cost lands	Why it was rejected
`hot_standby_feedback=on`	Primary (bloat)	With slots, xmin survives replica downtime; single replica, no spare
`max_standby_streaming_delay` 30s → 300s	Replica (replay lag)	Replay lag is what the health check ejects on; amplifies flapping
Health-check threshold 10 MB → 512 MB	Nothing measurable	The requirement was stated in seconds; any byte value is a guess at time
HAProxy `agent-check` on wall-clock lag	New moving part	Correct, but a new component in the infrastructure; deferred until measured
Drop `on-marked-down shutdown-sessions`	Nothing	Shipped

The third row is the one I would have gotten wrong on my own. I proposed raising the byte threshold, and the customer rejected it with a better argument than mine: the requirement was written in time (“a second of staleness is fine, half a minute is not”), and a byte threshold answers a different question. On steady write load the two are nearly interchangeable. On batch write load they diverge by orders of magnitude, which is the whole story of this cluster.

Two problems that looked like one

Conflicts and replica ejections clustered in the same minutes because they share a cause, not because one causes the other; barely one conflict in twenty lines up with an ejection. There were two symptom streams. One in the PostgreSQL log — a couple of hundred cancelled queries over a week and a half:

ERROR: canceling statement due to conflict with recovery
DETAIL: User query might have needed to see row versions that must be removed.
FATAL: terminating connection due to conflict with recovery

And one in the HAProxy log (N here is the live session count at the moment of the ejection — dozens, in this cluster):

Server slave/pg_replica is DOWN, reason: Layer7 wrong status, code: 503,
info: "Service Unavailable", check duration: 4ms. 0 active and 0 backup
servers left. N sessions active, 0 requeued, 0 remaining in queue.

Both landed in the same handful of minutes each hour. The connection seemed too obvious to check, which is precisely why it deserved checking.

Before I got there I burned time on two hypotheses that a timestamp comparison would have killed in a minute each:

Checkpoints. With checkpoint_timeout=15min the primary logs a checkpoint every quarter hour. The ejections land on entirely different minutes. No overlap at all.
The backup window. The backup script raises max_standby_streaming_delay to 3600 s for the duration of pg_dump, which looked like an excellent suspect. Every single ejection lasted between 6 and 27 seconds. There is not a single hour-long window in the data.

Then I lined up the two event streams properly (the logs are in different timezones, so this needs converting to one scale first) and compared: about a third of the replica ejections and barely a twentieth of the conflict events fall in shared windows. Two independent symptoms driven by the same cron job, not a chain.

I should have run that correlation first. It cost one command, and it invalidated a morning’s worth of theories I’d been carefully stacking on top of each other.

What the lag actually was, in bytes and in seconds

The replica was never slow at applying WAL; it was slow at receiving it, and the difference decides which knob is even relevant. I sampled pg_stat_replication from the primary once per second:

1
2
3
4


SELECT now()::time(0),
 pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS replay_bytes,
 pg_wal_lsn_diff(pg_current_wal_lsn(), flush_lsn) AS flush_bytes
FROM pg_stat_replication;

Put on a relative time scale, with the figures rounded off, a single burst looks like this:

T+0 s replay= 0 MB flush= 0 MB
T+1 s replay=150 MB flush=150 MB ← cron batch starts
T+2 s replay=170 MB flush=170 MB
T+3 s replay= 90 MB flush= 90 MB
T+5 s replay= 0 MB flush= 0 MB ← caught up

flush tracks replay to within a fraction of a percent. Nothing is queued waiting to be applied, so the lag is purely transport: the primary generated WAL faster than the link carried it. At rest the replica sits tens of kilobytes behind. The health-check threshold is 10 MB. A burst of ~170 MB clears that threshold by a factor of 17 and is gone within a few seconds.

Then the measurement that actually settled the argument, using the *_lag interval columns, sampled once a second over twenty minutes:

Metric	`replay_lag`
Median	single-digit milliseconds
p95	tens of milliseconds
Worst sample seen	a couple of seconds
Samples above 5 s	none
Samples above 30 s	none

The stated requirement was “a second is acceptable, half a minute is not.” The replica never came within three orders of magnitude of unacceptable. And the nature of the lag is confirmed again by subtracting one column from the other at the peaks — the three largest samples in the window, figures rounded:

peak 1 ~160 MB | flush 1.0 s | replay 1.0 s | delta: fractions of a millisecond
peak 2 ~100 MB | flush 0.7 s | replay 0.7 s | delta: fractions of a millisecond
peak 3 ~95 MB | flush 1.7 s | replay 1.7 s | delta: fractions of a millisecond

Applying WAL adds fractions of a millisecond. The two cron runs caught in that window sit exactly a quarter of an hour apart, which is the cron job signing its work.

What lag= in the Patroni health check actually guards

The lag= parameter covers exactly one scenario, “replica alive but behind,” because role and state are checked independently of it. The source is more useful than the documentation here: the docs describe what the endpoint is for, the code describes what it does. From patroni/api.py in v3.0.2:

1
2
3
4
5
6
7


max_replica_lag = parse_int(self.path_query.get('lag', [sys.maxsize])[0], 'B')
if max_replica_lag is None:
 max_replica_lag = sys.maxsize
is_lagging = leader_optime and leader_optime > replayed_location + max_replica_lag

replica_status_code = 200 if not patroni.noloadbalance and not is_lagging and \
 response.get('role') == 'replica' and response.get('state') == 'running' else 503

Two things fall out of those few lines. First, parse_int(..., 'B'): the threshold is bytes, full stop. There’s no time-based variant of this endpoint, so the entire discussion about “what number should we put there” was unanswerable until somebody said the units out loud. Second, is_lagging sits in a chain with role and state, each able to produce a 503 on its own. A genuinely broken replica gets caught by state/role anyway: when streaming breaks, the row just vanishes from pg_stat_replication and there’s no byte figure left to compute.

There’s a third detail that makes the threshold shakier than it looks. leader_optime is read from etcd, refreshed once per loop_wait — 10 seconds by default (Patroni settings) — so the byte figure being compared can be up to a loop_wait stale before you even pick a number to compare it against. Against bursts that come and go in a few seconds, the measurement interval is longer than the event.

Prometheus history sizes the one scenario the threshold does cover (pg_replication_lag_seconds, 15 s scrape). Over a month, lag exceeded one minute exactly once: the single real failure in that window ran for hours, when the replica’s VM came back from a hypervisor reboot without network.

A couple of seconds in the worst normal case against hours for a real failure. Nearly four orders of magnitude between them, and the threshold was sitting right up against the lower bound. A threshold that far from the signal it’s supposed to catch isn’t “tuned strictly,” it’s tuned arbitrarily. The group_vars file even carries a comment from whoever raised it earlier, explaining that the bump was meant to avoid false DOWNs during checkpoint and autovacuum. Against a ~170 MB burst, no value in the tens of megabytes was ever going to work.

Why pg_replication_lag_seconds lies when nobody is writing

On a replica, postgres_exporter computes this metric as now() - pg_last_xact_replay_timestamp(), so during write silence it grows on its own without the replica being behind by anything. Its p99 over the month runs to tens of seconds, and reading that as “the replica served data tens of seconds stale” is wrong: it means the last transaction it replayed was that old, because no newer transaction existed. On the primary the same metric is flat zero. Only direct replay_lag sampling from the primary is trustworthy.

The second limitation is resolution. At a 15-second scrape interval, a burst that lasts a few seconds is invisible by construction. For short events the instrument is the HAProxy log, not the dashboard.

Reading burst length out of the HAProxy log

When monitoring resolution is too coarse, the health check itself is a measuring instrument with known timing. With inter 3s fall 3 rise 2:

DOWN is declared after fall 3 × inter 3s = 9 seconds above threshold.
UP returns after rise 2 × inter 3s = 6 seconds below it.
Therefore burst length ≈ DOWN duration + 3 s.

Applied to thirty-odd ejections over a bit more than a week:

DOWN lasted	Share of ejections	Burst was	Survives `fall 5`?
6 s	about two thirds	~9 s	no — eliminated
9 s	a couple of cases	~12 s	no — eliminated
18 s	a couple of cases	~21 s	yes
21 s	roughly a fifth	~24 s	yes
27 s	one case	~30 s	yes

Two thirds of them lasted the minimum possible DOWN, meaning the burst barely crossed the threshold and was already over. Raising fall 3 → fall 5 requires 15 consecutive seconds above threshold and removes around 70% of the ejections, with no staleness risk given the numbers above. The remaining third ran 21–30 seconds and would still eject.

To be clear about status: that change is argued for with the numbers above and agreed in principle, but it is not applied. It sits behind the same sign-off as anything else touching the balancer, and the residual third of the ejections needs either a higher byte threshold or the time-based check I keep coming back to.

That arithmetic required no new tooling, no sampling, and no agreement from anyone. The data had been in the balancer log the whole time.

What I shipped, and how the rollout went sideways

One line removed from the read backend, leaving the failover-critical backends untouched:

1
2
3
4
5


 backend slave
 option httpchk GET /replica?lag=10485760 ...
 http-check expect status 200
- default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
+ default-server inter 3s fall 3 rise 2

The HAProxy 2.4 manual describes the option plainly: with shutdown-sessions, “all connections to the server are immediately terminated when the server goes down” (configuration manual, §5.2). On a read backend fronting a replica that comes back within seconds, that is damage with no upside. In the logs it killed dozens of sessions at once. The application does not see 40001 for those connections, it sees a severed socket, which the developer’s interceptor never catches. The replica can still be ejected during a burst; live reads now finish, and new connections fall through to use_backend master if { nbsrv(slave) eq 0 }. The option stays in the master and long_queries backends, where killing sessions during failover is the point.

Three things about the rollout worth stealing:

--check --diff on the whole playbook, not just my part. Someone had added weight 80 / weight 20 by hand on the server and never put them in the template. A normal run would have silently erased them. I pulled the repository up to the server’s actual state first, then applied my change on top.
A group_vars file turned out to be a hardlink shared by several inventories. One edit, every environment changed.
I rolled it out with a restart instead of a reload. In master-worker mode kill -USR2 to the master reloads without dropping anything: the master holds the sockets, the port never closes, and the old worker drains its sessions. A restart dropped every connection at once — which is precisely the damage this change exists to prevent. Shipping a fix for connection drops by dropping every connection is the kind of thing you only do once.

Why Patroni ignores the log settings in your playbook

Anything under bootstrap.dcs is applied once, at cluster initialization, and edits to it afterwards do nothing. I found this while looking for something else entirely: pg_log had never rotated. Tens of gigabytes on the primary, single-digit gigabytes on the replica, files that had been piling up for years, on a cluster whose Ansible template configures rotation perfectly well.

Nobody had edited anything by hand this time. The Patroni documentation states the rule with unusual force: “Once Patroni has initialized the cluster for the first time and settings have been stored in the DCS, all future changes to the bootstrap.dcs section of the YAML configuration will not take any effect!” (YAML configuration). The section “will be written into /<namespace>/<scope>/config of the given configuration store after initializing the new cluster,” and from that moment the copy in etcd is the one that counts.

Changing those values later goes through patronictl edit-config or the REST API, not the file (dynamic configuration). Which means a Patroni cluster has a whole class of settings where the playbook is documentation rather than configuration, and --check --diff will never tell you, because the file on disk matches the template exactly. It is right and the cluster still ignores it.

What the application layer got wrong

The interceptor I was asked to review was correct; the defects were around it. Findings handed back to the developers:

ERROR and FATAL are different events. ERROR: canceling statement due to conflict with recovery cancels the statement and leaves the connection usable. FATAL: terminating connection due to conflict with recovery does not. Over the week and a half of logs, nearly all the events were ERROR and a handful were FATAL. One retry strategy cannot serve both.
The retry loop was unreachable past its first iteration, so it retried once regardless of configuration.
The application under-reports by half. Over two weeks the database logged roughly twice as many conflict events as the application reported through its single Log::warning. That gap is why nobody knew the size of the problem.
application_name is empty in every event, so a conflict cannot be attributed to a service from the database logs — only to a database user.
A pre-existing lazy-connection bug in the Laravel wrapper. bindValues() opens with $this->getPdo()->getAttribute(...), and getPdo() resolves the lazy closure and opens a connection to the primary. Every query with bindings calls it, read-only ones included, so the read/write split silently leaks connections to the primary on any parameterised read.

Half of my initial review comments on the interceptor did not survive my own re-checking. Separating “I would have written this differently” from “this is a defect” is most of what code review is, and I am not always good at it.

Step by step

Count the events before theorizing: split ERROR from FATAL in the PostgreSQL log and get the rate per day.
Correlate every symptom stream by timestamp on one timezone scale, first, before building any causal story.
Sample pg_stat_replication from the primary once per second; compare replay_lsn against flush_lsn to separate transport lag from apply lag.
Sample the *_lag interval columns to get lag in seconds, which is the unit your requirement is almost certainly written in.
Read your health-check endpoint’s source for the units it actually accepts before negotiating a threshold value.
Derive burst duration from the balancer’s own inter/fall/rise timing when scrape intervals are too coarse.
Check whether hot_standby_feedback is safe for your topology: with slots and one replica, it is not.
Run --check --diff across the whole playbook before applying, and reload rather than restart.
For anything Patroni owns, read the live values with patronictl show-config rather than trusting the YAML in your repository.

Bottom line

Neither of the two canonical answers to conflict with recovery was usable on this cluster, and both failures were topology-specific rather than wrong in general. hot_standby_feedback assumes you have no slots or a spare replica; raising the streaming delay assumes nothing downstream measures replay position. Check those assumptions against your own cluster before pasting either one into postgresql.conf.

The larger lesson is about units. The health check measured “how much WAL has not arrived” in bytes while the business requirement said “how stale is the data I am serving” in seconds. Every discussion about the right threshold was unanswerable until somebody wrote both sentences down next to each other. The measured answer was single-digit milliseconds at the median against a real failure measured in hours — and the deferred work, an agent-check agent reporting up/down from now() - pg_last_xact_replay_timestamp(), is now a decision that can be made with numbers instead of a guess.

Gmail bounced everything the relay sent: three broken auth mechanisms and a silent OpenDKIM

BitPage — Thu, 06 Aug 2026 07:08:22 +0000

Short answer: Three independent mechanisms were broken at once, and fixing any one of them changed nothing visible. The domain had no v=spf1 record (only a dead SenderID one), the PTR resolved forward to a different server, and OpenDKIM was signing nothing because KeyTable and SigningTable were attached as refile: while their contents were in plain two-column format. That last one cost the most time: when OpenDKIM can’t find a key it logs absolutely nothing, so the daemon reads active, the config parses, the key is readable, and mail still leaves unsigned.

The server is an outgoing relay for transactional mail from a couple of dozen application hosts: password resets, notifications, alerts. Inbound mail for the domain goes elsewhere, to Google Workspace as MX. Postfix on a legacy container with more than two years of uptime, OpenDKIM 2.10.3, no Ansible, edits by hand only. The daily log summary read a four-digit bounced against a single-digit sent: thousands of rejections, a handful of deliveries. Password reset mail was among the casualties, so people could not get back into their accounts. Nobody had changed anything on this box in years, which turned out to be the whole story.

TL;DR

Root cause (DKIM): KeyTable/SigningTable mounted as refile: while the entries were plain example.org mail._domainkey.example.org lines. Under refile: the key is a regex matched against the whole From: address, so example.org never matches noreply@example.org. Changing one word per line to file: fixed it.
Also broken: InternalHosts listed fewer than ten addresses against a Postfix mynetworks two to three times longer. Mail from the app hosts was being verified instead of signed.
Also broken: the only SPF-shaped TXT record was spf2.0/mfrom,pra SenderID. RFC 7208 discards anything that doesn’t begin with v=spf1, so the domain effectively had no SPF record.
Also broken: the PTR pointed at a hostname whose A record led to a different machine. Forward-confirmed reverse DNS failed.
Trap: grep -c DKIM-Signature /var/log/mail.log returns 0 whether signing works or not. Postfix does not write message headers to the log.
Trap: testing with sendmail from the mail server itself passes, because 127.0.0.1 was in InternalHosts. Real traffic from other hosts kept going out bare.

Why a server nobody touched stopped being deliverable

The requirements moved while the configuration stood still. Gmail began requiring authentication from bulk senders in February 2024 and has been ramping enforcement since, and a config that predated those rules simply stopped qualifying (Email sender guidelines). There was no change window to roll back to, no deploy to blame. The rejection came back verbatim as:

550-5.7.26 Your email has been blocked because the sender is unauthenticated.
550-5.7.26 Gmail requires all senders to authenticate with either SPF or DKIM.
550-5.7.26 Authentication results:
550-5.7.26 DKIM = did not pass
550-5.7.26 SPF [example.org] with ip: [203.0.113.51] = did not pass

Both named mechanisms failing at once should have been the tell that this was not one bug. I read it as one bug anyway and spent the next hour paying for it.

Three of the four facts in my own notes were wrong

I started from a ticket written a day earlier that already contained a diagnosis, and rechecking it by hand killed three of its four claims. The notes said: PTR is fine and forward and reverse match; DNS is hosted on Cloudflare; DMARC is absent.

The PTR pointed at legacy.example.org, whose A record led to 198.51.100.216, a different server entirely. dig NS returned ns-*.awsdns-*, so DNS was on Route 53, not Cloudflare. DMARC was present and had been all along, v=DMARC1; p=none. One claim out of four survived. These were my own notes, from the previous day.

The other rejection message had been telling me about the PTR the whole time:

550-5.7.25 [203.0.113.51] The IP address sending this message does not have a PTR
550-5.7.25 record setup, or the corresponding forward DNS entry does not match the sending IP

The operative half is the second clause. Not “you have no PTR record” but “the forward entry doesn’t match”. I had read to the first familiar phrase, seen a PTR existed, and moved on. RFC 1912 §2.1 states the rule plainly: “Make sure your PTR and A records match. For every IP address, there should be a matching PTR record in the in-addr.arpa domain” (RFC 1912). The same section warns that getting this wrong “can cause loss of Internet services similar to not being registered in the DNS at all”, which is a fair description of a bounce column four digits wide.

The fix had two possible shapes. Editing the A record of legacy.example.org to point here would have satisfied the check, but that hostname is live and fronts an unrelated service, so the blast radius covered somebody else’s system. Repointing the PTR to mail.example.org, whose A record already resolved to the sending IP, changed nothing outside this server. I also set Postfix myhostname to mail.example.org so the HELO name matches the PTR.

Why the SPF record that existed didn’t count

There was a TXT record on the domain, and it was worth exactly nothing, because it was a SenderID record rather than an SPF one. It began spf2.0/mfrom,pra. RFC 7208 §4.5 is unambiguous about what a verifier does with that: “Starting with the set of records that were returned by the lookup, discard records that do not begin with a version section of exactly ‘v=spf1’” (RFC 7208).

Discarded means the domain had no SPF record at all as far as Gmail was concerned, which is precisely what SPF [example.org] ... = did not pass was reporting. A record that looks like SPF to a human and is invisible to every verifier is worse than an empty zone, because it stops anyone from looking further. Published in its place:

v=spf1 ip4:203.0.113.51 include:_spf.google.com include:spf.example.net ~all

Why grepping the mail log for DKIM-Signature proves nothing

Postfix does not write message headers to its log, so the count is zero whether signing works or not. My first check was grep -c "DKIM-Signature" /var/log/mail.log, which returned 0, and I took that as evidence that nothing was being signed. It wasn’t evidence of anything. The method cannot distinguish a broken signer from a working one, and I used it as the basis for the next hour of work.

The check that does work is to send a message to a local mailbox and read the headers of the delivered file, /var/mail/nobody. That shows the actual DKIM-Signature header or its absence, with no inference in between. Building that first would have made every subsequent change a one-minute yes or no. I built it fourth.

OpenDKIM reports active, parses clean, and signs nothing

OpenDKIM writes no log line when it fails to find a signing key for a message, so systemctl is-active returning active alongside a clean startup is zero evidence that signing happens. I found and fixed two genuine defects on the strength of that assumption, and neither produced a signature.

The first was the config itself. /etc/opendkim.conf had been truncated to four lines, with no Mode, no KeyTable, no SigningTable. The fingerprints of a package upgrade were right there: /etc/opendkim/ had been touched recently while the files inside it were years older. I restored the missing directives and restarted. Clean start, no errors, no signature.

The second was key ownership. The private key belonged to _apt:uuidd with mode 0600, and the daemon runs as opendkim. Checking as the service user rather than as root is the only way to see this:

su -s /bin/sh opendkim -c "head -c 20 /etc/opendkim/default/mail.private"
head: cannot open '/etc/opendkim/default/mail.private' for reading: Permission denied

A real defect, and chown opendkim:opendkim is mandatory. The key became readable. Still no signature. Two genuine bugs repaired, both necessary, neither sufficient, and my supply of hypotheses was gone.

The actual root cause was one word, and it was in the man page

The tables were attached as refile: while their contents were in plain two-column format, and under refile: the lookup key is treated as a regex matched against the entire From: address. I stopped guessing and opened man opendkim.conf on the server itself, same 2.10.3 as the running binary, which is more authoritative than whatever version the web is serving. The answer sits in one paragraph about dataset types:

“If this table specifies a regular expression file (“refile”), then the keys are wildcard patterns that are matched against the address found in the From: header field. For all other database types, the full user@host is checked first, then simply host, then user@.domain (with all superdomains checked in sequence…)”

The tables held lines like this:

example.org mail._domainkey.example.org

Under refile:, the pattern example.org is matched against noreply@example.org and does not match, so no key is found and the message leaves unsigned without a single log line. Under file:, the lookup falls through to the second step, “then simply host”, where example.org matches exactly. The fix was one word in two lines:

1
2
3
4


-KeyTable refile:/etc/opendkim/KeyTable
-SigningTable refile:/etc/opendkim/SigningTable
+KeyTable file:/etc/opendkim/KeyTable
+SigningTable file:/etc/opendkim/SigningTable

The other direction works too: rewrite every table entry to *@example.org and keep refile:. I rejected it because the tables covered several domains, and rewriting every line of data is a wider edit than changing the dataset type to the one that already matches the format on disk.

The second half of the same bug: InternalHosts was a fraction of mynetworks

OpenDKIM only signs mail arriving from hosts in InternalHosts; everything else it verifies instead, so any app server missing from that list was sending bare mail by design. The man page states the whole behaviour in one line, describing the option as identifying “a set internal hosts whose mail should be signed rather than verified” (typo included, as printed in 2.10.3).

InternalHosts pointed at TrustedHosts, which listed fewer than ten addresses. Postfix mynetworks listed two to three times as many. Syncing the short list against the long one tripled it, finally covering the application hosts that had been relaying through a server which was, on paper, configured to sign their mail. The two files are not required to match exactly, since they answer different questions, but a host that can relay and cannot be signed is almost always an oversight rather than a decision.

This is also why my manual test had lied to me earlier. Mail sent with sendmail from the relay itself originates from 127.0.0.1, which was in the list, so a signature appeared and I briefly thought the job was done. The traffic that actually mattered came from the application hosts, none of them in the file. Test along the path production traffic takes, not the path that is convenient from an SSH session.

The four defects and the check that actually proves each one

Broken	Symptom you see	Check that settles it
No `v=spf1` record, SenderID only	`SPF … = did not pass`	`dig +short TXT example.org` — the string must begin `v=spf1`
PTR forward mismatch	`550-5.7.25 … forward DNS entry does not match`	`dig -x <ip>`, then resolve that name’s A record back — it must return `<ip>`
`refile:` on plain-format tables	Nothing. No log line at all.	Read the `DKIM-Signature` header of a delivered message
`InternalHosts` narrower than `mynetworks`	Signs locally, unsigned from apps	Send from a real application host, not from the relay

The third row is the one worth internalising. Three of these four failures announce themselves somewhere. The dataset-type bug produces no error, no warning, and a healthy-looking service, and the only way to observe it is to inspect the output rather than the process.

Step by step

Build the verification path before touching anything: send to a local mailbox and read the delivered headers. Every later change then gives an immediate yes or no.
Recheck every fact you were handed, including notes you wrote yourself yesterday.
Read the whole SMTP rejection. 5.7.25 and 5.7.26 name different failures, and in both the specificity lives in the second clause of the sentence.
Resolve the sending IP with dig -x, then resolve the answer forward. Both directions, or it doesn’t count.
Confirm the TXT record begins with v=spf1 rather than merely mentioning SPF.
For DKIM: confirm the dataset type matches the table format, confirm the service user can read the key with su -s /bin/sh opendkim -c "…", and confirm InternalHosts covers every host in mynetworks.
Send the final test from a host that is not the mail server.

The end state was DKIM-Signature: v=1; a=rsa-sha256; d=example.org; s=mail in the delivered headers, spf=pass, dmarc=pass, and 250 2.0.0 OK from Gmail. Total time was about an hour and a half, most of it spent on the silent one.

What authentication didn’t fix: six messages in ten still bounce

Passing SPF and DKIM gets mail accepted, not trusted, and this domain’s reputation problem survived every fix above untouched. A full day of counters after the fix still showed more rejections than deliveries — roughly six messages in ten going nowhere — and among them close to a hundred a day addressed to @iclaud.com, a typo of @icloud.com that the application has been faithfully retrying for years. Bounce rate is not the metric Google publishes a threshold for; that one is spam rate, which bulk senders are asked to keep under 0.30% in Postmaster Tools. They are different numbers, but they feed the same reputation ledger, and 0.30% is a useful sense of the scale a receiver considers acceptable. Mail with flawless authentication still lands in the spam folder if the domain spends years hammering addresses that don’t exist.

The reason nobody knew is that the feedback loop had been cut at both ends. The server delivered non-delivery reports to the sender address, whose domain it considered local via mydestination, and no local user by that name existed, so the reports were discarded on arrival. That was hundreds of destroyed reports a day. The application mails a dead address, the report about it is deleted by the same server, nobody finds out, and the application keeps mailing.

What I deliberately did not do was tighten DMARC past p=none. Moving to quarantine before the statistics are clean would route legitimate mail to spam by my own hand, and there is no rush. Still outstanding: the envelope sender equals the From: address, where a dedicated bounce address with VERP belongs; certbot 0.12.0 is dead on this box, still calling the retired acme-v01 endpoint, with a certificate that expired years ago. And every change here was made by hand on a container that lives outside Ansible, so none of it exists in any repository. The next person to open this server will find no trace of why it works.

Bottom line

When a daemon reports active and does not do its job while logging nothing, stop repairing hypotheses and go read the man page for the semantics of the specific option you are relying on. Silence is not the absence of a fault. It is its own class of behaviour, and it is always described somewhere in the documentation, usually in a sentence about how lookups are performed.

The process mistake was ordering. I fixed three genuine defects (truncated config, key ownership, then the host list) before finding the one that actually mattered, and each time I expected the next test to pass. A reliable check built at minute one, reading the headers of a delivered message, would have turned each of those into a one-minute negative result instead of an hour of accumulating false hope. Build the instrument before you start turning things.

How to migrate a stateful Docker app off a server with a dying disk

BitPage — Fri, 24 Jul 2026 11:35:00 +0000

Short answer: When the source server’s disk fails on writes but still reads fine, treat it as read-only hardware. Stream pg_dump -Fc over SSH straight to the target machine, pull Docker images from your registry instead of rebuilding end-of-life Dockerfiles, and anchor every rsync exclude with a leading slash. That moved an app of some fifteen services, with a PostgreSQL database on the order of ten gigabytes, onto a new node without a single write to the failing disk.

The source was an old OpenVZ box with a degraded RAID and a disk that had quietly stopped accepting writes. Reads were healthy. On it lived a Dockerized PHP application: some fifteen services (nginx, php-fpm, a pool of queue workers, PostgreSQL, Redis), a database on the order of ten gigabytes, a few gigabytes of media. The job was to stand up a production-like copy on a new Proxmox node for testing, with the real traffic cutover scheduled later.

TL;DR

Fix: run pg_dump -Fc on the source but pipe it through ssh -T to the target’s disk — the source only reads, the target does all the writing.
Fix: don’t rebuild images based on EOL distributions; pull the exact production images from the registry, reusing /root/.docker/config.json from the source host.
Trap: rsync --exclude=log/ without a leading slash matches log at every depth. It silently removed vendor/yiisoft/yii2/log/ from the copy and the app answered HTTP 500.
Trap: a dump to a dying disk hangs without an error. Watch the output file’s size delta, not the process list.

Why pg_dump hangs on a dying disk instead of failing

A dump writing to a dead disk doesn’t error out; it stops making progress while the process stays alive. My first attempt was the obvious one: pg_dump -Fc into a local backup/ directory on the source. The file grew to roughly a gigabyte and froze. Two size checks six seconds apart returned the same size, with the process still running. The disk had died on the write path specifically, and a hung dump was exactly how that looked from userspace.

The liveness check lied to me too. The watcher used pgrep -f pg_dump, and its own command line contained the string pg_dump, so it matched itself and kept reporting the dump as alive. ps -C pg_dump (exact process name match) or a size delta on the output file are the checks that don’t fool themselves.

Streaming the dump: keep every write off the source

The fix is to leave the source with the read half of the work and move the write half to another machine. pg_dump runs on the source, but its output goes over SSH to the target’s healthy disk:

1
2
3


ssh -T old-host \
 'docker exec app-postgres pg_dump -Fc -U app appdb' \
 > /srv/dumps/appdb.dump

Dumping from a live database is fine here. The PostgreSQL 11 documentation states it directly: “pg_dump does not block other users accessing the database (readers or writers)” (pg_dump reference).

Two details matter for the transport. ssh -T disables pseudo-terminal allocation, and docker exec must run without -t: a tty mangles the binary custom-format stream. With both in place the gigabyte-and-a-half dump completed with exit code 0, and on the target pg_restore --no-owner -j 4 -d appdb unpacked it back into the same ten gigabytes or so, roughly a hundred tables, with the rum extension intact.

Pull EOL images, don’t rebuild them

If production images already exist in a registry, pull them; rebuilding their Dockerfiles years later is a dead end. My first docker compose pull on the new host failed with Error response from daemon: error from registry: access forbidden — a private GitLab registry the new machine wasn’t authorized against. Instead of fixing auth, I detoured into rebuilding the images from the same Dockerfiles production once used. That went like this:

Base image	Debian release inside	What apt says in 2026
`nginx:1.17.4`	buster	`Release 404 Not Found` on deb.debian.org
`postgres:11`	stretch	PGDG packages for PostgreSQL 11 removed, 404

Buster I could route to archive.debian.org. Then the stretch-based postgres:11 build died on the PGDG repository, which no longer serves packages for an EOL major at all. I stopped there and admitted the detour was me solving the wrong problem.

The actual answer was sitting on the dying host the whole time: it pulls those images itself, which means working registry credentials live in its /root/.docker/config.json (that’s where docker login stores them). Copying that one file to the new host let docker compose pull fetch the whole set of images — the exact bytes production runs, no build step.

The rsync exclude that deleted framework files

--exclude=log/ without a leading slash matches every directory named log at any depth, not just the top-level one you meant. I added it to skip a couple of gigabytes of application logs during the code-and-media rsync. The copy finished clean, and the app came up with HTTP 500:

include(/app/vendor/yiisoft/yii2/log/Logger.php):
failed to open stream: No such file or directory

Nothing in that message points at rsync. A missing framework file reads like a corrupted vendor directory, and I chased that theory first. The real cause: the unanchored pattern had stripped vendor/yiisoft/yii2/log/ out of the transfer. The rsync man page is explicit about the rule: “if the pattern starts with a / then it is anchored to a particular spot in the hierarchy of files, otherwise it is matched against the end of the pathname.”

The fix is one character: --exclude=/log/ anchors the pattern to the transfer root. A second rsync run then sent only the log directories it had wrongly skipped inside the code, a few seconds of incremental transfer.

Testing against live integrations without side effects

Queue workers and cron are what produce outbound effects; a web tier on top of a copied database is safe to start. This app talks to a real payment provider, CRM, SMTP, and ClickHouse, and the test copy had to come up without firing any of that. The compose setup was already split into docker-compose.yml (web) and docker-compose.queue.yml (workers and scheduler), so the whole safety switch was one environment variable:

1

COMPOSE_FILE=docker/docker-compose.yml docker compose up -d

Web and API up, every queue worker and the scheduler down, database an isolated copy. Smoke test: API returned 200, the site returned its usual 302, the admin backend rendered the login page.

One integration still broke: ClickHouse refused connections from the new node’s IP (“Connection refused”), and backend pages that touch it answered 503. External allowlists (database, mail, analytics) are part of the migration checklist, not something to discover during cutover.

Step by step

Establish the disk’s failure mode first: “reads work, writes hang” changes every decision after it.
Copy /root/.docker/config.json from the source to the target; docker compose pull the production images.
Stream the database: ssh -T source 'docker exec db pg_dump -Fc -U app appdb' > dump on the target.
Restore with pg_restore --no-owner -j 4 -d appdb.
Rsync code and media with anchored excludes only (--exclude=/log/, not --exclude=log/).
Bring up the web tier alone via COMPOSE_FILE, leaving queues and cron down.
Smoke-test, and collect every external service that needs the new IP allowlisted.
Cut over later in a quiet window: fresh dump, DNS, allowlists.

Bottom line

Dying hardware changes the rules: the source is read-only whether you accept it or not, so structure the whole migration around that. Stream the dump instead of writing it locally, pull images instead of rebuilding EOL bases, and anchor rsync excludes before they cut into vendor/. The copy went from “disk is dying” to a responding production clone in a few hours, and most of the lost time was me fighting battles (EOL apt archives, a phantom vendor corruption) that the constraints had already decided.

BitPage

PHP 5.6 and PHP2Shell: is it vulnerable, what fixes it, and how to prove you fixed it

TL;DR

Is 5.6 actually vulnerable

Where to get a fix

The gap, and the fact that it closed the next day

Trap: with the bailout patch alone there is no marker at all

Trap: the 2019 Dockerfile no longer builds

What changes for the application

Step by step

Bottom line

PHP2Shell on PHP 7: patch applied is not patch compiled is not patch working

TL;DR

Where to get a fixed PHP 7.x

Patch the last patch release of the branch

Trap 1: the dry run is clean and the build fails

Trap 2: the 7.1 package needed a second revision the next day

Trap 3: the build assert that asserts nothing

Trap 4: the patch is in and the version and ini say otherwise

Docker: your own base image, from the last official Dockerfile

CentOS 6: your own RPM, and nothing to roll back to

Traps in the old build environment

What changes for the application

Step by step

Bottom line

PHP2Shell on PHP 8: docker build --pull succeeded and PHP is still 8.2.29

TL;DR

Why the same tag with a fresh pull is not a fix

Supported branches: what to actually pin

EOL 8.0: build your own base image

Gotchas

The patch is in and the version says otherwise

That same test proves nothing on 8.2

The image is updated and the running PHP is not

docker-php-ext-install rebuilds from unpatched sources

A 2021 Dockerfile no longer builds, and without a build there is no patch

What changes for the application

Step by step

Bottom line

PHP2Shell on a mixed fleet: auditing 5.6 to 8.4 and proving each build is actually patched

TL;DR

What PHP2Shell actually is

Why a mass-scan script gives you the wrong answer

The per-machine checklist, read-only

Which branches have a fix at all

The ordering rule that matters

How to prove a build is patched

The marker table, per branch

The build assert that asserts nothing

Mitigations that do not work, and one that makes it worse

What the patch changes in application behaviour

Looking for traces

Step by step

Bottom line

Room 3 froze on a suspending @Transaction: the SQLite driver you pass picks your connection pool

TL;DR

Why does an Android app hang on the splash screen with no crash and no ANR?

How do you get a stack trace out of a process that has not crashed?

Why does the SQLite driver decide whether Room can deadlock?

Where exactly does a suspending @Transaction wait on itself?

Is this already fixed in a newer Room 3?

What does switching to BundledSQLiteDriver actually cost?

Why did the driver swap silently change PRAGMA synchronous?

How do you catch a pragma that only applies to some connections?

How do you force a pragma on every connection Room opens?

What I would check first next time

Versions this was reproduced and fixed on

How to never pay a generative API twice: a durable task table, not a retrying job queue

TL;DR

Why a job queue is the wrong unit of recovery for a paid call

What RFC 9110 actually says about retrying a POST

Do the generative APIs give you an idempotency key?

The boundary is before the request, not after the response

Three recovery branches after an expired lease

Why OUTCOME_UNKNOWN has to be terminal

Claiming work with FOR UPDATE SKIP LOCKED

Heartbeat, because not every step polls

Pinning the order with a test

The price in the docs is not the price on the invoice

Bottom line

`docker-php-ext-install` rebuilds from unpatched sources