some Allan Jude text from OpenZFS Mastery

from "OpenZFS Mastery"

**Short answer:** unrelated processes segfaulting all at once on a ZFS host, plus `systemd[1]: Freezing execution` in the journal, is kernel arithmetic rather than failing memory. In OpenZFS before 2.4.4 and 2.3.9, `zfs_fillpage()` computed the read length as `io_len = i_size - io_off` on unsigned types. Truncate a file in place at the moment an mmap page is faulted in, `i_size` drops below `io_off`, the difference underflows to near 2⁶⁴, and `dmu_read()` zero-fills physical memory far past the page it was asked for. The machine survives that. What finishes it off is systemd, whose `CrashAction` defaults to `freeze`. From outside it looked like a powered-off box: no SSH, no HTTP, no hypervisor UI. A hard reset from the provider's panel brought it back, which left the question nobody enjoys: what that was, and whether it happens again tomorrow. An hour of digging produced two answers instead of one. The ZFS bug is the trigger. The price of the failure was set by a systemd default. ## TL;DR * **Cause:** unsigned underflow in `zfs_fillpage()` when an mmap read races an in-place truncate. The kernel zero-fills physical pages belonging to whatever else was there, so the victims are random processes. * **Fixed in:** OpenZFS 2.4.4 and 2.3.9, both published 21 August 2026. On Proxmox, `zfsutils-linux` 2.4.4-pve1. * **Not fixed in:** 2.2.11, published the same minute as the other two. Verified by reading the code in the tag, not the release notes. * **Why the outage runs for hours:** `CrashAction=freeze` is the systemd default. A crashed PID 1 hangs forever, nothing restarts services, `systemctl` blocks, sshd dies later with nobody to bring it back. * **One-minute test against bad RAM:** ECC counters in `/sys/devices/system/edac/mc/mc0/` at zero and no `Machine Check` in the journal means the DIMMs are not your problem. * **Tell in the dump:** the length register of `memset` is not garbage, it is a tidy negative number, exactly minus another register. That is a size computation gone wrong, not failing hardware. * **One-line mitigation:** `CrashAction=reboot` in a drop-in turns a repeat from an overnight outage into a 10-second one. ## Why the machine looks powered off when it is not It looks identical to a powered-off machine from outside, which is why the first hypothesis is always wrong. No port answers, the provider's panel reports power as normal, and support has nothing to add. The obvious theory is that the box went down, so you start looking for the cause of a shutdown: an ACPI event, a watchdog, a power dip in the rack. The journal kills that theory. `journalctl --list-boots` shows the previous boot, and its entries run right up to a point a couple of minutes before the new boot began, which is the board's POST. The last line before the reset was an nginx worker dying: nginx[…]: [alert] …: worker process … exited on signal 11 The machine never powered off. It ran the whole time and kept writing to the journal that it was in trouble. Nobody outside could hear it. That yields a second, nastier fact. The moment things actually broke sits much earlier than the "crash time" you get from `last`. `last -x` will show `crash` without saying when it started. The real start is the first segfault in the avalanche, and it can be a whole night before the reset. ## Why it is not bad RAM, even though it looks exactly like bad RAM Simultaneous segfaults across unrelated processes is the classic signature of a failing DIMM, and the reflex it triggers is the wrong one. Before you schedule downtime for memtest86+, spend a minute on two counters: cat /sys/devices/system/edac/mc/mc0/ce_count # corrected errors cat /sys/devices/system/edac/mc/mc0/ue_count # uncorrected journalctl -b -1 -k | grep -i "machine check" Zeros in all three places on an ECC system mean memory as a subsystem is healthy. A single flipped bit would have been caught and logged. While you are there, confirm ECC is actually active rather than merely printed on the invoice: dmidecode -t memory | grep -E "Error Correction Type|Total Width|Data Width" `Total Width` needs to be 8 bits wider than `Data Width`, 72 against 64, because those are the physical bits doing the correction. Equal widths mean you have no ECC, whatever the spec sheet claims. That minute saves hours of downtime spent diagnosing the wrong subsystem. ECC protects against flipped bits, not against code that walks the ordinary execution path and writes zeros into the wrong place. ## Four dead ends, all through the same function Every hypothesis that came from searching the symptom was wrong, and they cost more time than the answer did. Writing them out because each one closes a direction: if your stack looks like this, you can skip these branches. **Dead end 1: a Direct IO race.** The first sensible search on the call stack landed on a real race in OpenZFS, where a concurrent `O_DIRECT` write can set `db->db_data = NULL` and `dmu_read_impl()` then faults on the dereference. The function names matched mine exactly. Checking the tunable made it look even better: # cat /sys/module/zfs/parameters/zfs_dio_enabled 1 I said out loud that this was the leading theory. The registers disagreed: what I had was not a NULL dereference but a `memset` with a sensible destination and an absurd length. Different mechanism. I should have checked that in the first minute, since the registers were on screen the whole time. **Dead end 2: SLUB corruption on encrypted ZFS.** Issue #18427 describes my symptom nearly word for word — mass userspace segfaults, reproducible across several releases. One command ended it: # zfs get -r encryption rpool NAME PROPERTY VALUE SOURCE rpool encryption off default **Dead end 3: an already-fixed regression in`dmu_read_impl`.** Issue #17886 is an oops in the same function, also with memory clobbered by a bad length, closed by PR #17915. It read like "known bug, fix exists, go upgrade." Then I read the PR: it restores lost parentheses in the `BRT_RANGESIZE_TO_NBLOCKS()` macro, and the author states the blast radius plainly — "this could cause small memory corruptions for vdevs bigger than 64TB+". A web-workload pool of a few terabytes is an order of magnitude short of that threshold. The stack in that bug also runs through `brt_load()` during pool import rather than through mmap under load. Same crash site, different bug. **Dead end 4: "the server was off for hours."** Covered above. It died against a continuous journal from the previous boot. The unpleasant summary: all four trails ran through `dmu_read_impl`. It is a popular function, "memory corruption" is far too general a symptom, and searching by symptom in that situation returns an endless feed of other people's misfortunes. The break came from arithmetic, not from searching. ## The giveaway: the memset length is exactly minus another register The whole answer sits in the oops dump, and it only needs counting. Trimmed to what matters: BUG: unable to handle page fault for address: ffff8983e8400000 #PF: supervisor write access in kernel mode #PF: error_code(0x0003) - permissions violation Comm: php-fpm RIP: 0010:memset+0xb/0x20 RAX: 0000000000020000 RSI: 0000000000000000 RDI: ffff8983e8400000 RDX: ffffffffff7a4000 R15: 000000000085c000 Call Trace: ? dmu_read_impl+0x21e/0x230 [zfs] dmu_read [zfs] zfs_fillpage [zfs] zfs_getpage [zfs] zpl_read_folio [zfs] filemap_read_folio filemap_fault __do_fault handle_mm_fault Read the `memset` arguments. Under the x86-64 SysV ABI those are `RDI` for destination, `RSI` for the fill byte, `RDX` for the count. `RSI = 0`, so it is zero-filling. `RDX = 0xffffffffff7a4000` looks like garbage at a glance. Now take `R15 = 0x85c000 = 8,765,440`. This is the part worth slowing down for: `0xffffffffff7a4000` is exactly −8,765,440 in two's complement. Not a random value, not a clobbered register. A tidy negative number. A negative length does not appear from nowhere. It means something subtracted a larger number from a smaller one in unsigned arithmetic. The culprit is not whoever corrupted memory, it is whoever computed the size. `RAX = 0x20000` fits the picture too: 128 KiB, the default ZFS `recordsize`. An ordinary record read whose length went backwards. With that hypothesis the search stopped being a search by symptom and became a search by function name plus exact version. The right issue was the first result. ## What is broken in ZFS: an invariant that does not survive the release build Two lines in `zfs_fillpage()` do the damage, and the guard in front of them does not exist in release builds. Here is the code from the 2.4.3 tag: loff_t i_size = i_size_read(ip); u_offset_t io_off = page_offset(pp); size_t io_len = PAGE_SIZE; ASSERT3U(io_off, <, i_size); if (io_off + io_len > i_size) io_len = i_size - io_off; The invariant is stated in plain sight: the page offset must be smaller than the file size. One line down, a subtraction is built on it. All honest, right up until you open the definition of `ASSERT3U` for a non-debug build in include/os/linux/spl/sys/debug.h: /* * Debugging disabled (--disable-debug) */ #ifdef NDEBUG #define ASSERT3U(x, y, z) \ ((void) sizeof ((uintptr_t)(x)), (void) sizeof ((uintptr_t)(z))) Two `sizeof` expressions that check nothing. The header says so directly: assertions are "compiled out when NDEBUG is defined, this is the default behavior for the SPL." Distributions, Proxmox included, build ZFS that way. So the invariant is documented, and unenforced exactly where enforcement would have mattered. Truncating the file in place while a page is being faulted in breaks that invariant. `i_size` drops below `io_off`, the subtraction underflows, and `io_len` becomes a number close to 2⁶⁴. `dmu_read()` takes it at face value and fills. The fix adds the runtime check the assertion pretended to be. From the 2.4.4 tag: /* * The page may be faulted in after the file has been truncated. * There is no data to read; just zero-fill the page. */ if (io_off >= i_size) { void *zva = kmap(pp); memset(zva, 0, PAGE_SIZE); kunmap(pp); ClearPageError(pp); SetPageUptodate(pp); return (0); } The commit is 223b8bc446, "linux: handle mmap read beyond file size", merged 30 June 2026 by Brian Behlendorf. Its trailers are worth reading: `Reported-by: Iliya Polihronov (@vnsavage) (Automattic)`, `Reviewed-by: Alexander Motin`. WordPress.com hit this independently, which fits — it is a PHP-shaped bug. `readfile()` on a cache file that another request is rewriting in place is the exact race. The crashing process in my dump is `php-fpm`, running under an unprivileged LXC. Its uid maps to 82 inside the container, which is `www-data` in Alpine, so it was PHP from an Alpine image in Docker inside LXC, on a Debian host. Four layers down from the kernel, and none of them at fault. ## Which versions are actually fixed, and which release from the same day is not Three OpenZFS releases went out on 21 August 2026, and the fix is in two of them. Checking release notes is not enough here, so I read `zfs_vnops_os.c` in each tag: Branch | Release | Published | `zfs_fillpage()` guarded ---|---|---|--- 2.4 | zfs-2.4.4 | 2026-08-21 | yes 2.3 | zfs-2.3.9 | 2026-08-21 | yes 2.2 | zfs-2.2.11 | 2026-08-21 | **no** The three tags were published within about a minute of each other. Release notes for 2.4.4 and 2.3.9 both carry the line `linux: handle mmap read beyond file size #18715`; the notes for 2.2.11 do not. In the 2.2.11 tree, `ASSERT3U(io_off, <, i_size);` and the bare subtraction are still there. The same holds in the 2.1 branch, so this is an old bug rather than a fresh regression, and downgrading to an older kernel does not help. Check your own build rather than trusting a version number: zfs version # then, for the tag you are on: curl -sf "https://raw.githubusercontent.com/openzfs/zfs/zfs-$(zfs version | head -1 | cut -d- -f2 | cut -d_ -f1)/module/os/linux/zfs/zfs_vnops_os.c" \ | grep -c 'io_off >= i_size' Nonzero means the guard is in. Use `curl -sf`: without `-f`, a nonexistent tag returns a 404 page that greps to "0 matches", which is indistinguishable from "the fix is missing." I nearly recorded that as a result once. On Proxmox the fixed package is already in the ordinary repository. In `pve-no-subscription` for trixie, `zfsutils-linux` is at 2.4.4-pve1, so this is a plain `apt` upgrade plus a reboot rather than anything exotic. Upstream the story is not closed. Issue #18787, opened 12 July 2026 by the Incus maintainer, who was losing hosts full of LAMP containers, is still open today even though the fix landed two weeks before it was filed. Track the PR number, not the ticket state. A regression test followed on 18 July. ## Why the outage lasts hours: CrashAction defaults to freeze The bug corrupts memory in milliseconds; the outage lasts all night because of a systemd default. Milliseconds after the corruption started, PID 1 caught its own crash: systemd[1]: Caught <ABRT>, from our own process. systemd[1]: Caught <ABRT>, dumped core as pid … systemd[1]: Freezing execution. `CrashAction=` arrived in systemd 256, alongside the `systemd.crash_action=` kernel command line option. The manual is unambiguous about the default: > Takes one of "freeze", "reboot" or "poweroff". Defaults to "freeze". If set to "freeze", the system will hang indefinitely when the system manager (PID 1) crashes. If set to "reboot", the system manager (PID 1) will reboot the machine automatically when it crashes, after a 10s delay. Freezing makes sense on a desktop, or anywhere an operator is standing next to the console and wants the corpse preserved. On rented hardware in someone else's data centre it means "hang forever and wait for a human to notice." With PID 1 frozen the system loses the ability to repair itself. Nothing restarts crashed services, `systemctl` blocks, a clean shutdown is impossible. sshd went next, and after its last line there is not one further mention of it in the journal: no login attempts, no refusals. Port 22 simply had nobody listening. nginx meanwhile kept crashing at the same instruction pointer, tens of thousands of times, roughly twice a second, all night. The master was alive and forking workers; each worker inherited poisoned memory and died at once. A restart would have fixed it, and there was no one left to issue one. The mitigation is one drop-in file: # /etc/systemd/system.conf.d/10-crash-reboot.conf [Manager] CrashAction=reboot systemd-analyze cat-config systemd/system.conf # inspect before applying systemctl daemon-reexec # apply to the running PID 1 A repeat now costs ten seconds instead of a night. That is the whole trade: I cannot remove the bug from a release that does not contain the fix, so I bought recovery time instead of prevention. ## Side gotchas worth stealing Two smaller findings came out of the same morning, and both are worth checking on any Docker host. **A`restart: on-failure` policy survives a crash but may not survive a clean reboot.** The containers came back after the hard reset because their processes were killed by SIGKILL, so the exit code was nonzero and the policy fired. On a graceful reboot dockerd sends SIGTERM, and anything that exits 0 (nginx does) is not a failure, so `on-failure` leaves it down. The configuration survives the disaster and can fail the maintenance window. Switching everything to `unless-stopped` with `docker update` fixes it live, no recreation and no downtime. **A container with no`restart:` line at all defaults to `no`.** One monitoring container was in that state, so after every reboot it stayed down, and the proxy that depended on it spun through dozens of restarts inside half an hour with `host not found in upstream`. `depends_on` does not help here: it orders `docker compose up` and has no effect when the daemon starts containers on boot. ## What to do right now 1. Check whether your ZFS build carries the guard: `zfs version`, then look for `io_off >= i_size` in `zfs_fillpage()` for that tag. Anything on 2.2.x or 2.1.x is exposed, including 2.2.11 from 21 August 2026. 2. Upgrade to 2.4.4 or 2.3.9. On Proxmox trixie that is `zfsutils-linux` 2.4.4-pve1 from `pve-no-subscription`, then a reboot into the new module. 3. If you cannot upgrade yet, cut the recovery time: drop in `CrashAction=reboot` and run `systemctl daemon-reexec`. Ten seconds against an open-ended hang. 4. Add an availability check from outside the host. Mine was found by a person arriving at work, which is a bigger hole than the kernel bug. 5. Audit Docker restart policies: `docker inspect -f '{{.Name}} {{.HostConfig.RestartPolicy.Name}}' $(docker ps -aq)`. Anything showing `no` or `on-failure` is a machine that may not come back from a planned reboot. ## Bottom line Three layers of blame, and you can only fix two of them. The ZFS underflow is upstream's to fix, and as of today it is fixed in 2.4.4 and 2.3.9 while 2.2.11 from the same day still carries it. The systemd default turned a memory-corruption event into an overnight outage, and that costs one line of configuration to change. The missing availability check is what stretched the outage until morning, and that one is nobody's fault but mine. The transferable part is the diagnostic move. When the kernel faults inside `memset` or `memcpy`, look at the length register before you look at anything else, and check whether it is a tidy negative number. If it is, stop investigating the hardware: someone computed a size wrong, and the fastest path to the answer is the function names in the stack plus your exact version, not the symptom. Primary sources: commit 223b8bc446 and PR #18715 in OpenZFS, issue #18787, the zfs-2.4.4 release, and systemd(1) for `systemd.crash_action=`. _Originally published at bitpage.me — BitPage, a technical blog on backend, databases, infrastructure and incident post-mortems._

from OpenZFS Mastery

Motivation and Context Add instructions for AI code agents Description Add the popular CLAUDE.md instructions for AI agents from: https://github.com/multica-ai/andrej-karpathy-skills This will help...

The new TrueNAS Proxmox plugin automates VM storage, ZFS snapshots, iSCSI, and NVMe/TCP while removing manual LUN management.

FreeCORE carries TrueNAS CORE 13.3 forward as an independently maintained operating system on FreeBSD, with in-place upgrades to 15.0.

This independent fork carries FreeBSD, OpenZFS, and jails forward for users who never wanted Linux.

This independent fork carries FreeBSD, OpenZFS, and jails forward for users who never wanted Linux.

This independent fork carries FreeBSD, OpenZFS, and jails forward for users who never wanted Linux.