citable URL: https://halobench.com/records/clm-0053/ — this address never moves; the anchor /records/#clm-0053 keeps resolving
A benchmarking-harness defect, not a hardware limitation, produced this lab's earlier "Nemotron-3-Super cannot allocate on ROCm at any depth" verdict (clm-0050's first amendment, corrected 2026-08-14). House standard on this hardware is `--load-mode none` (no mmap): it is in the llama-server serving flags and in every tau2/capability queue script, but four llama-bench throughput queue scripts never carried it, so every throughput number taken through them inherited llama-bench's own default of mmap. The gap sat between the serving configuration and the benchmark configuration — same lab, same box, two different flag sets.
Consequence measured: Nemotron-3-Super (~82 GB of weights on a 122 GB box) failed on stock ROCm 3653e6d at every depth tested under mmap with an HSA `BadAlloc` (run-0234, run-0235), and was recorded as a categorical backend limitation. Under `--load-mode none`, on the identical binary, it runs: 244.05 pp / 16.43 tg at d32768, median of 3 reps (run-0236, run-0237; an earlier single-rep check the same day landed within 2% at 248.79 pp / 16.39 tg, run-0243/run-0244 — see clm-0050's second amendment for the full comparison against Vulkan). The capability verdict was an artifact of the harness disagreeing with the serving configuration, not a finding about the hardware.
Cost quantified: mmap is not a throughput confound for a model that fits. A matched `--load-mode none` vs mmap pair on Qwen3.6-35B-A3B, same box, same boot session, 3 reps each (comfortably inside memory) measured +0.72% prefill / −0.12% decode (run-0238, run-0239 against run-0240, run-0241) — noise. An independent, two-days-earlier measurement of the same mmap arm (run-0116, run-0117) landed within 0.25% of this session's mmap median, so the result is not session-specific. Published relative numbers taken under mmap survive; mmap is a correctness lever only for a model near the memory ceiling, not a throughput one for a model that isn't.
Mechanism: under mmap the page cache holds a duplicate of the model (43 GiB observed resident) that competes with the KFD system-memory gate for the same host RAM. The kernel logs the refusal (`amdgpu: SVM mapping failed, exceeds resident system memory limit`) from roughly 2 minutes into the run; the userspace `BadAlloc` operators actually watch for does not surface for 55 to 72 minutes. Raising GTT from 84 to 122 GiB did not rescue the failing cell (run-0234, run-0235) — GTT and the KFD system-memory gate are separate accountants, so it was the wrong knob. `amdgpu.no_system_mem_limit=1` is inert once mmap is off (the postboot no-mmap reps behind run-0236/run-0237 ran with it set and were unaffected), and with mmap on it converts the fast, visible `BadAlloc` into an indefinite `svm_range_restore_work` hang with no watchdog-visible signal — measured directly in run-0242: a 900-second wall-clock bound was what ended that attempt, not the process, and the kernel log shows no `BadAlloc` at all, only the restore workqueue hogging the CPU with increasing frequency for the full window. Worse than the failure it was meant to fix.
The lesson: a capability verdict is only as good as the harness's agreement with the serving configuration it claims to describe. The way to catch this defect class is to diff benchmark flags against production serving flags before trusting a categorical result — a check this lab had not run until this diagnosis.
METHOD — the defect was found by diffing the flag sets of every queue script against the standing serving config. `docs/benchmark-runbook.md` and `bench/protocol.json` both name `--load-mode none` as mandatory on this unified-memory hardware, and it is present in every llama-server serving invocation and every tau2/capability queue script (e.g. `bench/queues/queue-e-nemotron-quant.sh`, cfg-0027/cfg-0028's serving flags). It was absent from the four llama-bench throughput queue scripts behind the perf-matrix and perf-matrix-gtt120 suites, which instead ran llama-bench's own default (mmap on). All four were rebuilt to carry `--load-mode none` after this finding.
THE OOM, AS FIRST RECORDED — cfg-0057 (ROCm, f16 KV) and cfg-0058 (ROCm, q8_0 KV) both failed at d32768: run-0234 hit a `BadAlloc` 93 times over a 55-minute window before the watchdog terminated it, run-0235 hit it 5 times over a longer window before terminating. journalctl -k at the same timestamps shows the real cause — "amdgpu: SVM mapping failed, exceeds resident system memory limit" — recorded continuously from within minutes of process start in both cases, 44,862 lines total across the two cells. Raising the boot-time GTT ceiling from the fleet default (~84 GiB) to 122 GiB (`amdgpu.gttsize=122880`) and re-running did not rescue either cell — the failure tracks host-RAM residency, not the GTT window.
THE FIX — re-running cfg-0057's arm with `--load-mode none` added and nothing else changed (cfg-0059, same binary, same box) produced a clean single-rep pass (248.788416 pp / 16.390674 tg at d32768, run-0243/run-0244) confirmed by a post-reboot 3-rep median (244.051835 pp / 16.434903 tg, run-0236/run-0237, stddev 3.76 pp / 0.03 tg) within 2% of the single check. run-0236/run-0237 is the fully-backed N=3 record and the one quoted as this cell's result. Against Vulkan's own run-0190/run-0191 at the same cell (139.810384 pp / 17.559139 tg, unaffected by the mmap question because it never hit the resident-memory ceiling ROCm did) that is ROCm +74.6% prefill / Vulkan +6.4% decode — see clm-0050's second amendment for the full reading of what that means for the per-model/per-phase rule.
THE CONTROL — before generalising "mmap doesn't matter for throughput," it needed a check on a model that fits, because Nemotron-3-Super's cell only demonstrates that mmap matters for CORRECTNESS near the ceiling, not that it is throughput- neutral away from it. The check was built as a true matched pair: cfg-0032 (ROCm, Qwen3.6-35B-A3B, f16 KV, d32768, mmap default — the same arm behind clm-0050's Qwen3.6-35B numbers) was re-measured in the SAME boot session as its `--load-mode none` companion, immediately before it (run-0240/run-0241: 597.271098 pp / 42.41958 tg, median of 3), rather than reusing the two-days-earlier run-0116/run-0117 (595.763132 pp / 42.326925 tg) as the baseline — same-session removes timing/thermal/session drift as a possible explanation for the delta. cfg-0060 is the identical arm with `--load-mode none` added: run-0238/run-0239, 601.555743 pp / 42.368806 tg, median of 3 — +0.72% pp / −0.12% tg over the same-session mmap median (+0.72%/-0.12% again if measured against the two-days-earlier run-0116/run-0117 instead: the two mmap baselines agree within 0.25% of each other). Both readings are inside the ~3% scatter line the runbook already treats as noise for this suite. Every relative number this lab has published under mmap on a model that fits survives this finding unmodified — the corpus's models sit comfortably under the box's 122 GiB pool with the single measured exception of Nemotron-3-Super's ~82 GB weight footprint. What does not survive is any categorical "cannot run"/"cannot allocate" verdict reached under mmap on a model near that ceiling — those need a `--load-mode none` re-check before they can be trusted, and Nemotron-3-Super's was the one this lab had made.
THE HANG VARIANT — a second postboot session added `amdgpu.no_system_mem_limit=1` to the kernel cmdline (cfg-0061) and re-ran both arms of the Nemotron cell. `--load-mode none` was unaffected (run-0236/run-0237, above) — that flag is inert once mmap is off. The mmap arm was not: run-0242 was killed by the queue's own 900-second bound (rc=124) with no `BadAlloc` ever surfacing; journalctl -k for the same window shows no "SVM mapping failed" either, only `workqueue: svm_range_restore_work [amdgpu] hogged CPU for >10000us` recurring with increasing frequency (4 times at +2m20s, 67 times by +15m, when the bound cut it off). Setting that flag while mmap stays on trades a failure an operator can at least detect for one a fixed-duration bound is the only thing that ends.
KERNEL VS USERSPACE DETECTION LAG — the kernel-side "SVM mapping failed" message is legible from roughly 2 to 5 minutes into a doomed run; the userspace `BadAlloc` an operator would actually be watching a process exit code or stderr for does not land for 55 to 72 minutes (run-0234's window: 55 min; run-0235's: 72 min). `dmesg`'s ring buffer rotates out under the message flood well before that; only `journalctl -k` (the persistent journal) still has the onset by the time the userspace symptom finally shows. See `docs/methodology-lessons.md` for this as a general detection-lag rule, not just the one incident.
CROSS-REFERENCE — this is the same failure class documented in `docs/aihydra-first-boot.md`'s "mmap double-residency" note and `docs/storage-and-lanes.md`'s `LLAMA_ARG_LOAD_MODE=none` rule and `bench/protocol.json`'s `--load-mode none on unified memory` entry, which records that the underlying defect "was hit FOUR separate times before being fixed properly" in serving code. This claim is the same defect class recurring a fifth time, in benchmark tooling rather than serving code, which is why it survived those four fixes untouched — they patched call sites that served requests, and none of them were a `llama-bench` invocation.