HALOBENCH — log https://halobench.com/log/ The lab notebook, generated from the record: claims landed, gate changes, run series, incidents and contributions — retractions badged inline. en-gb Sat, 15 Aug 2026 00:00:00 GMT [claim] On gfx1151 at f16 KV, Qwen3.8-27B (dense) is a fourth model measured under the per-model, per-phase backend rule (clm-0050), and it lands emphatically on the prefill-favours-ROCm side: DECODE is backend-independent on this model (every matched cell within 5%), but Vulkan PREFILL collapses with depth — 0.55x ROCm at d32768 on Q8_0 (87.82 vs 159.98 t/s) and 0.48x on UD-Q4_K_XL (99.80 vs 207.61) — and at d131072 stock Vulkan cannot complete the cell at all: 2 of 2 reps in BOTH quants aborted with vk::DeviceLostError, the kernel logging an amdgpu ring timeout and recovering the device by ring reset, while ROCm completed every cell it was offered (83.44 pp / 6.11 tg Q8_0, 95.59 pp / 8.14 tg UD-Q4_K_XL at d131072). https://halobench.com/log/#e-2026-08-15-1 https://halobench.com/log/#e-2026-08-15-1 Sat, 15 Aug 2026 00:00:00 GMT On gfx1151 at f16 KV, Qwen3.8-27B (dense) is a fourth model measured under the per-model, per-phase backend rule (clm-0050), and it lands emphatically on the prefill-favours-ROCm side: DECODE is backend-independent on this model (every matched cell within 5%), but Vulkan PREFILL collapses with depth — 0.55x ROCm at d32768 on Q8_0 (87.82 vs 159.98 t/s) and 0.48x on UD-Q4_K_XL (99.80 vs 207.61) — and at d131072 stock Vulkan cannot complete the cell at all: 2 of 2 reps in BOTH quants aborted with vk::DeviceLostError, the kernel logging an amdgpu ring timeout and recovering the device by ring reset, while ROCm completed every cell it was offered (83.44 pp / 6.11 tg Q8_0, 95.59 pp / 8.14 tg UD-Q4_K_XL at d131072). — records: clm-0054 claim [claim] draft-mtp speculation on Qwen3.8-27B (stock 3653e6d, gfx1151) peaks at spec-draft-n-max=3 over CLEAN cells — Vulkan Q8_0 17.75 t/s (2.26x its 7.86 no-speculation floor, acceptance 0.626), Vulkan UD-Q4_K_XL 26.68 t/s (2.23x, 0.623), ROCm Q8_0 18.25 t/s (2.33x, 0.6235) — and at n_max >= 4 the feature is BROKEN on this model: after accumulated generation volume in a live session (sequential VARIED prompts at full length; not fresh servers, not one repeated prompt, not short generations), generations start terminating at 1 token with <|im_end|> (id 248046). https://halobench.com/log/#e-2026-08-15-2 https://halobench.com/log/#e-2026-08-15-2 Sat, 15 Aug 2026 00:00:00 GMT draft-mtp speculation on Qwen3.8-27B (stock 3653e6d, gfx1151) peaks at spec-draft-n-max=3 over CLEAN cells — Vulkan Q8_0 17.75 t/s (2.26x its 7.86 no-speculation floor, acceptance 0.626), Vulkan UD-Q4_K_XL 26.68 t/s (2.23x, 0.623), ROCm Q8_0 18.25 t/s (2.33x, 0.6235) — and at n_max >= 4 the feature is BROKEN on this model: after accumulated generation volume in a live session (sequential VARIED prompts at full length; not fresh servers, not one repeated prompt, not short generations), generations start terminating at 1 token with <|im_end|> (id 248046). — records: clm-0055 claim [claim] Qwen3.8-27B's KV cache on llama.cpp costs exactly 64.00 KiB per token — the hybrid-attention allocation working as designed, measured byte-exact on this box: only 16 of the 64 layers carry a conventional KV cache (the 3:1 linear-to-full-attention layout), and 16 layers x 4 KV heads x 256 head_dim x 2 (K+V) x 2 bytes (f16) = 65,536 bytes/token. https://halobench.com/log/#e-2026-08-15-3 https://halobench.com/log/#e-2026-08-15-3 Sat, 15 Aug 2026 00:00:00 GMT Qwen3.8-27B's KV cache on llama.cpp costs exactly 64.00 KiB per token — the hybrid-attention allocation working as designed, measured byte-exact on this box: only 16 of the 64 layers carry a conventional KV cache (the 3:1 linear-to-full-attention layout), and 16 layers x 4 KV heads x 256 head_dim x 2 (K+V) x 2 bytes (f16) = 65,536 bytes/token. — records: clm-0056 claim [claim] Qwen3.8-27B met its registered agentic prediction: on the standing 5-task tau2 airline smoke subset it scored 1.000 (5/5, 21 tool-call messages, 0 empty assistant turns, VALID under the smoke gate) against the >= 0.80 bar recorded in the candidate record on 2026-08-14, BEFORE any measurement — and against the incumbent qwen36-27b-mtp's 0.80 on the identical tasks under the identical pinned-simulator protocol, where the incumbent failed task 2 and this model passed it. https://halobench.com/log/#e-2026-08-15-4 https://halobench.com/log/#e-2026-08-15-4 Sat, 15 Aug 2026 00:00:00 GMT Qwen3.8-27B met its registered agentic prediction: on the standing 5-task tau2 airline smoke subset it scored 1.000 (5/5, 21 tool-call messages, 0 empty assistant turns, VALID under the smoke gate) against the >= 0.80 bar recorded in the candidate record on 2026-08-14, BEFORE any measurement — and against the incumbent qwen36-27b-mtp's 0.80 on the identical tasks under the identical pinned-simulator protocol, where the incumbent failed task 2 and this model passed it. — records: clm-0057 claim [gate] qwen38-27b: listed → screened — Screen tier passed on release day, on-box. https://halobench.com/log/#e-2026-08-15-5 https://halobench.com/log/#e-2026-08-15-5 Sat, 15 Aug 2026 00:00:00 GMT qwen38-27b: listed → screened — Screen tier passed on release day, on-box. — records: clm-0055, clm-0056, clm-0057, run-0267 gate [gate] qwen38-27b: screened → benched — DAY-ONE SCREEN COMPLETE, all four phases, overnight on release day. https://halobench.com/log/#e-2026-08-15-6 https://halobench.com/log/#e-2026-08-15-6 Sat, 15 Aug 2026 00:00:00 GMT qwen38-27b: screened → benched — DAY-ONE SCREEN COMPLETE, all four phases, overnight on release day. — records: clm-0054, clm-0055, clm-0056, clm-0057, run-0271, run-0267, run-0268 gate [runs] 1 run landed on cfg-0066 (eos-cliffguard) https://halobench.com/log/#e-2026-08-15-7 https://halobench.com/log/#e-2026-08-15-7 Sat, 15 Aug 2026 00:00:00 GMT 1 run landed on cfg-0066 (eos-cliffguard) — records: run-0267 runs [runs] 1 run landed on cfg-0067 (mtp-probe) https://halobench.com/log/#e-2026-08-15-8 https://halobench.com/log/#e-2026-08-15-8 Sat, 15 Aug 2026 00:00:00 GMT 1 run landed on cfg-0067 (mtp-probe) — records: run-0268 runs [runs] 1 run landed on cfg-0066 (tau2-bench-airline) https://halobench.com/log/#e-2026-08-15-9 https://halobench.com/log/#e-2026-08-15-9 Sat, 15 Aug 2026 00:00:00 GMT 1 run landed on cfg-0066 (tau2-bench-airline) — records: run-0271 runs [claim] AMENDED 2026-08-14 — the rule below is PER-MODEL AND PER-PHASE, not fleet-wide; see the correction history and the amendment note. https://halobench.com/log/#e-2026-08-14-1 https://halobench.com/log/#e-2026-08-14-1 Fri, 14 Aug 2026 00:00:00 GMT AMENDED 2026-08-14 — the rule below is PER-MODEL AND PER-PHASE, not fleet-wide; see the correction history and the amendment note. — records: clm-0050 claim [claim] On gfx1151, stock ROCm 3653e6d's own BF16 KV decode is SLOWER than its own F16 KV decode: 29.26 t/s vs 42.35 t/s at 32,768 depth (Qwen3.6-35B-A3B, fa on) — a 31% penalty for switching to the KV type the community recommends for its quality, with prefill roughly unchanged (587.4 vs 595.0). stew675's rdna-boosts branch (commit ed89854) fixes it with a native BF16 flash-attention tile kernel: 47.31 t/s BF16 decode, +61.7% over stock's own BF16 and stock's best config on this chip. https://halobench.com/log/#e-2026-08-14-2 https://halobench.com/log/#e-2026-08-14-2 Fri, 14 Aug 2026 00:00:00 GMT On gfx1151, stock ROCm 3653e6d's own BF16 KV decode is SLOWER than its own F16 KV decode: 29.26 t/s vs 42.35 t/s at 32,768 depth (Qwen3.6-35B-A3B, fa on) — a 31% penalty for switching to the KV type the community recommends for its quality, with prefill roughly unchanged (587.4 vs 595.0). stew675's rdna-boosts branch (commit ed89854) fixes it with a native BF16 flash-attention tile kernel: 47.31 t/s BF16 decode, +61.7% over stock's own BF16 and stock's best config on this chip. — records: clm-0052 claim [claim] A benchmarking-harness defect, not a hardware limitation, produced this lab's earlier "Nemotron-3-Super cannot allocate on ROCm at any depth" verdict (clm-0050's first amendment, corrected 2026-08-14). https://halobench.com/log/#e-2026-08-14-3 https://halobench.com/log/#e-2026-08-14-3 Fri, 14 Aug 2026 00:00:00 GMT A benchmarking-harness defect, not a hardware limitation, produced this lab's earlier "Nemotron-3-Super cannot allocate on ROCm at any depth" verdict (clm-0050's first amendment, corrected 2026-08-14). — records: clm-0053 claim [gate] 4 candidates → screened: npu-embeddinggemma, npu-lfm2, npu-qwen3-4b-thinking, npu-whisper https://halobench.com/log/#e-2026-08-14-4 https://halobench.com/log/#e-2026-08-14-4 Fri, 14 Aug 2026 00:00:00 GMT 4 candidates → screened: npu-embeddinggemma, npu-lfm2, npu-qwen3-4b-thinking, npu-whisper gate [gate] qwen38-27b: listed → listed — RELEASED and CONFIRMED runnable. https://halobench.com/log/#e-2026-08-14-5 https://halobench.com/log/#e-2026-08-14-5 Fri, 14 Aug 2026 00:00:00 GMT qwen38-27b: listed → listed — RELEASED and CONFIRMED runnable. gate [runs] 8 runs landed on cfg-0050, cfg-0051 (llama-bench) https://halobench.com/log/#e-2026-08-14-6 https://halobench.com/log/#e-2026-08-14-6 Fri, 14 Aug 2026 00:00:00 GMT 8 runs landed on cfg-0050, cfg-0051 (llama-bench) — records: cfg-0050, cfg-0051 runs [runs] 44 runs landed on cfg-0052, cfg-0053, cfg-0056, cfg-0059, cfg-0060, cfg-0032, cfg-0062, cfg-0063, cfg-0064, cfg-0065 (llama-bench) https://halobench.com/log/#e-2026-08-14-7 https://halobench.com/log/#e-2026-08-14-7 Fri, 14 Aug 2026 00:00:00 GMT 44 runs landed on cfg-0052, cfg-0053, cfg-0056, cfg-0059, cfg-0060, cfg-0032, cfg-0062, cfg-0063, cfg-0064, cfg-0065 (llama-bench) — records: cfg-0052, cfg-0053, cfg-0056, cfg-0059, cfg-0060, cfg-0032, cfg-0062, cfg-0063, cfg-0064, cfg-0065 runs [runs] 8 runs landed on cfg-0054, cfg-0055 (llama-bench) https://halobench.com/log/#e-2026-08-14-8 https://halobench.com/log/#e-2026-08-14-8 Fri, 14 Aug 2026 00:00:00 GMT 8 runs landed on cfg-0054, cfg-0055 (llama-bench) — records: cfg-0054, cfg-0055 runs [runs] 2 runs landed on cfg-0057, cfg-0058 (perf-matrix-gtt120) https://halobench.com/log/#e-2026-08-14-9 https://halobench.com/log/#e-2026-08-14-9 Fri, 14 Aug 2026 00:00:00 GMT 2 runs landed on cfg-0057, cfg-0058 (perf-matrix-gtt120) — records: cfg-0057, cfg-0058 runs [runs] 1 run landed on cfg-0061 (memgate-test) https://halobench.com/log/#e-2026-08-14-10 https://halobench.com/log/#e-2026-08-14-10 Fri, 14 Aug 2026 00:00:00 GMT 1 run landed on cfg-0061 (memgate-test) — records: run-0242 runs [runs] 2 runs landed on cfg-0068, cfg-0069 (mtp-probe) https://halobench.com/log/#e-2026-08-14-11 https://halobench.com/log/#e-2026-08-14-11 Fri, 14 Aug 2026 00:00:00 GMT 2 runs landed on cfg-0068, cfg-0069 (mtp-probe) — records: cfg-0068, cfg-0069 runs [claim] Quantised KV on gfx1151 splits three ways by build. https://halobench.com/log/#e-2026-08-13-1 https://halobench.com/log/#e-2026-08-13-1 Thu, 13 Aug 2026 00:00:00 GMT Quantised KV on gfx1151 splits three ways by build. — records: clm-0051 claim [gate] 7 candidates → screened: gemma4-12b, glm-45-air, ling-30-flash, llama4-scout, maple-preview, nemotron35-lightning-30b, qwen36-27b-mtp https://halobench.com/log/#e-2026-08-13-2 https://halobench.com/log/#e-2026-08-13-2 Thu, 13 Aug 2026 00:00:00 GMT 7 candidates → screened: gemma4-12b, glm-45-air, ling-30-flash, llama4-scout, maple-preview, nemotron35-lightning-30b, qwen36-27b-mtp gate [gate] nemotron35-lightning-30b: listed → acquired — Official ggml-org Q4_K_M downloaded to aihydra (25,430,738,944 bytes, exactly the listed size) and sha256-verified against the HF LFS oid (6110e2e2e6cd324e6ee69ddced5a6b34fad6c94ca9827222a1e420fb92e3c90b). https://halobench.com/log/#e-2026-08-13-3 https://halobench.com/log/#e-2026-08-13-3 Thu, 13 Aug 2026 00:00:00 GMT nemotron35-lightning-30b: listed → acquired — Official ggml-org Q4_K_M downloaded to aihydra (25,430,738,944 bytes, exactly the listed size) and sha256-verified against the HF LFS oid (6110e2e2e6cd324e6ee69ddced5a6b34fad6c94ca9827222a1e420fb92e3c90b). gate [runs] 44 runs landed on cfg-0038, cfg-0039, cfg-0040, cfg-0041, cfg-0042, cfg-0043, cfg-0044, cfg-0045, cfg-0046, cfg-0047, cfg-0048, cfg-0049 (llama-bench) https://halobench.com/log/#e-2026-08-13-4 https://halobench.com/log/#e-2026-08-13-4 Thu, 13 Aug 2026 00:00:00 GMT 44 runs landed on cfg-0038, cfg-0039, cfg-0040, cfg-0041, cfg-0042, cfg-0043, cfg-0044, cfg-0045, cfg-0046, cfg-0047, cfg-0048, cfg-0049 (llama-bench) — records: cfg-0038, cfg-0039, cfg-0040, cfg-0041, cfg-0042, cfg-0043, cfg-0044, cfg-0045, cfg-0046, cfg-0047, cfg-0048, cfg-0049 runs [gate] deepseek-v4-flash: listed → listed — Identity now concrete via the r/LocalLLaMA Strix Halo guide thread: DeepSeek V4 Flash 0731, deepseek4 arch, 256 experts/6 active + 1 shared, MIT licence, 1M native context. https://halobench.com/log/#e-2026-08-12-1 https://halobench.com/log/#e-2026-08-12-1 Wed, 12 Aug 2026 00:00:00 GMT deepseek-v4-flash: listed → listed — Identity now concrete via the r/LocalLLaMA Strix Halo guide thread: DeepSeek V4 Flash 0731, deepseek4 arch, 256 experts/6 active + 1 shared, MIT licence, 1M native context. — records: clm-0031 gate [gate] nemotron35-lightning-30b: listed — Surfaced via an operator-shared r/AIDeveloperNews link ("NVIDIA has launched Nemotron 3.5 Lightning") plus a follow-up r/StrixHalo post on a community ROCmFP4 requant with hardware-matched Strix Halo numbers. https://halobench.com/log/#e-2026-08-12-2 https://halobench.com/log/#e-2026-08-12-2 Wed, 12 Aug 2026 00:00:00 GMT nemotron35-lightning-30b: listed — Surfaced via an operator-shared r/AIDeveloperNews link ("NVIDIA has launched Nemotron 3.5 Lightning") plus a follow-up r/StrixHalo post on a community ROCmFP4 requant with hardware-matched Strix Halo numbers. gate [runs] 32 runs landed on cfg-0032, cfg-0033, cfg-0034, cfg-0035 (llama-bench) https://halobench.com/log/#e-2026-08-12-3 https://halobench.com/log/#e-2026-08-12-3 Wed, 12 Aug 2026 00:00:00 GMT 32 runs landed on cfg-0032, cfg-0033, cfg-0034, cfg-0035 (llama-bench) — records: cfg-0032, cfg-0033, cfg-0034, cfg-0035 runs [runs] 16 runs landed on cfg-0036, cfg-0037 (llama-bench) https://halobench.com/log/#e-2026-08-12-4 https://halobench.com/log/#e-2026-08-12-4 Wed, 12 Aug 2026 00:00:00 GMT 16 runs landed on cfg-0036, cfg-0037 (llama-bench) — records: cfg-0036, cfg-0037 runs [claim] A second independent Strix Halo source (llama.cpp PR #26856 + its Reddit write-up) reports Vulkan ahead of ROCm on decode at depth by ~10.5% on a clean same-binary comparison — same direction as clm-0031's +55% but a fifth the magnitude, confirming that figure was mostly build-gap and private patches. https://halobench.com/log/#e-2026-08-11-1 https://halobench.com/log/#e-2026-08-11-1 Tue, 11 Aug 2026 00:00:00 GMT A second independent Strix Halo source (llama.cpp PR #26856 + its Reddit write-up) reports Vulkan ahead of ROCm on decode at depth by ~10.5% on a clean same-binary comparison — same direction as clm-0031's +55% but a fifth the magnitude, confirming that figure was mostly build-gap and private patches. — records: clm-0044 claim [claim] The KV dequant patch removes most of quantised KV's agentic cost, not just its speed cost: on identical seeded tasks, patched q8_0 takes +9.3% more turns than patched f16 (234 vs 214 over 11 paired tasks) where the stock build cost +39% (clm-0038). https://halobench.com/log/#e-2026-08-11-2 https://halobench.com/log/#e-2026-08-11-2 Tue, 11 Aug 2026 00:00:00 GMT The KV dequant patch removes most of quantised KV's agentic cost, not just its speed cost: on identical seeded tasks, patched q8_0 takes +9.3% more turns than patched f16 (234 vs 214 over 11 paired tasks) where the stock build cost +39% (clm-0038). — records: clm-0045 claim [claim] The community BF16 flash-attention predictions (clm-0044) reproduce on this hardware under single-binary methodology: at 32k depth, stock Vulkan leads ROCm +17.4% prefill / +17.1% decode at f16 KV, and the PR-26856 patch inverts prefill to ROCm +46.9% with bf16 KV. https://halobench.com/log/#e-2026-08-11-3 https://halobench.com/log/#e-2026-08-11-3 Tue, 11 Aug 2026 00:00:00 GMT The community BF16 flash-attention predictions (clm-0044) reproduce on this hardware under single-binary methodology: at 32k depth, stock Vulkan leads ROCm +17.4% prefill / +17.1% decode at f16 KV, and the PR-26856 patch inverts prefill to ROCm +46.9% with bf16 KV. — records: clm-0046 claim [claim] Nemotron's quant confound resolves cleanly: on 12 identical seeded tasks, UD-Q4_K_M and UD-IQ4_XS produce IDENTICAL reward on every task (0.583 both), but IQ4_XS takes +11% more total turns (630 vs 567). https://halobench.com/log/#e-2026-08-11-4 https://halobench.com/log/#e-2026-08-11-4 Tue, 11 Aug 2026 00:00:00 GMT Nemotron's quant confound resolves cleanly: on 12 identical seeded tasks, UD-Q4_K_M and UD-IQ4_XS produce IDENTICAL reward on every task (0.583 both), but IQ4_XS takes +11% more total turns (630 vs 567). — records: clm-0047 claim [claim] Simulator identity changes what tau2 measures. https://halobench.com/log/#e-2026-08-11-5 https://halobench.com/log/#e-2026-08-11-5 Tue, 11 Aug 2026 00:00:00 GMT Simulator identity changes what tau2 measures. — records: clm-0048 claim [claim] The 122B has a reproducible rule-precedence bug in policy application: on tau2 airline task 9 it cancels a partially-flown reservation in 7 of 8 trials, every failure with the identical signature — cancel_reservation called without ever checking flight status. https://halobench.com/log/#e-2026-08-11-6 https://halobench.com/log/#e-2026-08-11-6 Tue, 11 Aug 2026 00:00:00 GMT The 122B has a reproducible rule-precedence bug in policy application: on tau2 airline task 9 it cancels a partially-flown reservation in 7 of 8 trials, every failure with the identical signature — cancel_reservation called without ever checking flight status. — records: clm-0049 claim [gate] muse-glimmer-30b: blocked → screened — Screened on min-62bf73d (its minimum build, anchor-calibrated: +2.8%/+0.9% vs fleet baseline, identical tau2 capability). https://halobench.com/log/#e-2026-08-11-7 https://halobench.com/log/#e-2026-08-11-7 Tue, 11 Aug 2026 00:00:00 GMT muse-glimmer-30b: blocked → screened — Screened on min-62bf73d (its minimum build, anchor-calibrated: +2.8%/+0.9% vs fleet baseline, identical tau2 capability). gate [runs] 9 runs landed on cfg-0029, cfg-0030, cfg-0025, cfg-0031, cfg-0028, cfg-0027 (tau2-bench-airline) https://halobench.com/log/#e-2026-08-11-8 https://halobench.com/log/#e-2026-08-11-8 Tue, 11 Aug 2026 00:00:00 GMT 9 runs landed on cfg-0029, cfg-0030, cfg-0025, cfg-0031, cfg-0028, cfg-0027 (tau2-bench-airline) — records: cfg-0029, cfg-0030, cfg-0025, cfg-0031, cfg-0028, cfg-0027 runs [claim] The 122B's real τ²-bench airline score is 0.545 +/-0.208, not the 1.00 reported by 5-task runs, which sampled only the easiest tasks in the domain. https://halobench.com/log/#e-2026-08-10-1 https://halobench.com/log/#e-2026-08-10-1 Mon, 10 Aug 2026 00:00:00 GMT The 122B's real τ²-bench airline score is 0.545 +/-0.208, not the 1.00 reported by 5-task runs, which sampled only the easiest tasks in the domain. — records: clm-0037 claim [claim] Paired on identical tasks, q8_0 KV costs TURN EFFICIENCY: 228 turns against f16's 164 over the same 9 tasks, +39%, taking more turns on 6 of 9 and fewer on 1. https://halobench.com/log/#e-2026-08-10-2 https://halobench.com/log/#e-2026-08-10-2 Mon, 10 Aug 2026 00:00:00 GMT Paired on identical tasks, q8_0 KV costs TURN EFFICIENCY: 228 turns against f16's 164 over the same 9 tasks, +39%, taking more turns on 6 of 9 and fewer on 1. — records: clm-0038 claim [claim] On reward the 122B and Nemotron are indistinguishable, but reward is the wrong headline: on tasks both get RIGHT, the 122B needs 19 turns and 2.0 minutes against Nemotron's 26 and 3.7 — 37% fewer loops and 85% less wall-clock to the same correct answer. https://halobench.com/log/#e-2026-08-10-3 https://halobench.com/log/#e-2026-08-10-3 Mon, 10 Aug 2026 00:00:00 GMT On reward the 122B and Nemotron are indistinguishable, but reward is the wrong headline: on tasks both get RIGHT, the 122B needs 19 turns and 2.0 minutes against Nemotron's 26 and 3.7 — 37% fewer loops and 85% less wall-clock to the same correct answer. — records: clm-0039 claim [superseded] by clm-0042's per-task measurement, which found the true energy cost roughly 10x lower — this run's figures are whole-arm totals padded by model loading and non-scoring tasks, not the model's actual energy per answer, and must not be used to rank models. https://halobench.com/log/#e-2026-08-10-4 https://halobench.com/log/#e-2026-08-10-4 Mon, 10 Aug 2026 00:00:00 GMT by clm-0042's per-task measurement, which found the true energy cost roughly 10x lower — this run's figures are whole-arm totals padded by model loading and non-scoring tasks, not the model's actual energy per answer, and must not be used to rank models. — records: clm-0040, clm-0042 superseded [claim] The KV dequant patch cuts energy 42% at 200k context — 146.1 Wh unpatched against 85.3 Wh patched for the same throughput benchmark — and patched q8_0 (85.3 Wh) beats f16 (89.6 Wh). https://halobench.com/log/#e-2026-08-10-5 https://halobench.com/log/#e-2026-08-10-5 Mon, 10 Aug 2026 00:00:00 GMT The KV dequant patch cuts energy 42% at 200k context — 146.1 Wh unpatched against 85.3 Wh patched for the same throughput benchmark — and patched q8_0 (85.3 Wh) beats f16 (89.6 Wh). — records: clm-0041 claim [claim] Per-task energy windows, matched to a common task set across arms, put a correct τ² answer at 6.81 Wh on the 122B with f16 KV, 9.48 Wh with q8_0, and 12.75 Wh on Nemotron. https://halobench.com/log/#e-2026-08-10-6 https://halobench.com/log/#e-2026-08-10-6 Mon, 10 Aug 2026 00:00:00 GMT Per-task energy windows, matched to a common task set across arms, put a correct τ² answer at 6.81 Wh on the 122B with f16 KV, 9.48 Wh with q8_0, and 12.75 Wh on Nemotron. — records: clm-0042 claim [claim] Every τ² arm was run with --user-llm set to the same model as --agent-llm, so cross-model comparisons changed the agent AND the user simulator together — exactly what the runbook forbids ("hold both --user-llm and the judge fixed across comparisons, or results re-baseline silently"). https://halobench.com/log/#e-2026-08-10-7 https://halobench.com/log/#e-2026-08-10-7 Mon, 10 Aug 2026 00:00:00 GMT Every τ² arm was run with --user-llm set to the same model as --agent-llm, so cross-model comparisons changed the agent AND the user simulator together — exactly what the runbook forbids ("hold both --user-llm and the judge fixed across comparisons, or results re-baseline silently"). — records: clm-0043 claim [gate] 7 candidates → acquired: cascade2-30b, gemma4-26b, glm-47-flash, laguna-s-21, lfm2-24b, muse-glimmer-30b, qwen36-27b-mtp https://halobench.com/log/#e-2026-08-10-8 https://halobench.com/log/#e-2026-08-10-8 Mon, 10 Aug 2026 00:00:00 GMT 7 candidates → acquired: cascade2-30b, gemma4-26b, glm-47-flash, laguna-s-21, lfm2-24b, muse-glimmer-30b, qwen36-27b-mtp gate [gate] 5 candidates → screened: cascade2-30b, gemma4-26b, glm-47-flash, laguna-s-21, lfm2-24b https://halobench.com/log/#e-2026-08-10-9 https://halobench.com/log/#e-2026-08-10-9 Mon, 10 Aug 2026 00:00:00 GMT 5 candidates → screened: cascade2-30b, gemma4-26b, glm-47-flash, laguna-s-21, lfm2-24b gate [gate] 16 candidates → listed: deepseek-v4-flash, gemma4-12b, glm-45-air, ling-30-flash, llama4-scout, maple-preview, muse-glimmer-30b, npu-embeddinggemma, npu-embeddinggemma, npu-lfm2, npu-qwen3-4b-thinking, npu-qwen3-4b-thinking, npu-whisper, npu-whisper, qwen38-27b, qwen38-27b https://halobench.com/log/#e-2026-08-10-10 https://halobench.com/log/#e-2026-08-10-10 Mon, 10 Aug 2026 00:00:00 GMT 16 candidates → listed: deepseek-v4-flash, gemma4-12b, glm-45-air, ling-30-flash, llama4-scout, maple-preview, muse-glimmer-30b, npu-embeddinggemma, npu-embeddinggemma, npu-lfm2, npu-qwen3-4b-thinking, npu-qwen3-4b-thinking, npu-whisper, npu-whisper, qwen38-27b, qwen38-27b — records: clm-0031, clm-0034, clm-0015, clm-0018 gate [gate] 4 candidates → blocked: laguna-s-21, laguna-s-21, muse-glimmer-30b, muse-glimmer-30b https://halobench.com/log/#e-2026-08-10-11 https://halobench.com/log/#e-2026-08-10-11 Mon, 10 Aug 2026 00:00:00 GMT 4 candidates → blocked: laguna-s-21, laguna-s-21, muse-glimmer-30b, muse-glimmer-30b gate [runs] 2 runs landed on cfg-0026, cfg-0027 (tau2-bench-airline) https://halobench.com/log/#e-2026-08-10-12 https://halobench.com/log/#e-2026-08-10-12 Mon, 10 Aug 2026 00:00:00 GMT 2 runs landed on cfg-0026, cfg-0027 (tau2-bench-airline) — records: cfg-0026, cfg-0027 runs [superseded] the Pass^1 = 1.000 reported by this run came from a 3-task subsample biased toward the domain's easiest tasks — the other two of the original five never terminated and were excluded as infrastructure errors. https://halobench.com/log/#e-2026-08-09-1 https://halobench.com/log/#e-2026-08-09-1 Sun, 09 Aug 2026 00:00:00 GMT the Pass^1 = 1.000 reported by this run came from a 3-task subsample biased toward the domain's easiest tasks — the other two of the original five never terminated and were excluded as infrastructure errors. — records: clm-0030, clm-0037 superseded [claim] A community DeepSeek-V4-Flash report independently confirms our mmap/GTT double-residency finding, demonstrates a 120 GiB GTT ceiling in production use, and — most consequentially — reports Vulkan BEATING ROCm on 3 of 4 cells including 55% faster decode at depth. https://halobench.com/log/#e-2026-08-09-2 https://halobench.com/log/#e-2026-08-09-2 Sun, 09 Aug 2026 00:00:00 GMT A community DeepSeek-V4-Flash report independently confirms our mmap/GTT double-residency finding, demonstrates a 120 GiB GTT ceiling in production use, and — most consequentially — reports Vulkan BEATING ROCm on 3 of 4 cells including 55% faster decode at depth. — records: clm-0031 claim