citable URL: https://halobench.com/records/clm-0056/ — this address never moves; the anchor /records/#clm-0056 keeps resolving
Qwen3.8-27B's KV cache on llama.cpp costs exactly 64.00 KiB per token — the hybrid-attention allocation working as designed, measured byte-exact on this box: only 16 of the 64 layers carry a conventional KV cache (the 3:1 linear-to-full-attention layout), and 16 layers x 4 KV heads x 256 head_dim x 2 (K+V) x 2 bytes (f16) = 65,536 bytes/token. The community-reported 8.1 GiB KV footprint at 32k context for this model is 64-layer arithmetic and is wrong as a llama.cpp planning number: the measured allocation at 32k is 2.00 GiB. Full serving footprints measured at 262,144 context (f16 KV, MTP head not loaded): Q8_0 42.2 GiB, UD-Q4_K_XL 32.6 GiB — either quant fits a 120 GiB GTT window at FULL declared context with more than 75 GiB to spare, so context length is not a fit constraint for this model on 128 GB hardware. Loading the MTP head for speculation adds 430.4 MiB (Q8_0) or 272.6 MiB (UD-Q4_K_XL) of weights.
verified 2026-08-15 · volatility low
Note — the record's own working
METHOD — qwen38-screen phase A: llama-server load-only probes, both backends, both quants, n_ctx 4096 / 32768 / 262144, GTT usage read from the amdgpu telemetry before and after each load (gtt_used delta), server torn down between probes. Raw records: aihydra ~/bench-results/qwen38-screen/phaseA/footprint.jsonl (12 probes, all loaded clean) — measured footprints, not runs in the corpus sense, because a load probe produces no throughput or capability metric; the evidence lives in the phase A files and the arithmetic below is checkable from them.
THE PER-TOKEN NUMBER, derived two independent ways that agree exactly: - Measured: ROCm GTT delta between c4096 and c262144 is 16,128 MiB for BOTH
quants (Q8_0 27,118.5 -> 43,246.5 MiB; UD-Q4_K_XL 17,274.5 -> 33,402.5 MiB)
over a 258,048-token context difference = 0.0625 MiB = 64.00 KiB/token, exact
to the reported MiB. (Vulkan totals run ~0.4-0.8 GiB higher at large context —
compute-buffer overhead, not KV; the ROCm deltas are the clean read.)
- Arithmetic: the GGUF declares 64 layers, but only the 16 full-attention layers
allocate KV (llama.cpp implements the hybrid layout for this arch); 16 x 4
kv_heads x 256 head_dim x 2 (K and V) x 2 bytes = 65,536 B/token. At 32,768
tokens: 2.00 GiB. At 262,144: 16.0 GiB.
THE COMMUNITY CORRECTION — the 8.1 GiB-at-32k figure circulating for this model (r/StrixHalo, same thread as the first Vulkan throughput rows) matches allocating a full-attention cache across ALL 64 layers (~8.6 GiB by the same arithmetic, within reporting error of the 8.1 read). Whatever produced that reading, current llama.cpp does NOT allocate it: the hybrid layout is honoured and the planning number for fit maths is 64 KiB/token, 4x smaller. This was flagged as an open instrumentation question on the candidate record before the screen ran, and the screen settled it on the first phase.
FIT — at the full declared 262,144 context: Q8_0 total 42.2 GiB (43,246.5 MiB GTT), UD-Q4_K_XL 32.6 GiB (33,402.5 MiB) against the box's 120 GiB GTT window (gttsize=122880). Weights alone: ~27.1 GiB Q8_0, ~17.3 GiB UD-Q4_K_XL (c4096 loads, which carry only 0.25 GiB of KV). The MTP-head weight deltas are from the GGUF tensor tables (mtp_num_hidden_layers 1: 430.4 MiB at Q8_0, 272.6 MiB at UD-Q4_K_XL); phase A probed serving loads without speculation, so those tensors were not resident in the probes above.