citable URL: https://halobench.com/records/clm-0055/ — this address never moves; the anchor /records/#clm-0055 keeps resolving
draft-mtp speculation on Qwen3.8-27B (stock 3653e6d, gfx1151) peaks at spec-draft-n-max=3 over CLEAN cells — Vulkan Q8_0 17.75 t/s (2.26x its 7.86 no-speculation floor, acceptance 0.626), Vulkan UD-Q4_K_XL 26.68 t/s (2.23x, 0.623), ROCm Q8_0 18.25 t/s (2.33x, 0.6235) — and at n_max >= 4 the feature is BROKEN on this model: after accumulated generation volume in a live session (sequential VARIED prompts at full length; not fresh servers, not one repeated prompt, not short generations), generations start terminating at 1 token with <|im_end|> (id 248046). The decisive measurement is that the TARGET model's own pre-sampling distribution puts im_end at logprob -0.084 (~92%) on a prompt the same server answers normally with speculation off — the speculative path is corrupting the target's forward pass, not merely mis-accepting drafts — and ignore_eos:true restores both correct text AND draft accounting. The hazard compounds into a measurement artifact: llama.cpp reports a 1-token generation as 1,000,000 tokens/s, so unfiltered throughput averages FLATTER exactly the broken cells; a community sweep on this silicon reporting a peak at n_max=5 sits on what is measured here as one of the two worst cells (11/15 degenerate on Q8_0), under a workload shape (single repeated codegen prompt) that this trigger analysis shows cannot reproduce the failure. Recommendation from measurement: draft-mtp ON, n_max hard-capped at 3 until the mechanism is understood upstream.
METHOD (sweep) — llama-server harness (llama-bench has no speculative support; spec params are force-set to 0 there), 5 varied prompts x 3 reps = 15 generations per cell, greedy (temperature 0, seed 42), n_predict 500, cache_prompt off, -c 32768, --load-mode none, stock 3653e6d, IOMMU on. The floor (spec off) travels in-session with every cell; the RATIO is the deliverable, absolutes are not comparable to llama-bench numbers. "degenerate" = predicted_n <= 1, and the per-cell degenerate fraction is part of the record:
Vulkan sweep, decode t/s median of clean samples (degenerate/attempted):
| n_max | Q8_0 | degen | UD-Q4_K_XL | degen | |---|---|---|---|---| | off | 7.86 | 0/15 | 11.98 | 0/15 | | 2 | 16.50 | 0/15 | 23.74 | 0/15 | | 3 | 17.75 | 0/15 | 26.68 | 0/15 | | 4 | (artifact) | 12/15 | 25.55 | 5/15 | | 5 | (artifact) | 11/15 | 24.73 | 7/15 | | 6 | 34.41 | 3/15 | 43.72 | 2/15 |
The Q8_0 n_max=4/5 "medians" computed naively are 1,000,000 t/s — the 1-token sentinel value, not a throughput; that is the artifact hazard in one line. The n_max=6 cells are faster on their clean samples and markedly LESS degenerate than 4/5 — unexplained, and left as an open question rather than a recommendation. The clean-cell peak is n_max=3 on every arm measured. The n_max >= 4 cells are deliberately NOT recorded as run records; the clean cells behind this claim are (run-0268 ROCm Q8_0, run-0269 Vulkan Q8_0, run-0270 Vulkan UD-Q4_K_XL).
THE EOS CLIFF, characterised (full evidence bundle: ~/bench-results/qwen38-screen/eos-cliff-evidence.md on the bench box, prepared for upstream review): - Trigger needs accumulated generation volume in one server session: fresh server
per request 0/6 degenerate; one repeated prompt sequentially 0/10; varied
prompts at n_predict 48 (~480 tokens total) 0/10; varied prompts at n_predict
500 hit the cliff by request 2-3 (~1,400 tokens) — 11/15 degenerate at n_max=5.
Recovery is possible mid-session (a later request produced a clean 500), so the
state is not latched.
- Affected requests show tokens_cached == tokens_evaluated (the KV slot WAS
reset — not stale prompt cache) and draft_n/draft_n_accepted ABSENT entirely,
vs 666-847 drafted on healthy requests in the same session.
- Not a sampler artifact: reproduces under greedy (seeds 42 and 1234,
bit-identical sequences) and under the vendor thinking sampler (temp 1.0,
top_p 0.95, top_k 20, min_p 0) at seeds 42 and 7 with seed-varying rates.
Bit-identical greedy failure sequences across four independent sessions,
across server restarts and drop_caches.
- Worse on Q8_0 (12/15 at n_max=4) than UD-Q4_K_XL (5/15) — the OPPOSITE of what
a quantisation-noise story predicts.
- ignore_eos:true control: same server, same n_max, same prompt — correct text
returns AND draft accounting reappears (draft_n 341, accepted 129). The
corruption expresses entirely through the EOG token.
- CHECKED against the nearest upstream issues; none matches this signature:
23302/23335 are draft-mtp token DIVERGENCE with complete generations (Metal);
25618 is quant-dependent divergence (and the Q8-worse observation here cuts
against it); 26750 is a CUDA acceptance collapse — HIP acceptance here (0.6235)
is statistically indistinguishable from Vulkan (0.6261), so it does not
generalise to this box.
CONFIDENCE NOTES — what is and is not established. Established by measurement: the trigger conditions, the n_max threshold, the pre-sampling distribution shift onto im_end, the ignore_eos control, backend-indifferent acceptance at n_max=3, and determinism. NOT established: the root cause (the MTP head's internal state is the natural suspect but was not instrumented), why a single repeated prompt escapes, and why n_max=6 partially recovers. ROCm at n_max >= 4 was not probed for the symptom (the clean ROCm n_max=3 cell is not evidence either way). UPSTREAM: not yet filed — the evidence bundle is written for review first; this claim should be re-verified against whatever the eventual issue thread establishes. Confidence is medium on the characterisation and the n_max<=3 operating rule; the mechanism language above is deliberately interpretation-free.
COMMUNITY CONFLICT, stated fairly: the r/StrixHalo sweep reporting draft-mtp peaking at n_max=5 for this model family ran a different build lineage (the strix-halo-vulkan fork, b10283/b10397 era), amd_iommu=off, and a single repeated 500-token codegen prompt — the exact workload shape measured here as unable to trigger the cliff — and llama-bench-style unfiltered averages cannot distinguish a fast cell from a broken one once 1-token generations enter the mean. Their numbers may be correct for their harness; the recommendation that follows from them is what this claim disputes for live serving. Guard evidence for the recommended operating point: 25 sequential full-length vendor-sampler generations at n_max=3 on the tau2 serving config, 0/25 degenerate, ~11,000 generated tokens in one session (run-0267) — deeper session volume than any failing cell needed.