Research · local models · known state
Measured Recipes, Frozen Expectations, and Self-Describing Runs for Eight Quantized LLMs on One RTX 3090
Eight quantized language models — a 27 B dense, a 35 B-A3B mixture of experts, a 12 B dense, a 26 B-A4B block-diffusion model and four 4–9 B models — are served one at a time on a single RTX 3090 through a swapping proxy and used by an agent orchestrator. Before this work each model's deployment, call settings and quality evidence lived in three files that drifted independently, and an adversarial review of a planned experiment found that the baseline it needed did not exist. We define a known state per model — one versioned spec holding deployment assertions, a call recipe, behavioural contracts and frozen expectations with explicit tolerances — and a thirteen-probe suite that measures it, including a whole-file edit task modelled on the orchestrator's actual job alongside the usual write-a-function task. A first sweep (200 scored code replies, 50 scored edits, 75 minutes of GPU time) found that the function task is bimodal at temperature 0.2 on three of the four large models (the same configuration scored 8.0, 6.0 and 6.4 on Gemma-4-12B across three runs) while the edit task was 7/7 in every sample on six of eight models; that at temperature 0 the dense models are byte-deterministic but the MoE model is not; that reasoning_effort exists only in one of the eight chat templates, so three "thinking" recipes had been measuring something else; and that the strongest model scores the same with thinking off at 40 % of the tokens. Two method rules came out of the data: freeze tolerances derived from the observed spread, and recipe selection that treats variants within the noise floor as ties broken by cost. Every number is traceable to a run record that embeds the prompts, request extras and container command it was produced with.
Doug (freePerro) with Claude · September 2026 · working draft — results sections are filled from the first full sweep; every number in this paper is traceable to a run record under logs/state/.
A local model served on consumer hardware is not a checkpoint. It is a checkpoint, a quantization, a serving engine and its flags, a context window chosen by VRAM rather than by the model card, a prompt template with thinking on or off, a sampling configuration, and an output budget. Change any of these and the thing that answers is different. In the setup studied here, eight such models share one RTX 3090 through a swapping proxy, and the software that routes work to them (an agent orchestrator) needs to know, for each one, what it can be relied on to do.
Before this work, that knowledge was distributed across three files that could drift independently — the docker command per model, a shell script with the call flags used for the last measurement, and the orchestrator's per-model profiles — and its quality evidence was three hand-scored samples copied into a results file. An adversarial review of a proposed follow-up experiment (measuring how quality degrades with prompt length) established that the follow-up could not be interpreted, because the baseline it would be measured against was itself unknown: the same model had scored 8/8 in one run and error/4/4 in another with a different prompt; the strongest model's 8/8 ×3 had been obtained with an output budget twice what the proposal assumed; and with three samples per cell an 8-item test cannot support a pass/fail threshold.
This paper describes the remedy — a per-model specification that is the single source of truth, a probe suite with frozen expectations and explicit tolerances, a recipe-selection procedure that chooses call settings by measurement, and run records that carry everything they were produced with — and reports the first full sweep across all eight models. It also reports what the sweep found that the previous measurements had missed.
Contributions:
reasoning_effort is honoured by one template in eight; thinking off matches thinking on at 40 % of the tokens on the strongest model; a diffusion model's max_tokens behaves as a target, not a cap; two sampling/thinking claims in the previous results file did not reproduce.Executable tests as the metric. Scoring generated code by running hidden tests rather than by string similarity follows HumanEval [1]; we keep the model's own tests as a secondary signal and score only on tests it never saw.
Prompt and sampling sensitivity. Small changes in prompt formatting move benchmark scores by large margins [2]; the "tuned" prompt variant used here (a one-block system prompt plus a stated unit convention, with temperature 0.2) had lifted one model from 6/8 to 8/8 ×3 in an earlier controlled re-test and was assumed to help every model. §8 tests that assumption per model.
Long-context degradation. Position and length effects ("lost in the middle" [3], RULER [4], NoLiMa [5], the Chroma context-rot report [6]) motivated the deferred cliff measurement; this paper establishes the zero-length floor that measurement needs.
Serving. vLLM [7] with paged attention serves every model; weights are 4-bit (AWQ [8], GPTQ [9], AutoRound [10]) except the bf16 small models; one model uses a speculative drafter [11], which is rejection-sampled and therefore does not change the output distribution — we treat it as a speed feature, not a quality confound.
One RTX 3090 (24 GB GDDR6X, Ampere sm_86) power-limited to 250 W. Desktop rendering is pinned to a separate RTX 5060 so that the 3090 holds only the inference engine; this was verified with nvidia-smi immediately before the sweep (one process on the card: VLLM::EngineCore). Host: Linux 7.0, Python 3.14.
llama-swap v251 listens on 127.0.0.1:11440 and runs one docker run per model, stopping the previous container before starting the next (one resident model; a swap costs 1–5 minutes of container start and graph capture). Seven models run on the stock vllm/vllm-openai:v0.28.0 image; Qwen3.8-27B runs on a third-party image built on a vLLM 0.27.1 fork with a W4A16 requant and a DFlash2 drafter. Every entry sets useModelName (without it llama-swap forwards the alias verbatim and the inner vLLM returns 404 — a fault found the day before this work). Two proxy routes matter: /v1/* (OpenAI and Anthropic APIs) and /upstream/<model>/*, through which /tokenize returns the exact token count and the live max_model_len.
| id | checkpoint / quant | params | window (--max-model-len) | status |
|---|---|---|---|---|
| qwen3.8-27b | Qwen3.8-27B, AutoRound W4A16 body + GPTQ-int4 head, DFlash2 W4A16 drafter | 27 B dense | 65,536 | keep |
| ornith-1.5-35b | Ornith-1.5-35B-A3B (Tiel-Coder weights), AutoRound int4, bf16 MTP head | 35 B MoE, 3 B active | 32,768 | keep |
| gemma-4-12b | Gemma-4-12B-it, official QAT W4A16 | 12 B dense | 65,536 | keep |
| diffusiongemma-26b | DiffusionGemma-26B-A4B-it, AWQ-INT4 (symmetric, g32), block diffusion | 26 B MoE, 4 B active | 32,768 | bench-only |
| ornith-1.5-9b | Ornith-1.5-9B, bf16 | 9 B | 65,536 | candidate-retire |
| qwen3.8-9b-distill | Qwen3.8-9B-Distill, bf16 | 9 B | 65,536 | candidate-retire |
| qwen3.5-4b | Qwen3.5-4B, bf16 | 4 B | 131,072 | candidate-retire |
| qwen3.5-4b-awq | Qwen3.5-4B, AWQ-INT4 | 4 B | 131,072 | candidate-retire |
Windows are VRAM choices (KV-cache budget on a 24 GB card), not checkpoint limits; Qwen3.8-27B is natively 262K. status is an operational label carried in the spec: keep models receive routed work, bench-only is measured but not routed, candidate-retire is measured once so that the decision to drop it is recorded with numbers.
For each model, one YAML file (specs/<model>.yaml) holds:
max_model_len, max_num_seqs, and a list of substrings that must appear in the docker command. The deployment probe asserts these against the live llama-swap.yaml; the tokenize probe asserts max_model_len against the running server.off, or a reasoning_effort level, or the template default), prompt variant (tuned / plain), max_tokens, sampling parameters or no_sampling_params, sample count. This is the call recipe: the suite derives the exact request extras from it, and the orchestrator's shipped profile is generated from it.temperature) encoded as expectations rather than treated as failures.The spec is the only place any of this is typed. The docker configuration is asserted against it, the orchestrator's profiles are derived from it, and every run record embeds the prompts, request extras and verbatim docker command it ran with, so that a record from a month ago can be compared with today's without guessing what changed.
| probe | request | locks in | passes when |
|---|---|---|---|
listed | GET /v1/models | the model is offered | id present |
deployment | reads llama-swap.yaml | the docker cmd matches the spec | all cmd_contains substrings and the image present; useModelName set |
tokenize | POST /upstream/<m>/tokenize | live window = spec; swap time | max_model_len equal, count > 0 |
short | "Reply with exactly: ok" ×2 | instruction following; warm TTFT | reply contains "ok" (or the recorded eos quirk) |
thinking | "What is 17*23? Answer with the number only." | the thinking mode does what it says | off → 0 reasoning tokens; on → some; finish_reason=stop; 391 present |
sampling | "Say hi." with temperature 0.2 | whether sampling params are accepted | 200, or 400 where the spec says no_sampling_params |
over_window | max_tokens = max_model_len + 1 | the 400 body names the limit | 400 and body contains a limit marker |
length_cap | the code task at max_tokens 16 | truncation reported as length | finish_reason == "length" |
code ×N | §5.2 | code-quality floor | mean within tolerance of frozen |
edit ×N | §5.3 | whole-file edit floor | mean within tolerance of frozen |
cache | 6.1K-token prompt, then a second turn | prefill rate; prefix-cache TTFT | second-turn TTFT ≤ 2× frozen |
tools | one tool call | OpenAI tool calling | right function and argument |
anthropic | POST /v1/messages | the Anthropic route | text returned (or the recorded value) |
Contract probes send only the thinking kwargs; task probes send the full recipe. All chat calls are streamed with stream_options.include_usage, so token counts are the server's, and finish_reason is captured from the stream. The over_window and length_cap probes exist because the orchestrator's overflow classifier and its output-floor logic depend on those two server behaviours; a serving-engine upgrade that changed either would be caught here before any work order was affected.
Write parse_door_size(s: str) -> tuple[int, int] that parses sizes such as 16x7, 16' x 7', 16 ft by 7 ft, 192" x 84" into inches, with eight pytest cases. This is the task from the earlier bake-off, unchanged, so the scores are comparable. Two prompt variants: plain (the bare request) and tuned (a system prompt demanding exactly one fenced block, no revisions, raw regex strings; a user prompt that states the unit convention; temperature 0.2, top_p 0.95, presence_penalty 1.0). Scoring: every fenced block is joined and written as a module; if that does not parse, the first block that parses and defines the function is used instead; the model's own tests run first, then eight hidden external tests the model never saw. The reported score is hidden passes out of 8; compiles records whether the module parsed and pytest collected.
Given a 60-line door_quote.py (a dataclass, five pure functions and a formatter), add a keyword parameter rush: bool = False to quote(); when true, add a rush_fee line item equal to 12 % of panels + labor + opener, placed after opener and before subtotal, included in the subtotal so that discount and tax apply to it; when false, the returned dict must be exactly what it was; change nothing else; return the entire file in one fenced block.
This models the orchestrator's implementer, which asks a local model for a whole file with one change and replaces the file with the reply. Scoring: the last fenced block that defines quote is written as door_quote.py and run against seven hidden tests — constants unchanged, helpers unchanged, signature, no-rush output identical to before, rush line item present and in the right position, rush × discount × tax arithmetic, validation and formatting unchanged. Two further measurements are recorded: preserved, difflib's line-similarity between the original and the returned file, and Δ lines, the unified-diff line count. The reference solution scores 7/7 with Δ4; the unchanged file scores 4/7, failing exactly the three rush tests.
For each model, refine.py runs the code task five times under each variant listed in the spec — thinking off or reasoning_effort=low, prompt tuned or plain, max_tokens 4096 or 8192 (DiffusionGemma has neither thinking nor sampling variants: vLLM rejects both for diffusion models) — and ranks variants by mean hidden score, then compile rate, then fewest length-capped replies, then fewest completion tokens — with a correction for noise that the first sweep forced (§9): a variant within the best variant's noise floor (max(1.0, 2 × SE of its samples), capped at 2.0) with no worse compile or length-cap rate is a tie, and a tied variant displaces the best only if it needs at least 25 % fewer completion tokens. The winner is written into the spec's recipe; the whole table is kept under refined and in a JSON record. The recipe is chosen on the code task and the edit task is then measured with it; §10 lists this as a limitation.
modeltest.py --freeze writes the run's measurements into expectations. A later run passes a task probe while its mean is at least the frozen mean minus a tolerance, its compile rate is within 0.2 of the frozen rate, and its length-capped rate is not more than 0.2 above; speed passes while median decode is at least 0.8× the frozen median and the cached second turn is at most 2× the frozen value; contract probes must return the recorded value.
The tolerance was initially a fixed 1.0 (one test case, the resolution of an 8-item test with N = 5). The first freeze changed the rule: Ornith-1.5-35B scored 8, 8, 8, 6, 0 in recipe selection and 0, 8, 4, 4, 6 in the freeze run five minutes later with the identical recipe, while its edit task scored 7, 7, 7, 7, 7 in both. A fixed ±1.0 would have called the second run a regression. The tolerance is now max(1.0, 2 × SE) where SE is the standard error of the N samples — 2.65 for that model's code task and 1.0 for its edit task — and the per-sample values and standard deviation are frozen alongside the mean so the spread is visible, not hidden in an average.
The sweep ran 2026-08-31 from 20:28 to 21:43 (eight models, refine then freeze, one resident at a time), followed by a fix-up pass to 21:53 that re-judged Qwen's variants under the corrected tie rule, relabelled Gemma's recipe, re-froze both, ran a greedy A/A on the three kept models and re-measured DiffusionGemma's sampling contract per parameter. In total 200 code replies and 50 edit replies were scored by hidden tests. Tables 1–4 are generated by bin/paper_tables.py from the run records; the run id in the last column of Table 1 names the JSON file each row comes from.
| model | status | recipe (thinking / prompt / max_tokens) | code /8 mean (samples) | SD | edit /7 mean (samples) | preserved | tokens/reply (code) | decode tok/s | prefill tok/s | cached turn | swap | run |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| qwen3.8-27b | keep | off / tuned / 4096 | 8 (8,8,8,8,8) | 0.0 | 7 (7,7,7,7,7) | 0.967 | 1005 | 184.5 | 1161 | 0.75 s | 53.4 s | 20260831-214346 |
| ornith-1.5-35b | keep | off / tuned / 4096 | 4.4 (0,8,4,4,6) | 2.97 | 7 (7,7,7,7,7) | 0.922 | 352 | 214.9 | 6621 | 0.31 s | 0.0 s | 20260831-203413 |
| gemma-4-12b | keep | off / tuned / 8192 | 6.4 (8,0,8,8,8) | 3.58 | 7 (7,7,7,7,7) | 0.92 | 568 | 77.6 | 2233 | 0.095 s | 60.3 s | 20260831-214607 |
| diffusiongemma-26b | bench-only | off / tuned / 2048 (no sampling params) | 3.6 (4,0,8,6,0) | 3.58 | 7 (7,7,7,7,7) | 0.947 | 440 | 407.3 | 1967 | 2.041 s | 0.0 s | 20260831-204828 |
| ornith-1.5-9b | candidate-retire | off / tuned / 4096 | 3.2 (2,4,4,0,6) | 2.28 | 6.8 (7,7,6,7,7) | 0.943 | 565 | 44.4 | 3705 | 0.143 s | 0.0 s | 20260831-210903 |
| qwen3.8-9b-distill | candidate-retire | default / tuned / 8192 | 1.2 (2,4,0,0,0) | 1.79 | 5.4 (5,6,6,5,5) | 0.91 | 781 | 44.5 | 3709 | 0.135 s | 0.0 s | 20260831-212017 |
| qwen3.5-4b | candidate-retire | off / tuned / 4096 | 1.6 (0,0,8,0,0) | 3.58 | 7 (7,7,7,7,7) | 0.955 | 1101 | 75.5 | 6320 | 0.085 s | 0.0 s | 20260831-213628 |
| qwen3.5-4b-awq | candidate-retire | off / tuned / 4096 | 1.8 (3,6,0,0,0) | 2.68 | 7 (7,7,7,7,7) | 0.942 | 863 | 170.4 | 6757 | 0.088 s | 0.0 s | 20260831-214237 |
gemma-4-12b 20260831-214607: failing probes code — 77.6 tok/s median, 568 tok meandiffusiongemma-26b 20260831-204828: failing probes short, thinking, sampling — warm ttft 0.25s, 1 tok, budget 8; mode=off reasoning 0 chars, finish=stop, answer MISSING in content; 200qwen3.8-9b-distill 20260831-212017: failing probes anthropic — stop=max_tokens blocks=['thinking']qwen3.5-4b 20260831-213628: failing probes anthropic — stop=max_tokens blocks=['thinking']qwen3.5-4b-awq 20260831-214237: failing probes anthropic — stop=max_tokens blocks=['thinking']qwen3.8-27b
| variant | mean /8 | samples | compile | length-capped | tokens | reasoning chars | tok/s |
|---|---|---|---|---|---|---|---|
| thinking=low prompt=plain max_tokens=8192 | 8 | 8,8,8,8,8 | 1.0 | 0.0 | 2836 | 6560 | 188.9 |
| thinking=low prompt=tuned max_tokens=8192 | 7.6 | 8,8,8,6,8 | 1.0 | 0.0 | 2331 | 4975 | 194.8 |
| thinking=off prompt=tuned max_tokens=4096 | 7.4 | 5,8,8,8,8 | 1.0 | 0.0 | 950 | 0 | 178.8 |
| thinking=off prompt=plain max_tokens=4096 | 7.2 | 6,8,8,8,6 | 1.0 | 0.0 | 1300 | 0 | 182.1 |
chosen: {"thinking": "off", "prompt": "tuned", "max_tokens": 4096} (rule cheap, tie floor 1.0; run 20260831-203446)
ornith-1.5-35b
| variant | mean /8 | samples | compile | length-capped | tokens | reasoning chars | tok/s |
|---|---|---|---|---|---|---|---|
| thinking=off prompt=tuned max_tokens=4096 | 6 | 8,8,8,6,0 | 0.8 | 0.0 | 484 | 0 | 217.0 |
| thinking=low prompt=tuned max_tokens=8192 | 4.4 | 0,6,8,8,0 | 1.0 | 0.0 | 4514 | 12228 | 177.2 |
| thinking=off prompt=plain max_tokens=4096 | 3.2 | 5,0,5,2,4 | 0.8 | 0.0 | 631 | 0 | 207.0 |
| thinking=low prompt=plain max_tokens=8192 | 2.6 | 0,8,5,0,0 | 0.8 | 0.4 | 6721 | 20141 | 162.2 |
chosen: {"thinking": "off", "prompt": "tuned", "max_tokens": 4096} (rule best, tie floor None; run 20260831-202801)
gemma-4-12b
| variant | mean /8 | samples | compile | length-capped | tokens | reasoning chars | tok/s |
|---|---|---|---|---|---|---|---|
| thinking=low prompt=tuned max_tokens=8192 | 8 | 8,8,8,8,8 | 1.0 | 0.0 | 672 | 0 | 76.6 |
| thinking=off prompt=tuned max_tokens=4096 | 6 | 8,8,6,0,8 | 0.8 | 0.0 | 805 | 0 | 77.1 |
| thinking=off prompt=plain max_tokens=4096 | 4.8 | 6,8,6,2,2 | 1.0 | 0.0 | 724 | 0 | 77.3 |
chosen: {"thinking": "low", "prompt": "tuned", "max_tokens": 8192} (rule cheap, tie floor 1.0; run 20260831-204059)
diffusiongemma-26b
| variant | mean /8 | samples | compile | length-capped | tokens | reasoning chars | tok/s |
|---|---|---|---|---|---|---|---|
| thinking=off prompt=tuned max_tokens=2048 | 8 | 8,8,8,8,8 | 1.0 | 0.0 | 483 | 0 | 330.7 |
| thinking=off prompt=tuned max_tokens=4096 | 4 | 8,4,8,0,0 | 1.0 | 0.0 | 868 | 0 | 413.6 |
| thinking=off prompt=plain max_tokens=2048 | 1.6 | 0,8,0,0,0 | 0.4 | 0.0 | 690 | 0 | 348.1 |
chosen: {"thinking": "off", "prompt": "tuned", "max_tokens": 2048} (rule cheap, tie floor 1.0; run 20260831-204610)
ornith-1.5-9b
| variant | mean /8 | samples | compile | length-capped | tokens | reasoning chars | tok/s |
|---|---|---|---|---|---|---|---|
| thinking=off prompt=tuned max_tokens=4096 | 2.4 | 5,0,2,5,0 | 1.0 | 0.0 | 576 | 0 | 44.5 |
| thinking=off prompt=plain max_tokens=4096 | 1.2 | 0,0,0,6,0 | 0.4 | 0.2 | 1881 | 0 | 44.2 |
| thinking=default prompt=tuned max_tokens=8192 | 0 | 0,0,0,0,0 | 0.0 | 0.6 | 7530 | 22332 | 44.0 |
chosen: {"thinking": "off", "prompt": "tuned", "max_tokens": 4096} (rule cheap, tie floor 2.0; run 20260831-204900)
qwen3.8-9b-distill
| variant | mean /8 | samples | compile | length-capped | tokens | reasoning chars | tok/s |
|---|---|---|---|---|---|---|---|
| thinking=off prompt=tuned max_tokens=4096 | 3.6 | 2,6,4,0,6 | 0.8 | 0.2 | 1516 | 0 | 44.5 |
| thinking=default prompt=tuned max_tokens=8192 | 3.2 | 0,0,0,8,8 | 1.0 | 0.0 | 916 | 717 | 44.5 |
| thinking=off prompt=plain max_tokens=4096 | 2.2 | 1,0,0,5,5 | 0.8 | 0.2 | 1540 | 0 | 44.6 |
chosen: {"thinking": "default", "prompt": "tuned", "max_tokens": 8192} (rule cheap, tie floor 2.0; run 20260831-211143)
qwen3.5-4b
| variant | mean /8 | samples | compile | length-capped | tokens | reasoning chars | tok/s |
|---|---|---|---|---|---|---|---|
| thinking=off prompt=tuned max_tokens=4096 | 4 | 0,8,6,6,0 | 1.0 | 0.0 | 783 | 0 | 75.6 |
| thinking=off prompt=plain max_tokens=4096 | 1.4 | 3,2,2,0,0 | 1.0 | 0.0 | 870 | 0 | 75.2 |
| thinking=default prompt=tuned max_tokens=8192 | 0 | 0,0,0,0,0 | 0.2 | 1.0 | 8192 | 20484 | 74.0 |
chosen: {"thinking": "off", "prompt": "tuned", "max_tokens": 4096} (rule cheap, tie floor 2.0; run 20260831-212423)
qwen3.5-4b-awq
| variant | mean /8 | samples | compile | length-capped | tokens | reasoning chars | tok/s |
|---|---|---|---|---|---|---|---|
| thinking=off prompt=tuned max_tokens=4096 | 1.2 | 0,0,0,6,0 | 1.0 | 0.0 | 647 | 0 | 172.0 |
| thinking=off prompt=plain max_tokens=4096 | 1 | 0,3,0,0,2 | 0.8 | 0.0 | 644 | 0 | 165.8 |
| thinking=default prompt=tuned max_tokens=8192 | 0 | 0,0,0,0,0 | 0.8 | 0.4 | 4083 | 13973 | 169.9 |
chosen: {"thinking": "off", "prompt": "tuned", "max_tokens": 4096} (rule cheap, tie floor 2.0; run 20260831-213852)
| model | listed | deployment | tokenize | short | thinking | sampling | over_window | length_cap | cache | tools | anthropic |
|---|---|---|---|---|---|---|---|---|---|---|---|
| qwen3.8-27b | ✓ ['qwen3.8-27b'] | ✓ matches | ✓ 65536 | ✓ ok | ✓ none/answered | ✓ accepted (all 5 ac | ✓ 400+named | ✓ length | ✓ 0.75 | ✓ ok | ✓ ok |
| ornith-1.5-35b | ✓ ['ornith-1.5-35b'] | ✓ matches | ✓ 32768 | ✓ ok | ✓ none | ✓ accepted | ✓ 400+named | ✓ length | ✓ 0.31 | ✓ ok | ✓ ok |
| gemma-4-12b | ✓ ['gemma-4-12b'] | ✓ matches | ✓ 65536 | ✓ ok | ✓ none/answered | ✓ accepted (all 5 ac | ✓ 400+named | ✓ length | ✓ 0.09 | ✓ ok | ✓ ok |
| diffusiongemma-26b | ✓ ['diffusiongemma-2 | ✓ matches | ✓ 32768 | ✗ eos | ✗ none | ✗ accepted | ✓ 400+named | ✓ length | ✓ 2.04 | ✓ ok | ✓ ok |
| ornith-1.5-9b | ✓ ['ornith-1.5-9b'] | ✓ matches | ✓ 65536 | ✓ ok | ✓ none/answered | ✓ accepted (all 5 ac | ✓ 400+named | ✓ length | ✓ 0.14 | ✓ ok | ✓ ok |
| qwen3.8-9b-distill | ✓ ['qwen3.8-9b-disti | ✓ matches | ✓ 65536 | ✓ ok | ✓ some/answered | ✓ accepted (all 5 ac | ✓ 400+named | ✓ length | ✓ 0.13 | ✓ ok | ✗ no-text |
| qwen3.5-4b | ✓ ['qwen3.5-4b'] | ✓ matches | ✓ 131072 | ✓ ok | ✓ none/answered | ✓ accepted (all 5 ac | ✓ 400+named | ✓ length | ✓ 0.09 | ✓ ok | ✗ no-text |
| qwen3.5-4b-awq | ✓ ['qwen3.5-4b-awq'] | ✓ matches | ✓ 131072 | ✓ ok | ✓ none/answered | ✓ accepted (all 5 ac | ✓ 400+named | ✓ length | ✓ 0.09 | ✓ ok | ✗ no-text |
| model | re-test n=3 | this sweep, same variant, n=5 | mean | verdict from n=3 | verdict from n=5 |
|---|---|---|---|---|---|
| qwen3.8-27b | 8,8,8 | 5,8,8,8,8 | 7.4 | usable (mean 8) | usable (mean 7.4, SD 1.34) |
| ornith-1.5-35b | err,4,4 | 8,8,8,6,0 | 6 | unusable (mean 4) | usable (mean 6, SD 3.46) |
| gemma-4-12b | 8,8,6 | 8,8,6,0,8 | 6 | usable (mean 7.3) | usable (mean 6, SD 3.46) |
| diffusiongemma-26b | 0,0,8 | 8,8,8,8,8 | 8 | unusable (mean 2.7) | usable (mean 8, SD 0.0) |
| ornith-1.5-9b | 5,0,2 | 5,0,2,5,0 | 2.4 | unusable (mean 2.3) | unusable (mean 2.4, SD 2.51) |
| qwen3.8-9b-distill | 2,6,4 | 2,6,4,0,6 | 3.6 | unusable (mean 4) | unusable (mean 3.6, SD 2.61) |
| qwen3.5-4b | 0,8,6 | 0,8,6,6,0 | 4 | unusable (mean 4.7) | unusable (mean 4, SD 3.74) |
| qwen3.5-4b-awq | 2,0,4 | 0,0,0,6,0 | 1.2 | unusable (mean 2) | unusable (mean 1.2, SD 2.68) |
Greedy A/A (temperature 0, top_p 1, five samples, code task). Qwen3.8-27B: 8,8,8,8,8, five byte-identical replies (one md5), 1,122 tokens mean. Gemma-4-12B: 8,8,8,8,8, five byte-identical replies, 525 tokens. Ornith-1.5-35B: 4,0,8,8,8, five different replies (five md5s, 1,239–1,597 bytes, first divergence at byte 89). The same container without MTP speculative decoding (ornith-1.5-35b-nomtp, run ornith-1.5-35b-nomtp/refine-20260831-222154) still produced five different greedy replies, 0,0,8,4,8, at 168 tok/s instead of 215–237. Runs qwen3.8-27b/refine-20260831-214535, gemma-4-12b/refine-20260831-214847, ornith-1.5-35b/refine-20260831-214923.
Chat-template audit (local tokenizer files). reasoning_effort appears in the Qwen3.8-27B template (reasoning_effort|default('xhigh'), accepted values xhigh/medium/low) and in none of the other seven. Gemma-4 (12B and DiffusionGemma): enable_thinking | default(false) — thinking off unless asked. Ornith-1.5 (35B and 9B), Qwen3.8-9B-Distill, Qwen3.5-4B (both): Qwen3-style, enable_thinking is false emits an empty <think> block, otherwise thinking is on by default.
Three of the four large models flipped between runs of the same recipe minutes apart: Ornith-1.5-35B 8,8,8,6,0 → 0,8,4,4,6; DiffusionGemma 8,8,8,8,8 → 4,0,8,6,0; Gemma-4-12B 8,8,8,8,8 → 8,8,6,0,8 → 8,0,8,8,8, where the first two runs were labelled "low" and "off" but rendered the identical prompt (§9.3), making them an accidental A/A test: means 8.0, 6.0 and 6.4 from one configuration. The per-sample standard deviations in Table 1 (2.3–3.6 on an 8-point scale) say the same thing. The outcomes are bimodal — a reply is usually a correct program or a broken one — so a mean over five is a coarse estimate and a threshold of one test case cannot separate signal from sampling. Qwen3.8-27B is the exception: 8/8 in 15 of 15 samples at temperature 0.2 under its chosen recipe (5/8 once in the plain-ordering refine table).
The edit task did not flip. Six of eight models scored 7/7 in all five samples, with 92–97 % of the file preserved and diffs of 4–11 lines (the reference solution is Δ4); Ornith-9B scored 7,7,6,7,7 and Qwen3.8-9B-Distill 5,6,6,5,5. Copying a 60-line file and inserting one keyword parameter is a lower-entropy job than writing a parser from a specification: most tokens are constrained by the input, and the free choices are few. That is also the orchestrator's job, which is the point of the task: on this evidence the whole-file Replace step is not where the small models fail, and a 4 B model that scores 1.6/8 on the function task still returned a correct edited file five times out of five.
At temperature 0 the two dense models produced five byte-identical replies each — the stack (vLLM 0.28.0 and the 0.27.1 fork, W4A16 Marlin kernels, the DFlash2 drafter, prefix caching) is deterministic end to end for them, and all of their temperature-0.2 spread is sampling. Ornith-1.5-35B is not deterministic at temperature 0: five different replies diverging within the first hundred bytes. It is the only kept model that is a mixture of experts (AutoRound int4 with Marlin MoE kernels) and the only one using MTP speculative decoding. Removing MTP did not restore determinism (five different greedy replies without it), so the drafter is exonerated and the source is in the MoE path — expert routing or the int4 Marlin MoE kernels — which is the next isolation experiment (VLLM_MARLIN_USE_ATOMIC_ADD, then prefix caching, one at a time). For the suite this means a greedy regression mode pins the dense models exactly (any change is real) but cannot pin Ornith, which keeps the N = 5, spread-tolerance regime.
The "tuned" prompt (one fenced block, stated unit convention, temperature 0.2) had been adopted on the strength of one model's re-test. Table 2 measures it per model against the plain prompt at the same thinking setting: decisive for DiffusionGemma (8.0 vs 1.6, compile rate 1.0 vs 0.4), large for Ornith (6.0 vs 3.2) and Gemma-4-12B (6.0 vs 4.8), and within noise on Qwen (7.4 vs 7.2 with thinking off; 7.6 vs 8.0 with effort low). It helps the models that need help.
Thinking is a different story per template. On Qwen3.8-27B, whose template honours reasoning_effort, effort=low and thinking-off both reach 8/8 — at 2,502 versus 1,005 completion tokens in the freeze runs — so the recipe is thinking off. On Ornith, whose template has no reasoning_effort, the "low" variants were default-effort thinking: 12–20 K characters of reasoning per reply, two of five replies length-capped at 8,192 tokens, and worse scores (4.4 and 2.6). On Gemma-4-12B the kwarg is silently ignored, which is how the A/A test in §9.1 happened. The deployment probe now reads the chat template and fails any recipe that claims an effort level the template cannot honour; the paper's own first refine tables carry the mislabelled rows, kept as recorded.
Table 4 puts the 2026-08-30 re-test (three samples, tuned, thinking off) beside the same variant in this sweep (five samples). Ornith-1.5-35B was "err, 4, 4" and read as unusable; five samples gave 8,8,8,6,0. DiffusionGemma was 0, 0, 8 and read as the weakest; five samples gave 8,8,8,8,8 — and the freeze run five minutes later gave 4,0,8,6,0. Both verdicts were sampling. The honest statement is that on these two models the function task's outcome distribution is wide enough that neither three nor five samples locate the mean to within a test case; the frozen expectations carry the per-sample values and a tolerance of two standard errors so that the next run is judged against the spread, not the point.
Decode rates (Table 1) confirm the earlier observation that a 27 B W4A16 model on this card is faster than a 9 B bf16 one (185 vs 44–45 tok/s; the bf16 models are bandwidth-bound). Prefill separates the architectures more than the sizes: Ornith 6,600 tok/s, the 4 B models 6,300–6,800, Gemma-12B 2,200, Qwen-27B 1,160. DiffusionGemma decodes at 407 tok/s but its second turn costs 2.0 s (canvas re-encode) and its max_tokens behaves as a generation target — 2,048 gave 8/8 ×5 at 483 tokens, 4,096 gave 8,4,8,0,0 at 868 tokens with the same prompt — a property of block diffusion that a token cap designed for autoregressive models does not anticipate.
The four candidate-retire models scored 1.2–3.2/8 on the function task under their best measured recipe, at 44–170 tok/s. Three of them return a correct whole-file edit 7/7 ×5. Whether that is worth a swap slot is an operational decision; the numbers to make it with are now in the specs.
Four of the eight freeze runs reported FAIL. None was a model regression; each was a probe doing its job on the spec's own claims: Qwen's short probe returned reasoning-only under effort=low at an 8-token budget (probe fixed: 512 tokens when the recipe thinks); Gemma's thinking probe found no reasoning tokens under a recipe labelled "low" (the template finding); DiffusionGemma's sampling probe found temperature, top_p, presence_penalty, seed and min_p all accepted where the spec, copied from the previous results file, said rejected (spec corrected, orchestrator profile and its pinned test updated); and the Qwen3-style models' anthropic probe returned only a thinking block because /v1/messages cannot switch thinking off (frozen as the recorded state). A regression suite that only ever passes has not been pointed at anything.
temperature 0.2 with N = 5 the resolution is about one test case; on models with bimodal outcomes the standard error is larger and the tolerance widens accordingly. Ranking by mean first means 8, 8, 8, 6, 0 beats 7, 7, 7, 7, 7 unless the second is 25 % cheaper; the per-sample values are reported so a reader can rank differently.anthropic probe cannot switch thinking off on /v1/messages; Qwen-family models may spend the budget reasoning, so the observed value is frozen rather than asserted./tokenize.Everything is in the llm-stack repository: config/llama-swap.yaml, specs/, bin/, docs/, and every run record under logs/state/<model>/ (replies are kept locally, not committed). The exact prompts, hidden tests and payloads are in docs/prompts.md with content hashes that the per-model documents and run records repeat.
bin/modeltest.py <model> # regression against the frozen spec
bin/modeltest.py <model> --freeze # re-freeze after an intentional change
bin/refine.py <model> --apply # re-pick the recipe by measurement
bin/state-all.sh # refine + freeze, every model, in order
bin/state_report.py # STATE.md
bin/model_docs.py # docs/<model>.md and docs/prompts.md
bin/profiles-from-specs.py --write # orchestrator shipped profilesCommit hashes for the sweep reported here: llm-stack ebfaf45 (sweep outputs), b415ee5 (fix-up applied); ai-pitlane e55d18c (plan 0007, regression graph, shipped profiles). The paper itself is docs/paper.md in the same repository.
The measurement this floor was built for: pad the same tasks with real source files at 4/8/16/32/48K tokens, with the zero-pad cell taken from the same run, N = 5, a criterion relative to each model's own frozen mean, and the top size set by max_model_len − max_tokens − margin; and the orchestrator-side arm — one real work order with 8/16/32 KB of ground truth, scored on whether the whole-file replacement applies and its tests pass. Also: isolating Ornith's greedy nondeterminism to a kernel or flag now that MTP is excluded, recipe selection on the edit task, KV-cache dtype (fp8) as a way to widen windows without changing the checkpoint, and a test in the orchestrator that its shipped profiles equal what the specs imply.
All entries verified against their abstract pages on 2026-08-31.