Three vLLM models on one RTX 3090, measured
Qwen3.8-27B, Ornith-1.5-35B-A3B (Tiel's weights) and DiffusionGemma-26B-A4B, each booted through llama-swap on this workstation, timed with the same harness, and judged by whether the code they returned actually runs.
Scoreboard
Qwen3.8-27B
PASSOrnith-1.5-35B-A3B
PASSDiffusionGemma-26B-A4B
PASS with workaroundsVerdict
Qwen3.8-27B is the coder. With thinking off and a 4 K budget, or thinking on with reasoning_effort=low, it was the only model to pass every case — and it did so on the first sample. It is the slowest to prefill (1,180 tok/s) and the slowest to boot, but the prefix cache makes multi-turn work fine.
Ornith-1.5-35B is the fast lane: 200 tok/s, 6,800 tok/s prefill, 0.3 s cached turns. On this quote-heavy prompt its code was inconsistent (4/8, 6/8, one that does not compile). Good for review passes, bulk edits, and anything where speed matters more than the last 20% of correctness.
DiffusionGemma-26B is an experiment that works: 300–420 tok/s of real code at 6/8, but it needs the Triton attention backend, rejects sampling parameters, and gets little from the prefix cache. Keep it for drafts and for watching where diffusion LMs go.
Small round: none of the 4–9 B models beat the big three on this prompt. Gemma-4-12B (official QAT int4) is the one worth keeping — 6/8 twice, compact, tools and thinking behave. The bf16 9 B models are bandwidth-bound at 44 tok/s (slower than the 27 B W4A16 with a drafter) and Ornith-9B, despite its agent-harness benchmarks, scored 0/8 single-shot. The 4 B class loops and mis-quotes; quantizing it makes it 2.3× faster, not more correct.
Two rules for all of them on this card: never leave Qwen-family thinking at default effort in an agent loop (both Qwen and Ornith spent 8 K tokens deliberating over a 30-line parser and never answered), and give code replies at least 4 K tokens — three "failures" in the first pass were my own truncation.
Speed
| model | decode, short (tok/s) | decode at 6 K ctx | prefill (tok/s) | cached 2nd turn TTFT | tool call | Anthropic API |
|---|---|---|---|---|---|---|
| Qwen3.8-27B | 157–198 | 159.3 | 1,180 | 0.77 s | OK (qwen3_coder parser) | OK |
| Ornith-1.5-35B-A3B | 204.6 | 138.0 | 6,775 | 0.31 s | OK | OK |
| DiffusionGemma-26B-A4B | 423.3 | 382.4 | 2,003 | 1.87 s | OK (gemma4 parser) | OK |
Ornith and Qwen are autoregressive with speculative decoding, so tok/s moves with draft acceptance and with content (code accepts better than prose). DiffusionGemma denoises a 256-token canvas at a time; its 300–420 tok/s is real, but a 12-token answer still costs a full canvas, and its prefix cache does little because the canvas is re-encoded each block. Thinking-on runs report tok/s over reasoning tokens.
Quality of returned work
Same prompt to every model: write parse_door_size(s) -> (width_in, height_in) for forms like 16x7, 16' x 7', 16 ft by 7 ft, 192" x 84", plus eight pytest cases. The reply is extracted and executed twice — against the model's own tests, then against an external set. A tool-call check (correct function and argument) and an Anthropic-format request round it out.
Qwen3.8-27B PASS
returned code, first 42 lines
```python
import re
from typing import Optional
def parse_door_size(s: str) -> tuple[int, int]:
"""
Parse garage-door size strings into (width_in, height_in) in inches.
Supported formats include:
- "16x7" (feet)
- "16' x 7'" (feet)
- "16 ft by 7 ft" (feet)
- '192" x 84"' (inches)
- Mixed forms like "10'6\" x 7'0\"" (feet and inches)
Returns:
Tuple of (width_in_inches, height_in_inches)
"""
s = s.strip().lower()
# Try to parse using a general regex that captures numbers and units
# We'll look for patterns like: number(unit) [separator] number(unit)
# First, try to detect if the primary unit is inches (no feet, just inches)
# or feet (default)
# Let's parse tokens: find all number+unit pairs
# Units: ft, ', foot, feet, in, ", inch, inches
# Separators: x, by, *
# Remove common separators and normalize
# Let's use regex to find two dimension components
# Pattern for a single dimension:
# Can be: number(ft/in) or number' number" (mixed) or just number (assume ft unless specified as inch)
# Let's split on 'x', 'by', '*' etc.
# Replace separators with a single delimiter
s_norm = re.sub(r'\s*(x|by|\*)\s*', '|', s)
parts = s_norm.split('|')
… (118 more lines)returned code, first 42 lines
We need answer user's request: "Write a Python function `parse_door_size(s: str) -> tuple[int, int]` that parses garage-door sizes like "16x7", "16' x 7'", "16 ft by 7 ft", "192\" x 84\"" into (width_in, height_in) in inches. Handle feet, inches, and mixed forms. Include 8 pytest cases. Reply with code only."
Need provide code only. Need think details. Function parse string into inches. Need support feet, inches, mixed forms. Examples: "16x7" likely feet? Garage-door sizes: "16x7" means 16 ft x 7 ft? Usually width and height in feet. "16' x 7'" feet. "16 ft by 7 ft". "192\" x 84\"" inches. Handle mixed forms e.g., "16' 6\" x 7' 2\""? Mixed forms could be "16' 6 x 7' 2"? Need parse both width and height as either integer/inches or feet or feet+inches, with units optional. Could implement robust regex.
Need include 8 pytest cases. Reply code only, likely full code with function and tests. Must be valid Python. Need decide behavior for ambiguous no unit: "16x7" likely feet if both numbers < some threshold? Garage door sizes "16x7" are feet. "192x84" likely inches. We need handle feet, inches, and mixed forms. Without units maybe infer: if value >= 24? inches? But 16x7 likely feet => 192,84. If no unit and numbers small (<12 maybe inches? But height 7 could be feet). How infer? Could treat unitless as feet if values seem like feet (<=12? but width 16 >12). Garage doors: width 7-24 ft, height 6-14 ft. Inches can be 84-192. So no units could be feet if both values < 15? 16 >15? Hmm 16x7: 16 ft, 7 ft. Inches 192x84. Need infer no-unit: maybe if any number >= 24, inches; if both <= 12, feet? But 16 is >12 but likely feet. Could use threshold: if both numbers < 30? feet? But 24 ft doors? 24 inches? Mixed? Need robust for typical garage door.
Spec: "parses garage-door sizes like "16x7", "16' x 7'", "16 ft by 7 ft", "192\" x 84\"" into (width_in, height_in) in inches. Handle feet, inches, and mixed forms." No explicit ambiguous no unit inference. Could require units for inches; unitless default to feet? But "192 x 84" perhaps should be inches? They only show "192\" x 84\"" with quotes, not unitless. If unitless default feet, "192x84" -> 2304,1008, not inches. But might be desired if no units: numbers large => inches. We can infer.
Need produce only code. Need include pytest cases. Could define parse_door_size(s): normalize, replace commas? Use regex to find two measurements separated by x, X, by, space. Each measurement can be e.g.:
- "16"
- "16'"
- "16 ft"
- "16 feet"
- "16'"
- "16'6" (16 feet 6 inches) maybe no inch mark?
- "16' 6\""
- "16 ft 6 in"
- "16 ft. 6 in."
- "16'6\""
- "16' 6''" (double prime for inches)
- "16"
- "192\""
- "192 in"
- "192 inches"
- "192"
Need return tuple[int,int]. Probably if missing units infer. Mixed forms: one part feet, another inches? e.g. "16 ft x 84 in". Or combined feet+inches for each dimension.
Need design regex. Simpler: split string on separators: x, X, by, ' x ', etc. But if measurement includes space like "16 ft 6 in" contains spaces. Need split into two tokens at separator. Use regex split: `re.split(r'\s*(?:x|by|×)\s*', normalized)`? But "16' x 7'" ok. "16 ft by 7 ft" ok. If string has no x but just space? "16' 7'"? Could handle if not split. But separator may be `x` or `by` or `×` or comma? We'll support x/X/×/by.
Normalize: lower? Need preserve quotes. Replace unicode quotes? Could map: `s = s.replace('×', ' x ').replace('by', ' x ')`? But unit "by" only separator. If measurement includes "feet" maybe no. Better split with regex that captures separator including `by`. Need be careful "16 ft by 7 ft": split at ` by ` => ["16 ft", "7 ft"].
For each measurement, parse to inches.
Possible approach: parse each dimension with regex:
- If contains feet marker and inch marker: pattern `(?P<ft>\d+(?:\.\d+)?)\s*(?:'|ft(?:s?)?\.?)\s*(?P<in>\d+(?:\.\d+)?)\s*(?:"|in(?:ches?)?\.?)`? Need handle `16' 6"`, `16 ft 6 in`, `16'6"`, `16 ft 6 in.`, maybe `16'6`.
- If contains feet marker only: `(?P<ft>\d+(?:\.\d+)?)\s*(?:'|ft(?:s?)?\.?)`
- If contains inches marker only: `(?P<in>\d+(?:\.\d+)?)\s*(?:"|in(?:ches?)?\.?)`
- If unitless: infer.
But mixed forms could be "16 ft x 84 in". parse each dimension individually.
… (247 more lines)returned code, first 42 lines
```python
import re
def parse_door_size(s: str) -> tuple[int, int]:
"""Parse garage-door size strings like '16x7', '16' x 7'', '16 ft by 7 ft',
'192" x 84"' into (width_inches, height_inches)."""
s = s.strip().lower()
# Normalize separators: "by" -> "x", strip extra spaces around x
s = re.sub(r"\bby\b", "x", s)
s = re.sub(r"\s+x\s+", "x", s)
parts = s.split("x")
if len(parts) != 2:
raise ValueError(f"Cannot parse door size: {s!r}")
def _parse_part(part: str) -> int:
part = part.strip()
is_inches = (
'"' in part
or " in" in part
or " inch" in part
or " inches" in part
)
m = re.search(r"[\d]+(?:\.\d+)?", part)
if not m:
raise ValueError(f"No number found in: {part!r}")
num = float(m.group())
if is_inches:
return int(num)
# bare numbers and feet default to feet
return int(num * 12)
return (_parse_part(parts[0]), _parse_part(parts[1]))
… (37 more lines)- Best code of the three when given room: thinking off with a 4 K budget scored 8/8 on the first sample; thinking on with
reasoning_effort=lowscored 8/8 own and 8/8 external in 2,243 tokens. - Default thinking effort never answered: 8,192 tokens of reasoning, no final code — same failure as Ornith. Set
chat_template_kwargs.reasoning_effort="low"for agent loops. - One of three no-think samples does not compile (
\'|'quoting inside a raw regex) — the quote-heavy prompt trips both Qwen-family models about one time in three. - Prefill 1,180 tok/s is the price of a dense 27 B; Ornith prefills 5.7× faster. Prefix cache (0.77 s) makes multi-turn fine.
- Prep (download + requant) ran ~35 min on CPU; first boot compiles for ~4 min and caches to the
qwen-cachevolume.
Ornith-1.5-35B-A3B PASS
returned code, first 42 lines
```python
import re
import pytest
def parse_door_size(s: str) -> tuple[int, int]:
"""Parse a garage-door size into a (width_in, height_in) tuple (in inches).
Accepts forms such as:
"16x7", "16' x 7'", "16 ft by 7 ft", "192\" x 84\""
Feet (' / ft) and inches (\" / in) are recognized and mixed forms are
allowed, e.g. "4' 6\" x 8'".
"""
tokens = re.findall(r"[+-]?\d+(?:\.\d+)?\s*(?:'|"'"'"'"|ft|in|inches|inche)?", s)
tokens = [t for t in tokens if t.strip()]
if len(tokens) < 2:
raise ValueError(f"Could not parse dimensions from {s!r}")
def to_inches(token: str) -> float:
token = token.strip().lower()
if token.endswith(("ft", "feet")):
return float(re.sub(r"(ft|feet)?", "", token)) * 12.0
if token.endswith(("in", "inches", "inche")):
return float(re.sub(r"(in|inches|inche)?", "", token))
# No unit: default to inches if it's a bare number, but allow bare feet
# only when a trailing ' or the whole value already looked like feet.
return float(token)
w = to_inches(tokens[0])
h = to_inches(tokens[1])
return round(w), round(h)
@pytest.mark.parametrize(
"s, expected",
[
("16x7", (16, 7)),
("16' x 7'", (16, 7)),
("16 ft by 7 ft", (16, 7)),
('192" x 84"', (192, 84)),
('4\' 6" x 8\'', (54, 96)),
("4 ft 6 in by 8 ft", (54, 96)),
… (7 more lines)returned code, first 42 lines
The user wants a Python function `parse_door_size(s: str) -> tuple[int, int]` that parses garage-door sizes in various formats into (width_in, height_in) in inches.
Formats to handle:
- "16x7"
- "16' x 7'" (feet with apostrophe)
- "16 ft by 7 ft"
- "192" x 84"" (inches with double-quote)
Need to handle feet, inches, and mixed forms.
Let me write a robust parser using regex.
The idea:
- Extract two numbers
- Determine their units (feet or inches)
- Convert everything to inches
Let me think about the units. Feet can be denoted by `'` (apostrophe) or `ft`. Inches can be denoted by `"` (double quote) or `in`.
If no unit is specified (like "16x7"), they're already in inches (standard garage door notation where 16x7 means 16 feet x 7 feet? Wait, no).
Actually, garage door sizes like "16x7" typically mean 16 feet wide x 7 feet tall. But the question says "into (width_in, height_in) in inches." Hmm.
Wait, let me reconsider. "16x7" - is that feet or inches? In garage door context, "16x7" means 16 feet wide by 7 feet tall. But the function returns inches.
Hmm, but then "16' x 7'" would be 16 feet x 7 feet = 192" x 84". And "192" x 84"" would be 192 inches x 84 inches.
So these should be consistent! "16' x 7'" = 192 x 84, and "192" x 84"" = 192 x 84. Yes!
So "16x7" — if interpreted as feet — would be 192 x 84. That makes sense with the garage door convention.
Wait, but that's ambiguous. Let me think again. The examples:
- "16x7"
- "16' x 7'"
- "16 ft by 7 ft"
- "192" x 84""
If "16' x 7'" = 16 feet x 7 feet = 192 x 84 inches.
And "192" x 84"" = 192 inches x 84 inches.
So both give (192, 84). That means "16x7" is also feet, giving (192, 84).
… (499 more lines)- Thinking mode never finished the coding task: 4,096 then 8,192 tokens of reasoning (25 KB) with no final answer. Use thinking off for code, or budget ≥16 K tokens.
- Code quality on the quote-heavy parser prompt was inconsistent across 3 no-think samples: 4/8, 6/8, and one reply that does not compile (unterminated string inside a raw regex). Bare-feet forms (
16x7) were the common miss. - Marlin linear + Marlin WNA16 MoE kernels (Ampere path).
- Thinking on by default; reasoning lands in a separate field. Budget max_tokens for it or pass enable_thinking=false.
- Asymmetric int4 checkpoints (cyankiwi, ulkaa) are rejected by vLLM's MoE kernels; MIRALABS' symmetric one is 25.6 GB — too big. biMEMO's was the only single-card fit.
DiffusionGemma-26B-A4B PASS with workarounds
returned code, first 42 lines
```
import re
import pytest
def parse_door_size(s: str) -> tuple[int, int]:
"""
Parses garage door strings into (width_inches, height_inches).
Handles formats: 16x7, 16' x 7', 16 ft by 7 ft, 192" x 84"
"""
# Normalize string: remove whitespace and lowercase
s = s.lower().strip()
# Split by common delimiters: 'x', 'by', or just whitespace
parts = re.split(r'x|by', s)
if len(parts) != 2:
raise ValueError(f"Could not parse door dimension string: {s}")
def to_inches(part: str) -> int:
part = part.strip()
# Extract numbers
match = re.search(r'(\d+)', part)
if not match:
raise ValueError(f"Could not find dimension in: {part}")
value = int(match.group(1))
# Check for units
if "ft" in part or "'" in part:
return value * 12
elif '"' in part or "in" in part:
return value
else:
# Default to inches if no unit provided
return value
width = to_inches(parts[0])
height = to_inches(parts[1])
return (width, height)
@pytest.mark.parametrize("inp, expected", [
("16x7", (192, 84)),
("16' x 7'", (192, 84)),
… (13 more lines)returned code, first 42 lines
python
import re
import pytest
def parse_door_size(s: str) -> tuple[int, int]:
"""
Parses garage door strings into (width_inches, height_inches).
Handles formats: 16x7, 16' x 7', 16 ft by 7 ft, 192" x 84"
"""
# Normalize the string
s = s.lower().strip()
# Split by common delimiters: 'x', 'by', or whitespace
parts = re.split(r'\s*[xbby]\s+', s)
if len(parts) != 2:
raise ValueError(f"Could not parse string: {s}")
def to_inches(dim: str) -> int:
dim = dim.strip()
# Extract numeric part
match = re.search(r'(\d+)', dim)
if not match:
raise ValueError(f"Could not find dimension in: {dim}")
value = int(match.group(1))
# Check for units
if "'" in dim or "ft" in dim:
return value * 12
elif '"' in dim or "in" in dim:
return value
# Default fallback to inches if no unit provided
return value
return to_inches(parts[0]), to_inches(parts[1])
@pytest.mark.parametrize("input,expected", [
("16x7", (192, 84)),
("16' x 7'", (192, 84)),
("16 ft by 7 ft", (192, 84)),
("192\" x 84\"", (192, 84)),
("8'x8'", (96, 96)),
… (10 more lines)- Default FlashInfer backend crashes at CUDA-graph capture: plan() gets a per-request causal tensor where it wants a bool. FLASH_ATTN refuses head_dim 256. TRITON_ATTN works.
- temperature / seed / min_p are rejected (HTTP 400) for diffusion models in 0.28.0.
- "Reply with exactly: ok" returns an immediate EOS (1 token) in both thinking modes; ordinary prompts are fine.
- Prefix cache barely helps (1.87 s second turn) — canvas re-encoding dominates.
Fit on the card
| model | checkpoint | weight load | resident | KV budget | VRAM serving | cold start |
|---|---|---|---|---|---|---|
| Qwen3.8-27B | syv-ai requant of Qwen/Qwen3.8-27B (AutoRound W4A16 body + GPTQ-int4 lm_head/MTP) + DFlash2 W4A16 drafter | 6.7 s (+0.3 s drafter) | 15.02 GiB | pinned 5.58 GB → 68,605 tok (1.05× at 64 K) | 22,897 MiB | ~4.5 min first boot (compile), ~1.5 min warm |
| Ornith-1.5-35B-A3B | biMEMO/Ornith-1.5-35B-A3B-int4-AutoRound-MTP (20.9 GB; the same weights Tiel-Coder is quantized from) | 28.9 s + 3.7 s MTP | 18.77 GiB | 2.09 GiB → 70,870 tok (2.16× at 32 K) | 22,229 MiB | ~3 min |
| DiffusionGemma-26B-A4B | cyankiwi/diffusiongemma-26B-A4B-it-AWQ-INT4 (17.2 GB) | 17–21 s | 15.58 GiB | 5.04 GiB → 102,853 tok (3.14× at 32 K) | 23,535 MiB | ~2.5 min |
The stack
One endpoint, three containers
llama-swapon127.0.0.1:11440speaks OpenAI and Anthropic; model idsqwen3.8-27b,ornith-1.5-35b,diffusiongemma-26b(aliasesqwen,tiel,dgemma).- Each model is a
docker runpinned to the 3090 by UUID; one resident at a time;docker stopon swap. gemma-4-26b-a4bis aliased to Qwen so ai-pitlane'sreviewer/local_aiandplan_deepwork unchanged.
ai-pitlane playground
- Project
llm-stack-bakeoff: probe → bench Qwen → bench Ornith → bench DiffusionGemma → summary, allpython_scriptnodes, $0 cap, manual launch. Writesai-queue/llm-stack/bakeoff.md. - Design doc
/design/llm-stack-3090carries the architecture diagram. - venv rebuilt; 1,662 tests pass.
Small-model round
Same harness, same prompt, same scoring — models in the 4–12 B class on the 3090, in bf16 where they fit and with an official or well-formed int4 where they don't, plus a 4-bit copy of the 4 B to measure what quantization costs.
Ornith-1.5-9B
runs, but 0/8 on codeQwen3.8-9B-Distill
runs; 4/8 bestQwen3.5-4B
runs, but unusable for codeGemma-4-12B-it
best of the small round · 6/8Qwen3.5-4B AWQ-INT4
runs, but unusable for code| model | decode, short (tok/s) | decode at 6 K ctx | prefill (tok/s) | cached 2nd turn TTFT | tool call | Anthropic API |
|---|---|---|---|---|---|---|
| Ornith-1.5-9B | 44.4 | 43.8 | 3,670 | 0.15 s | OK | FAIL (thinking ate the budget) |
| Qwen3.8-9B-Distill | 44.8 | 44.2 | 3,665 | 0.15 s | OK | FAIL (thinking ate the budget) |
| Qwen3.5-4B | 76.4 | 75.0 | 6,369 | 0.09 s | OK | FAIL (400-token budget consumed by thinking) |
| Gemma-4-12B-it | 78.7 | 71.0 | 2,280 | 0.09 s | OK (gemma4 parser) | OK |
| Qwen3.5-4B AWQ-INT4 | 172–175 | 166.3 | 6,790 | 0.09 s | OK | FAIL (thinking ate the budget) |
| model | checkpoint | weight load | resident | KV budget | VRAM serving | cold start |
|---|---|---|---|---|---|---|
| Ornith-1.5-9B | ornith-ai/Ornith-1.5-9B bf16 — Qwen3.5-9B base + coding/agentic RL; card: Terminal-Bench 46.2, SWE-bench Verified 70.6 | ~15 s | ≈18 GiB (bf16) | not captured | — | ~2 min |
| Qwen3.8-9B-Distill | empero-ai/Qwen3.8-9B-Distill — full-parameter distillation of Qwen3.8-2.4T into the Qwen3.5-9B architecture (~70 K teacher traces) | ~15 s | ≈18 GiB (bf16) | not captured | — | ~2 min |
| Qwen3.5-4B | Qwen/Qwen3.5-4B bf16 — the official small baseline | ~10 s | ≈9 GiB (bf16) | not captured (container swapped out before the log was read) | — | ~2 min |
| Gemma-4-12B-it | google/gemma-4-12B-it-qat-w4a16-ct — Google's own quantization-aware int4, compressed-tensors | 5.0 s | 8.28 GiB | 288,717 tokens (4.4× at 64 K) | 21,003 MiB | ~2 min |
| Qwen3.5-4B AWQ-INT4 | cyankiwi/Qwen3.5-4B-AWQ-4bit — same model as above at 4-bit, to measure what quantization costs | ~5 s | ≈3.5 GiB | not captured | — | ~2 min |
Ornith-1.5-9B runs, but 0/8 on code
returned code, first 42 lines
```python
import re
def parse_door_size(s: str) -> tuple[int, int]:
s = s.strip()
# Format: "16 ft by 7 ft" or "16 feet and 8 inches"
ft_parts = re.findall(r'(\d+)\s*feet?', s, re.IGNORECASE)
if ft_parts:
ft = int(ft_parts[0])
st = ft * 12
inch_parts = re.findall(r'(\d+)\s*inch(?:es)?', s, re.IGNORECASE)
if inch_parts:
st += int(inch_parts[0])
return (ft * 12, st)
# Format: '16\' x 7\'', "16\" x 84\"", 16x7, "16' 8\" x 7'"
m = re.search(r"(\d{1,3})('?\"?| x | by )(\d{1,3})('?\"?)", s)
if m:
w = int(m.group(1))
h = int(m.group(2))
inches = (w if m.group(1) == w else 0)
inches2 = (h if m.group(1) == h else 0)
width_in = inches + inches2
return (width_in, height_in)
```
I broke my thinking here since I didn't finish writing the code. Let me rebuild this properly:
```python
import re
def parse_door_size(s: str) -> tuple[int, int]:
s = s.strip()
# Format: "16 ft by 7 ft", "16 feet and 8 inches", "16' x 7'", "16\" x 84\"", "16x7", "16' 8\" x 7'"
ft_parts = re.findall(r'(\d+)\s*feet?', s, re.IGNORECASE)
if ft_parts:
ft = int(ft_parts[0])
st = ft * 12
inch_parts = re.findall(r'(\d+)\s*inch(?:es)?', s, re.IGNORECASE)
if inch_parts:
st += int(inch_parts[0])
… (189 more lines)returned code, first 42 lines
The user wants a Python function that parses garage-door sizes into (width_in, height_in). Let me think about the requirements:
1. Parse strings like:
- "16x7" → 16 inches x 7 inches
- "16' x 7'" → 16 ft x 7 ft = 192x84
- "16 ft by 7 ft" → 16 ft by 7 ft = 192x84
- "192\" x 84\"" → 192 inches x 84 inches
- Handle feet, inches, and mixed forms
2. Return a tuple of (width_in, height_in) in inches.
3. Include 8 pytest cases.
Let me think about the parsing logic:
- Numbers can be in feet or inches.
- Feet indicated by `'` or `ft` or `feet` or `feet "` etc.
- Inches indicated by `"` or `in` or `inch` or `inches`.
So I need to extract numbers and their units, then convert feet to inches (multiply by 12), and handle the width/height order.
Let me write a robust regex-based parser.
Approach:
1. Find all number-unit pairs.
2. For each, determine if it's feet or inches.
3. Convert to inches.
4. First number is width, second is height.
Let me think about what counts as feet vs inches:
- `'` (apostrophe) → feet
- `"`, `\"` → inches
- `ft`/`feet`/`foot` → feet
- `in`/`inch`/`inches` → inches
But careful: `x` unit (like "cm in x ft")? Probably not needed. But "16 ft by 7 ft" uses "by". Let me handle general.
Let me write the parser:
```python
import re
… (509 more lines)- 44 tok/s: 19.3 GB of bf16 weights and no drafter, so decode is pure bandwidth — slower than the 27 B W4A16 + DFlash2 build and the 35 B MoE. An int4 build would roughly triple it, but every published int4 of this model carries bf16 vision/embedding tensors and is 8.6–18 GB.
- With thinking off it still reasons in the open: sample 1 was 11 code fragments separated by "Let me write the actual clean version now…", cut off at 4,096 tokens.
reasoning_effort=lowran to 8,192 tokens with no answer. - Its cleanest sample (878 tokens, 10 own tests) raises
IndexError: no such groupon every input — a regex group-index bug. 0/8 on all four samples. - The card's SWE-bench 70.6 is earned inside an agent harness with tools and iteration; single-shot code generation on a quote-heavy spec is a different task, and here it did not deliver.
Qwen3.8-9B-Distill runs; 4/8 best
returned code, first 42 lines
```python
import re
from typing import Tuple
def parse_door_size(s: str) -> Tuple[int, int]:
"""
Parse garage-door size strings into a tuple of (width_in, height_in) in inches.
Supported formats:
- "16x7"
- "16' x 7'"
- "16 ft by 7 ft"
- "192\" x 84\""
- "16 ft by 7 in"
- "16' x 7\""
- "2'4\" x 9'6\""
- "36 in x 84 in"
Args:
s: Input string representing a door size.
Returns:
A tuple (width_in, height_in) where both values are integers in inches.
Raises:
ValueError: If the input string cannot be parsed.
"""
s = s.strip()
# Pattern for: A_unit x/unit Feet/ft by Feet/ft/inches Inch/in or similar
# Capture two components separated by 'x' or 'by' with unit labels:
match = re.match(
r"^\s*"
r"(\d+(?:'\d+)?\s*(?:ft?|inches?)?|\"\d+?(?:/?\"?\s*(?:ft?|inches?)?)?)\s*"
r"(?:x|X|x?|by?)"
r"\s*"
r"(\d+(?:'\d+)?\s*(?:ft?|inches?)?|\"\d+?(?:/?\"?\s*(?:ft?|inches?)?)?)\s*$",
s,
re.IGNORECASE,
)
if not match:
… (60 more lines)returned code, first 42 lines
Thinking Process:
1. **Analyze the Request:**
* Function name: `parse_door_size`
* Input: `s: str`
* Output: `tuple[int, int]` representing (width_in, height_in) in inches.
* Input formats to handle: "16x7", "16' x 7'", "16 ft by 7 ft", "192\" x 84\"".
* Logic: Convert feet and inches to total inches, parse dimensions.
* Constraints: Must include 8 `pytest` test cases.
* Output format: Code only.
2. **Determine Parsing Logic:**
* Need to strip units (ft, ft in, ", ").
* Need to split dimensions (x, by).
* Need to handle mixed units (e.g., "5'6"" or "5 ft 6 in" or "16' 7\"" or "16 ft 8 in"). The problem description gives examples like "16' x 7'" and "192\" x 84\"", implying the whole dimension might be in inches or feet. It also says "Handle feet, inches, and mixed forms".
* Let's look closer at the examples:
* "16x7" -> 16, 7
* "16' x 7'" -> 16*12 + 0, 7*12 + 0 (assuming no sub-inches) OR 16 inches width, 7 inches height. Wait, usually door sizes are like 16' x 8'. So 16 feet is 192 inches.
* "16 ft by 7 ft" -> 16*12, 7*12.
* "192\" x 84\"" -> 192, 84.
* "Mixed forms" implies inputs like "5'6" or "5 ft 6 in".
* Regex approach is best here.
* Split string by separators ('x', ' by ', ' ' ) to get two parts.
* For each part, parse numbers and units.
* Units: 'ft', "'", '"', 'in', 'inch' (maybe).
* If 'ft' unit present: multiply value by 12.
* If '"' unit present: treat as inches.
* If 'in' unit present: treat as inches.
* If just number: treat as inches (common default or assume feet? In "192\" x 84\"", it's inches. In "16x7", could be feet or inches? Usually stripped units are ambiguous. However, for the function signature `parse_door_size`, returning a specific int means making a decision. Let's assume for bare numbers, if the other has 'ft' maybe all are feet? No, "16x7" in door context likely means feet if it's a garage door. But often "16x7" just means inches in casing context. Let's look at "192\" x 84"". "16 x 7" (without quotes) -> if it's feet, (192, 84). If inches, (16, 7).
* Actually, a robust parser for "Width" "Height" usually looks for the trailing unit.
* Let's refine the parsing logic per component:
1. Replace slashes, quotes with delimiters.
2. Split by 'x', ' by ', ' ' (careful with spaces).
3. Strip unit suffixes from each numeric part.
4. If unit is 'ft' -> *12.
5. If unit is 'in' or '"' -> result as is.
6. If no unit -> assume inches? Or feet? The prompt examples "16x7" -> likely 16 inches, 7 inches? Or 16 feet, 7 feet? Garage door standard 16' x 7' is possible. But "16x7" usually implies inches in DIY unless 'x' implies feet (like U.S. drywall is usually feet width).
7. Actually, "Handle feet, inches, and mixed forms" suggests if I write "16' x 7'", 16 is feet. If I write "16" x 84"", 16 is inches.
8. So the rule is:
* Nominal Number: > 1 -> Feet (Garage doors are usually >100 inches). But "16x7" might be small shed door.
* Better approach: If `ft` found, convert. If `"` found, it's inches. If bare number, assume inches (since "192\" x 84\"" are inches). Wait, if input is "16x7", output (16, 7).
* Wait, let's look at "16 ft by 7 ft".
… (335 more lines)- Same 44 tok/s bandwidth ceiling as Ornith-9B (identical architecture, bf16, no drafter).
- The best-behaved small model: three compact no-think replies (0.9–1.3 K tokens) with no restart loops. But only one compiles — 4/8, getting the bare-feet forms right and the inch-quoted ones wrong — and it wrote no tests of its own; the other two fail on quote escaping (unterminated string; invalid syntax).
reasoning_effort=lowis not honored (8,192 tokens, no answer) — the distill did not inherit Qwen3.8's effort control.- Usable for review/summarization at 9 B; not reliable enough for generation on this kind of spec.
Qwen3.5-4B runs, but unusable for code
returned code, first 42 lines
```python
import re
def parse_door_size(s: str) -> tuple[int, int]:
"""
Parses garage-door sizes like "16x7", "16' x 7'", "16 ft by 7 ft", "192\" x 84\""
into (width_in, height_in) in inches.
Handles:
- Numeric only (assumed inches)
- Numeric + feet (ft, ', space-separated scalar)
- Numeric + inches (numeric or "). In mixed cases, last unit implies height unit.
- Space, ' x ', or " by " as separators.
"""
s = s.strip()
# Split by common separators: x, ' x ', " by ", spaces/generic separators
# Try to split by ' x ' or " x " or " by " first
split_by_x = re.split(r'[\s]*\s*x\s*[\s"']*|[\s]*\s*by\s*[\s"]*', s, flags=re.IGNORECASE)
if len(split_by_x) != 2:
# If no split by 'x' or 'by', assume single string; split on spaces between numbers if needed
# For simplicity, assume single or exactly space-separated parts; try to split by space
# If numbers contain ', or quotes, strip them.
parts = [p.strip() for p in s.split()]
else:
parts = split_by_x
def parse_number許 unit_str:
unit_str = unit_str.replace('"', '').strip()
# Handle ' ft ', ' ft', ' feet ', ' feet', 'ft ', ' ft'
unit = unit_str.lower()
if unit == 'ft' or unit == 'feet':
if unit_str.isdigit():
return int(unit_str), 'ft'
# If there's a decimal point, we consider inches if the digit is a single-char diff.
# But typical usage is whole number.
# Now treat as given number with inches or ft.
# We support both: "16" (16 inches), "16'" or "16 ft", "16 ft 6" (mixed).
# We'll convert to a float then to integer at end, handling the last part as inches if possible
# If the unit is absent, assume inches.
… (359 more lines)returned code, first 42 lines
Thinking Process:
1. **Analyze the Request:**
* Function name: `parse_door_size`
* Input: string `s` (example: "16x7", "16' x 7'", "16 ft by 7 ft", "192\" x 84\"")
* Output: tuple of two integers `(width_in, height_in)`
* Requirements: Handle feet (`'`, `ft`, `''`?), inches (`"`, `inch`, no suffix), mixed forms.
* Constraint: Handle the logic correctly to extract numbers and units.
* Constraint: Include 8 pytest test cases.
* Constraint: Reply with code only.
2. **Determining Conversion Logic:**
* 1 foot = 12 inches.
* Input strings might contain:
* Pure inches (e.g., "192\" x 84\"") -> Note: The example uses backticks in the prompt's example `"` but usually, it's just a character. I need to handle `"` (which is often escaped in prompt strings) literally.
* Pure feet (e.g., "16' x 7'", "16' x 7'")
* Feet abbreviation (e.g., "16 ft by 7 ft")
* Mixed (e.g., "16 ft x 8'")
* Separators: `x`, `x`, `hyphen`, `space`, `by`.
* Numbers: Need to be floats or integers for intermediate calculation, then converted to int for result.
3. **Reference Handling:**
* The prompt example shows `192\"`. This looks like Python string representation. I should assume the input string will contain the literal characters `'` or `ft` or `"` or `x` or ` by ``.
* Wait, the prompt example `192\" x 84\"` suggests the actual input string might contain `"` (double quote) but escaped in the documentation. Or it might contain the character `'`.
* Commonly used units:
* `'` (apostrophe) = foot
* `ft` = foot
* `"` = inch
* No unit = assume inches or feet? Usually, sizes like `16x7` imply just numbers, but context implies units. Let's assume if no unit is specified but numbers are given, they might be ambiguous. However, for robustness, usually, explicit units are preferred. Let's excuse assume no explicit unit is treated as inches (since doors are often expressed as inches, but garage doors are often 7ft high (~84in) or 7ft high (84in). 192 inches is ~16ft wide. 84 inches is ~7ft high. So `192x84` are definitely inches).
* Logic refinement:
1. Extract all numbers from the string.
2. Determine units (`'`, `ft`, `"`, empty).
3. Since the input format is `N1 unit1 by unit2` or `N1 unit1 x N2 unit2`, regex is best.
4. Regex: `r'(\d+(?:\.\d+)?)\s*(?:'|\s*ft\s*)?\s*(?:x|\s*by\s*)?\s*(\d+(?:\.\d+)?)\s*(?:'|\s*ft\s*)?\s*(?:'?|\"|\s*\#)??'?` ... This is getting complicated.
5. Simpler approach: Replace units with a number.
* `ft` -> 12, `'` -> 12.
* `"` -> 1, empty -> 1 (assume inches if no unit found? No, wait).
* If I see `192"`, it's 192 inches.
* If I see `16`, is it 16 inches or 16 feet? In garage doors, width might be 16 (meaning 16ft or 6'4"? 16 is usually a shorthand for 16ft). Height 7 usually means 7ft (84in).
* However, the prompt asks to handle mixed forms.
* Let's look at standard regex patterns for mixed units:
* `^\s*(\d+(?:\.\d+)?)\s*([\'\"]?\salt)?` -> This is tricky.
… (403 more lines)- Never finishes. All three no-think samples hit the 4,096-token cap and the
reasoning_effort=lowrun hit 8,192: the model writes a complete solution, announces "Final clean output below:", and starts again. Its own test expectations were wrong too (16x7 → (16, 7)). - Scored on the first block that compiles: best sample 2/8 external (only the inch-quoted forms); the other three have no compiling block at all.
- Decode is only 76 tok/s despite 4 B parameters: no speculative decoding and bf16 weights, so it is pure bandwidth — the 27 B W4A16 + DFlash2 build is 2× faster.
- Fine for classification/extraction chores; not a coder on this prompt.
Gemma-4-12B-it best of the small round · 6/8
returned code, first 42 lines
```python
import re
def parse_door_size(s: str) -> tuple[int, int]:
# Regex to find numbers and their associated units (ft, ', ", feet)
# Matches: 16, 16', 16", 16 ft, 16 feet
pattern = r"(\d+)\s*(?:'|\"|ft|feet|inch|inches)?"
matches = re.findall(pattern, s.lower())
def convert_to_inches(match_str: str) -> int:
num_part = match_str[0]
unit_part = match_str[1]
val = int(num_part)
if any(u in unit_part for u in ("'", "ft", "feet")):
return val * 12
return val
# Extract all numeric parts converted to inches
sizes = []
for m in matches:
sizes.append(convert_to_inches(m))
if len(sizes) >= 2:
return (sizes[0], sizes[1])
return (0, 0)
if __name__ == "__main__":
import pytest
@pytest.mark.parametrize("input_str, expected", [
("16x7", (16, 7)),
("16' x 7'", (192, 84)),
("16 ft by 7 ft", (192, 84)),
("192\" x 84\"", (192, 84)),
("8x9", (8, 9)),
("10ft x 8ft", (120, 96)),
("12' x 12'", (144, 144)),
("24 inch x 36 inch", (24, 36)),
])
def test_parse_door_size(input_str, expected):
assert parse_door_size(input_str) == expected
… (1 more lines)returned code, first 42 lines
```python
import re
def parse_door_size(s: str) -> tuple[int, int]:
"""
Parses garage door sizes into a tuple of (width_in, height_in).
Supports formats like "16x7", "16' x 7'", "16 ft by 7 ft", "192\" x 84\"".
"""
# Normalize the string: remove quotes and whitespace, convert to lowercase
clean_s = s.replace('"', '').lower().replace("'", "").replace(" ", "")
# Regex to find numbers and the separator (x, by, or just a space/gap)
# We look for parts that represent dimensions
parts = re.findall(r'(\d+)', clean_s)
if len(parts) < 2:
raise ValueError("Could not find two dimensions in input string.")
raw_w = int(parts[0])
raw_h = int(parts[1])
# Determine if units are feet or inches based on keywords
is_feet = "ft" in clean_s or "'" in s
is_inches = '"' in s or "inch" in clean_s
# Logic to decide multiplier:
# If "ft" or "'" is present, treat as feet unless "inch" is also present.
# If no units are provided, we assume inches (standard for door codes like 16x7).
# However, if "ft" is found, multiply by 12.
multiplier = 1
if is_feet and not is_inches:
multiplier = 12
return (raw_w * multiplier, raw_h * multiplier)
if __name__ == "__main__":
import pytest
def test_parse_door_size():
assert parse_door_size("16x7") == (16, 7)
assert parse_door_size("16' x 7'") == (192, 84)
assert parse_door_size("16 ft by 7 ft") == (192, 84)
… (8 more lines)- The small model to keep. Compact replies (480–880 tokens), thinking mode that actually returns (645 tokens at
effort=low), tool calls and the Anthropic route both working, and 6/8 on two of four samples. - Fails are the same two bare-feet forms (
16x7,9x8) as its DiffusionGemma sibling — a family habit of reading unitless numbers as inches. One sample scored 0/8 and one did not compile. - Wrote one pytest case instead of the eight asked for — it under-delivers on the test half of the prompt.
- Marlin int4 at 8.3 GiB resident leaves a 289 K-token KV pool: this is the model to give a whole repository to.
Qwen3.5-4B AWQ-INT4 runs, but unusable for code
returned code, first 42 lines
```python
import re
def parse_door_size(s: str) -> tuple[int, int]:
"""
Parses garage-door sizes like "16x7", "16' x 7'", "16 ft by 7 ft", "192\" x 84\""
into (width_in, height_in) in inches.
Supported formats:
- Plain numbers: "10x10"
- With units: "10' 9\"", "10'09\"", "10 ft 9 in", "10' 9'", "10 ft by 9 ft"
Logic:
- If both parts contain 'ft' or inches without a 'prime', treat as converted fully to inches.
- If primes (') are present, treat the number before the prime as feet.
- If a mix: assume feet as inches * 12 if no inch part.
"""
s = s.strip()
s = s.lower()
# Handle single number (no x or by)
s = re.sub(r'\s*x\s+', ' ', s)
s = re.sub(r'\s+by\s+', ' ', s)
s = s.strip()
# Determine if we have prime feet or just numbers
has_prime = "'" in s
has_decimals_or_inches = '"' in s
# Normalize spaces
s = s.replace("ft", " ").replace("in", " ").replace("ft ", " ").replace("in ", " ")
s = s.replace('ft "', "ft ').replace(' in "', "in ").replace('ft-', ' ft ').replace('in-', ' in ')
parts = s.split('by')
if len(parts) > 1:
parts = [p.strip() for p in parts]
else:
parts = [s, '']
parts = [p.strip() for p in parts]
… (91 more lines)returned code, first 42 lines
```python
import re
def parse_door_size(s: str) -> tuple[int, int]:
s = s.strip()
numbers = []
units = [''] * len(numbers)
i = 0
while i < len(s):
match = re.match(r'(\d+(?:\.\d+)?)', s[i:], 0)
# Find all numbers in sequence
if match:
num = float(match.group(1))
numbers.append(num)
# Determine unit
current = s[i:]
unit_match = re.match(r'(ft|feet|\'|\")', current[l().group(1)), end()
unit_match.group
i += match group.findlen ends with
else:
if numbers[0]:
numbers = [int(round(number)) for number in numbers]
else:
numbers = [0, 0]
# Return as tuple
return (numbers[0], numbers[1])
# 8 pytest cases
def test_parse_door_size_basic():
assert parse_door_size("16x7") == (192, 84)
def test_parse_door_size_feet():
assert parse_door_size("16' x 7'") == (192, 84)
def test_parse_door_size_ft_by():
assert parse_door_size("16 ft by 7 ft") == (192, 84)
… (13 more lines)- 2.3× faster than the bf16 build (175 vs 76 tok/s): with no speculative decoding, decode is pure weight bandwidth and int4 Marlin reads a quarter of the bytes.
- Same restart habit as bf16 — two of three no-think samples looped to the 4,096 cap;
reasoning_effort=lowdid finish (2,749 tokens). - The two replies that finished have real syntax errors (an unterminated string literal; a
)closing a[). 0/8 on every sample. - Conclusion for the 4 B class: quantize it for speed if you use it at all, but this prompt is beyond it.
Controlled re-test: prompt + sampling
Same task, three changes: the unit convention stated in the prompt (bare numbers are feet), a system prompt demanding exactly one fenced block with no revisions and raw single-quoted regexes, and temperature 0.2, top_p 0.95, presence_penalty 1.0 instead of each checkpoint's temperature 1.0 default. DiffusionGemma keeps its default sampling (vLLM 0.28 rejects the parameters for diffusion models). Each model runs in its best baseline mode, three samples.
| model | mode | baseline · external passes (per sample) | won't compile | tuned · external passes (per sample) | won't compile |
|---|---|---|---|---|---|
| Qwen3.8-27B | thinking · effort=low | 16/16 8 · 8 | 0/2 | 24/24 8 · 8 · 8 | 0/3 |
| Ornith-1.5-35B-A3B | thinking off | 18/32 4 · 6 · err · 8 | 1/4 | 8/24 err · 4 · 4 | 1/3 |
| DiffusionGemma-26B-A4B | thinking off | 16/24 6 · 6 · 4 | 0/3 | 8/24 0 · 0 · 8 | 0/3 |
| Ornith-1.5-9B | thinking off | 0/32 0 · err · 0 · 0 | 1/4 | 7/24 5 · 0 · 2 | 0/3 |
| Qwen3.8-9B-Distill | thinking off | 8/32 4 · err · err · 4 | 2/4 | 12/24 2 · 6 · 4 | 0/3 |
| Qwen3.5-4B | thinking off | 4/32 2 · err · err · 2 | 2/4 | 14/24 0 · 8 · 6 | 0/3 |
| Gemma-4-12B-it | thinking off | 6/32 0 · err · 6 · 0 | 1/4 | 22/24 8 · 8 · 6 | 0/3 |
| Qwen3.5-4B AWQ-INT4 | thinking off | 0/32 err · err · 0 · err | 3/4 | 6/24 2 · 0 · 4 | 0/3 |
Playground run (ai-pitlane)
The llm-stack-bakeoff graph launched from the dashboard as a transient systemd unit: probe the endpoint, bench each model in turn (one GPU, so the chain is sequential, with a llama-swap swap between models), then write ai-queue/llm-stack/bakeoff.md.
| run | nodes | wall | outcome |
|---|---|---|---|
| graph-20260830T062950…p461826 | probe ok → bench_qwen ok → bench_ornith ok → bench_dgemma failed → summary failed | 200 s | failed at bench_dgemma — llama-swap was still running the pre-Triton DiffusionGemma command (config edited, process not restarted) |
| graph-20260830T063545…p498911 | probe ok → bench_qwen ok → bench_ornith ok → bench_dgemma ok → summary ok | 273 s | complete — all three models benched through one endpoint with two swaps; artifact ai-queue/llm-stack/bakeoff.md written |
| graph-20260830T134547…p3051319 (small) | probe ok → bench_ornith9b ok → bench_qwen9b ok → bench_qwen4b ok → bench_gemma12b ok → bench_qwen4b_awq ok → summary ok | 536 s | complete — five small models benched sequentially through :11440 with four swaps; artifact ai-queue/llm-stack/small-bakeoff.md |
- Per-node wall incl. swap: Ornith-9B 175 s, Qwen3.8-9B-Distill 104 s, Qwen3.5-4B 120 s, Gemma-12B 74 s, Qwen3.5-4B-AWQ 62 s. Speeds reproduced within 1 tok/s of the manual runs.
- The run's replies were scored as a fourth sample each: Ornith-9B 0/8, Qwen3.8-9B-Distill 4/8, Qwen3.5-4B 2/8, Gemma-4-12B 0/8 (its four samples are 0 · err · 6 · 6 — high variance), Qwen3.5-4B-AWQ does not compile.
- Graph
llm-stack-small-bakeoff, 7 python_script nodes, $0.00.
Process notes
1. Establishing ground truth on the machine before touching models
- Read the GPU state with
nvidia-smi,lspci -nn, and sysfs DRM connectors, not from memory notes. This caught two things the notes had wrong: both monitors were cabled to the 3090, and the VS Code instance GNOME restored at login had bypassed the render-node pin. - Fixed at the source: cables moved to the 5060, a
mutter-device-preferred-primaryudev rule for the 5060 (PCI id 10de:2d05), a full logout. Verified by a new gnome-shell PID and the 3090 dropping from 383 MiB to 31 MiB of stubs. A lock/unlock did not re-pick the GPU — only a real session restart does. - Set the 3090 to 250 W via a systemd oneshot, because every reference benchmark for this card was taken at 250 W and batch-1 decode is bandwidth-bound (a 450 W 4090 measured only +1.9%).
2. Model selection — primary sources over blog posts
- Pulled the Hugging Face API (creation dates, download counts, safetensors sizes,
config.jsonarchitectures andquantization_config) instead of trusting "best local LLM 2026" articles, several of which still recommended 2024 models. - Read each model card's own benchmark table, then cross-checked against a third party's comparison that ran all candidates on one harness (peculiar-ragdoll's SWE-bench-Live 25-problem set). Vendor numbers were kept but labeled.
- Ampere filter applied up front: NVFP4 (Blackwell only) and FP8 weights (no native sm_86 FP8 GEMM) were excluded regardless of download counts.
3. Serving-stack research
- Two reference deployments for Qwen3.8-27B on exactly this card were read in full:
syv-ai/qwen38-27b-rtx3090(vLLM 0.27.1 fork) and0x7067/qwen38-27b-rtx3090-llamacpp(patched llama.cpp), including their issue trackers (the FlashInfer+MTP k=4 crash, the recurring Xid 31, the sm80 Marlin repack fault, the 16 GB RAM failure). - For Ornith/Tiel and DiffusionGemma there was no single-3090 vLLM report, so the checkpoints were vetted by reading
quantization_configdirectly: asymmetric int4 MoE (cyankiwi/ulkaa Ornith) is rejected by vLLM's Marlin WNA16 MoE kernels; MIRALABS' symmetric requant is 25.6 GB (too big for one card); biMEMO's AutoRound int4 at 20.9 GB with an intact BF16 MTP head was the only fit. For DiffusionGemma, cyankiwi's AWQ-INT4 was confirmed symmetric (group 32) from its config before download. - vLLM support was verified in the source tree at the
v0.28.0tag:DiffusionGemmaForBlockDiffusionandQwen3_5MoeMTPin the model registry,--diffusion-config/--language-model-only/--speculative-configinarg_utils.py, and the diffusion sampler defaults read fromgeneration_config.json.
4. Swapping three models on one card
- vLLM itself is one-model-per-process; its Sleep Mode frees VRAM but does not switch models.
llama-swapv251 fronts arbitrary commands (its README lists vLLM and Docker explicitly), so onedocker runper model withcmdStop: docker stopgives a single OpenAI + Anthropic endpoint on:11440with one model resident at a time. Aliases keep old callers (gemma-4-26b-a4b) working without code changes. - Each container is pinned to the 3090 by GPU UUID so nothing can land on the 5060.
5. Test harness
bench.py(stdlib only): streamed chat to measure TTFT and decode tok/s from server-reportedcompletion_tokens; a fixed coding prompt (garage-door size parser + 8 pytest cases); a ~6 K-token prefill test followed by a second turn to measure prefix-cache TTFT; an OpenAI tool call checked for the right function and argument; an Anthropic/v1/messagescall.quality.py: extracts the code block from the saved reply, runs the model's own pytest cases, then an external 8-case set the model never saw. "Quality" here means the code executes and passes, not that it reads well.- Thinking is on by default for Ornith and Qwen; each is measured both ways because reasoning tokens inflate tok/s and eat the max_tokens budget (the first Anthropic test "failed" only because 32 tokens all went to the thinking block).
6. What went wrong and what was done about it
- Ornith download stalled on a
.gitignore.lockbecause~/.cache/huggingfaceis root-owned from an old container run; the--local-dirdownload recovered on its own, but the xet cache fell back to a slow path. Not fixed (needs sudo); noted. - DiffusionGemma on vLLM 0.28.0 crashed at CUDA-graph capture:
flashinfer prefill_wrapper.plan()argument #14 "Expected bool but got Tensor" — a FlashInfer 0.6.16 API mismatch on the bidirectional-attention path. No matching upstream issue existed. Worked through alternative attention backends and eager mode (see results). - Container images pulled at the same time as 38 GB of weights; the Qwen lane (image → 20 GB download → CPU requant → compile) is the long pole and was started first for that reason.
7. ai-pitlane as the playground
- venv built, 1,662 tests green. New project
llm-stack-bakeoff(5python_scriptnodes, $0 hard cap, manual trigger) chains probe → bench Qwen → bench Ornith → bench DiffusionGemma → summary, sequential because one GPU. Design docplans/design/llm-stack-3090.mdcarries the architecture diagram, per the repo's rule that diagrams never go on the canvas.
8. The playground run, and one more lesson
- Launched
llm-stack-bakeoffthrough the dashboard (POST /api/graph/launch, a transient systemd unit). probe → Qwen → Ornith ran clean (Ornith's warm swap took 93 s, not 3 min); DiffusionGemma "exited prematurely" through llama-swap even though the same flags booted directly. - Cause: llama-swap reads its YAML at startup and I had edited the DiffusionGemma entry (Triton backend, Gemma4 tool parser) without restarting it, so the run used the old FlashInfer command and hit the known crash. Restarted llama-swap, re-verified, re-launched. Rule: edit config → restart llama-swap (and do not
pkill -fa pattern that matches your own shell).
9. Small-model round (afternoon)
- Candidates chosen from the HF API the same way (created since May, ≤ 12 B, Ampere-compatible weights), then vetted by size: every published int4 of a Qwen3.5-9B-family model carries bf16 vision/embedding tensors and is 8.6–18 GB, so the 9 B models were run in bf16 instead. Doug ruled the RTX 5060 out for inference, so the planned 5060 lane was dropped before it ran.
- Download throughput was per-connection capped (~5 MB/s from the HF CDN); five parallel streams gave 30 MB/s. The fixed
~/.cache/huggingfaceownership (chowned) removed the xet permission errors but was not the bottleneck. - Scoring rule added: when a reply's concatenated code does not compile (a model that restarts itself, or truncation), score the first block that compiles and defines the function. This rescued exactly one sample (Qwen3.5-4B, 2/8) and left every genuine syntax error as a failure.
- Benches were queued as a single serial chain gated on each download's completion, so the GPU never idled while the next weights were arriving. A missing
exit=stamp in the five-stream downloader stalled the chain once; a watcher that stamps logs when the download process exits fixed it. - Finding that generalizes: at batch 1 on a 3090, a bf16 9–10 B model with no drafter decodes at ~44 tok/s — slower than a 27 B W4A16 model with speculative decoding. Weight bytes per token, not parameter count, set the speed.
10. Controlled re-test — prompt + sampling
- Found that every model had been benchmarked at
temperature 1.0(each checkpoint'sgeneration_configdefault; the harness sent nothing). Re-ran all eight with three changes: the unit convention stated in the prompt (bare numbers are feet), a system prompt demanding one fenced block with no revisions and raw single-quoted regexes, andtemperature 0.2 / top_p 0.95 / presence_penalty 1.0. DiffusionGemma kept default sampling (vLLM 0.28 rejects those parameters for diffusion models). Each model in its best baseline mode, three samples, ~30 min through llama-swap. - Effect: compile failures fell from 11 of 33 baseline samples to 2 of 24; restart loops disappeared everywhere (replies 420–1,100 tokens instead of hitting the cap). Correctness moved most where the unit convention was the miss (Gemma-12B 0·err·6·6 → 8·8·6; Qwen3.5-4B never-compiles → 0·8·6). Qwen3.8-27B went from 8/8 on most samples to 8/8 on all three with all own tests passing. Ornith-35B did not improve (err·4·4) and still wrote a double-quoted raw regex containing
"despite the instruction. - Interpretation: about half of the "quality" gap in the baseline was mine — an ambiguous spec and chat-default sampling. The remaining gap (Ornith-9B, the 4 B class) is the model.
Files
~/llm-stack/config/llama-swap.yaml— the three model definitions with every flag and the reasons for them~/llm-stack/bin/bench.py,bin/quality.py,bin/build_report.py— harness and this page's generator~/llm-stack/RESULTS.md,PROCESS.md,logs/(boot logs, bench outputs, saved replies)~/Desktop/ai-pitlane/ai-queue/operations/projects/llm-stack-bakeoff.graph.json~/Desktop/local-ai-research-2026-08-30/— the parallel 11-agent research pass (synthesis in12-SYNTHESIS.md)- Research brief that preceded this: 3090 Local Coding Stack