A day spent teaching an agent pipeline what its local models' context windows actually are — and the three faults that only showed up when the real server answered.
On a 24 GB RTX 3090 the weights of a 27B-class model take 17–19 GB, and the KV cache gets what is left. That leftover, divided by the bytes each token costs — two per element on all three lanes, which keep their KV cache in bf16 — is the context window you actually run, whatever the model card says. On the workstation from the bake-off, that comes to:
| Model (alias) | Running window | KV budget | Concurrency at full ctx | Native context |
|---|---|---|---|---|
qwen3.8-27b | 65,536 | 68,605 | 1.05× | 262,144 |
ornith-1.5-35b | 32,768 | 70,870 | 2.16× | — |
diffusiongemma-26b | 32,768 | 102,853 | 3.14× | — |
Concurrency = KV budget ÷ window. Below 2×, a second full-length request queues behind the first — the strongest lane is effectively single-flight.
Two consequences. The model you rely on for correctness is natively 262K but runs at 64K, and the two faster models run at half that. And any pipeline node that reserves output tokens up front — before counting its input — is spending a budget it never measured. The pipeline in question is AI Pitlane, whose local-model nodes had been written against a 131K llama.cpp lane that no longer existed.
Before touching code it was worth knowing whether "the window" is even the right number to budget against. The literature is unusually consistent: it is not. The usable window of an open-weight model is much smaller than the advertised one, and in coding agents the few controlled comparisons that exist all favour less context.
| Source | Finding | Number |
|---|---|---|
| NoLiMa (Adobe, 2025) | Effective length at 85% of short-context score. Llama 3.3 70B claims 128K. | 2K |
| NoLiMa | Gemma 3 27B accuracy at 32K vs 73.3 at 1K | 9.5 |
| Lost in the Middle (2023) | GPT-3.5 multi-doc QA, answer 1st → 10th of 20 docs; the middle scored below closed-book (56.1) | 75.8 → 53.8 |
| SWE-agent (2024), Table 3 | File viewer 100 lines vs the whole file | 18.0 vs 12.7 |
| SWE-agent | Keep last 5 observations vs full history | 18.0 vs 15.0 |
| Context as a Tool (2025) | Milestone compression vs append-only, Qwen2.5-Coder-32B on SWE-bench Verified | 57.6 vs 49.8 |
| Multi-turn (Microsoft/Salesforce, 2025) | Average drop when instructions arrive across turns; o3 98.1 → 64.1 | −39% |
The open-weight rows are the previous model generation. The shape is what carries; the magnitude on the models above is unmeasured and is the first follow-up below.
The practitioner guidance (Anthropic's context-engineering post, the Manus write-up, aider's repo map) converges on the same moves: budget before you call, mask stale tool output, pass paths instead of contents, put the ask at the end, keep the prompt prefix stable for the KV cache. The cheapest of those is the first, and it was the one the pipeline lacked.
The local implementer node defaulted to max_tokens: 16000. vLLM validates prompt + max_tokens ≤ window before generating, so that reservation comes off the top before any input is counted. The work-order picker inlines each input file verbatim up to 40 KB — about 10–13K tokens of code — with no total cap. And the node asks for whole-file Replace output rather than a patch, so every edited file is paid for twice: once in, once out.
Segments to scale. On the 32K lanes the node overflowed on the second inlined file. Reasoning tokens, when thinking is on, come out of the same completion budget as the code — which is why the bake-off's rule is "thinking off, or effort low, for code."
vLLM's refusal is an HTTP 400 whose body carries the reason. Each of the four local plugins had its own copy of a request helper that called raise_for_status() inside a bare except and re-raised ConnectionError("vLLM request failed: …") — keeping the status and URL, dropping the body. A budget overflow was recorded in the run as a connection failure, indistinguishable from the server being down. Which, that morning, it also was.
The design question was where a context window belongs. It is a fact about the server, not the graph: the same pipeline should run on a different card without editing a node. So the window lives with the endpoint, as a model profile — one row per alias with its window, whether thinking should be off or at a reasoning effort, and whether the endpoint rejects sampling parameters (one of the three does).
"model_profiles": {
"http://127.0.0.1:11440/v1/chat/completions": {
"qwen3.8-27b": {"context_window": 65536, "thinking": "off", "source": "manual"},
"ornith-1.5-35b": {"context_window": 32768, "thinking": "off", "source": "shipped"},
"diffusiongemma-26b": {"context_window": 32768, "no_sampling_params": true, "source": "shipped"}
}
}Three writers, in order of authority: typed on the connections page (manual), read from the server's /v1/models when it reports max_model_len (probe, which never overwrites a typed value — and which, it turned out, has nothing to read behind this router; see below), and a shipped table for the aliases the reference machine serves (shipped, shown as a default until confirmed).
Every local node now posts through one client instead of four private copies. It resolves the window (a per-node override, else the profile, else unknown — in which case budgeting is disabled and the log says so), estimates the prompt at 3.5 bytes per token, and plans max_tokens: 0 means fit — the room left after prompt and margin, capped at the plugin's old fixed default, which became its ceiling. Thinking goes out as chat_template_kwargs. The HTTP body survives. A new context_overflow halt category labels the failure, placed above the spending category because "ContextBudgetError" contains the word budget and a prompt that does not fit is not a spending problem.
The settings surface needed nothing new. The pipeline already resolves node config → project defaults → plugin default, injecting a project default only where a plugin's constructor accepts the key. Declaring thinking, context_window, context_margin_tokens and on_overflow on the four plugins meant one line in a project's defaults — "thinking": "off" — reaches every local node in the graph. That line is the setting anyone will actually touch, so it got a field and a one-click template on the project-settings panel.
This repository requires an adversarial review pass before any cross-cutting commit: a second, fresh-context reviewer whose only job is to find the assumption that makes the fix a waste of time. It found one.
"Fit" treated any positive room as a valid max_tokens. There was no floor. A 28K-token prompt on a 32K lane — the exact case that used to hit vLLM's 400 — would now send max_tokens ≈ 4,256; a 29K prompt, 3,256. The model gets cut mid-file. Nothing reads finish_reason. The directions parser requires a closing fence, so it silently drops the unterminated section and applies the rest. Earlier files get committed under a subject line claiming all of them, and the run reports complete.
The change converted a loud, correctly-labelled failure into a silent partial commit. On the 32K lanes it was written for, that is net-negative.
Two small changes fixed it, both from the auditor's report. plan_budget gained a floor — 4,096 tokens for code replies, the bake-off's own measured rule ("1K truncates Qwen mid-file"), 256 for the reviewer whose reply is a one-line verdict — and refuses the call before it is made when the room is below it. And the client raises on finish_reason == "length" rather than returning the fragment. The audit also caught that typing a thinking preference on a shipped alias promoted the whole row to "set by you", which would have frozen a shipped guess against the probe forever; source is now the provenance of the window alone.
A request that fits the window is not a request that works. Any budget that shrinks output toward zero as input grows needs a floor, and any pipeline that applies model output needs to check whether the model was allowed to finish. The old fixed 16,000 was wrong, but at least it failed where you could see it.
The stack had been down all day: the machine rebooted at 06:06 and the model router was a hand-started process. Making it a boot-enabled user service took two minutes; what followed took the afternoon, because three things were wrong that no unit test, no mock and no amount of reading could have shown.
The router's own /v1/models is built from its config; it carries no max_model_len. Autofill would only work against a bare vLLM port. The code handled it correctly — it recorded nothing — but the design assumption that Test would fill the table was wrong for this topology.
Windows come from shipped or typed profiles behind a router. Documented, not fixed.
The vLLM fork serving the 27B says max_tokens=N cannot be greater than max_model_len=max_total_tokens=M — none of the stock "maximum context length" wording the overflow markers matched. The budget error fell through as generic, and the halt classifier called it unknown.
Markers and halt patterns gained max_model_len; retested → context_overflow.
The first end-to-end run failed with "The model gemma-4-26b-a4b does not exist" — a legacy name the pipeline's reviewer still asks for, which the router maps to the 27B Qwen container. The router swapped that container in — the call still took 53 s of container start — but forwarded the alias name, and the vLLM inside only accepts its served name, qwen3.8-27b. The alias the config carried "so the pipeline works unchanged" had never once done so.
useModelName on all eight router entries; verified on three aliases across two models.
With those fixed, the numbers that mattered: the real implementer node, default config, ran a real work-order against a throwaway repository and produced a correct commit; the same node with max_tokens: 16 hit finish_reason=length, failed with a named error, committed nothing, left the tree clean, and classified as context_overflow. Container start 53 s with the torch.compile caches already on disk — the bake-off's ~3 min cold start and 4.5 min first boot include building them; later swaps 38–81 s; thinking off and reasoning_effort: low both accepted; a ~63K-token prompt refused before any call with the arithmetic in the message; the no-sampling-params model answered with temperature correctly withheld.
1,748 tests, 66 of them new. One shared client where there were four. Five settings that did not exist that morning. Three faults only the live server could show — two fixed, one documented; the oldest had been there a year. Paid spend: nothing — every call went to the card under the desk.
--kv-cache-dtype fp8 halves that, which on paper doubles every KV budget in the first table: the two 32K lanes could run at 64K, and the Qwen lane would leave single-flight (1.05× → 2.1×). The evidence that 8-bit is close to free is the llama.cpp thread in the sources — q8 K and V flipped at most 4 of 500 answers on any model tested, while q4 flipped 3/500 on Qwen3.8-27B but 375/500 on Qwen2.5-7B; the K half is the sensitive one. Those counts are multiple-choice, not code, and whether the Ampere attention backends here (FLASH_ATTN on the Qwen fork, TRITON_ATTN for DiffusionGemma) and the DFlash2 drafter accept fp8 KV at all is unmeasured — so it gets scored with (1) before it becomes a setting.on_overflow: trim_ground_truth is declared and fails loudly. It needs the picker to cap the total ground truth and emit signatures for files the task will not write.max_tokens. Same client, one family at a time.