Research · local models · Gemma 4 settings

Thinking Off, Temperature Zero

What Gemma 4's Settings Actually Do on One RTX 3090

Doug (freePerro), with Claude2026-09-02RTX 3090 · 24 GB · 250 Wsource: llm-stack/docs/gemma-settings.md

Abstract

A question about the newest Gemma turned into a day of measuring what Gemma 4's serving settings really do. The newest checkpoints that fit a 24 GB card were already installed; the 31B does not fit one RTX 3090 on vLLM at all. What was new was Google's multi-token-prediction drafter, which took the 12B from 78 to about 135 tokens per second at the same scores, once two things were learned: the drafter must come from the same QAT pipeline as the target (the QAT drafter accepts 3.1 tokens per step, the plain one 2.6), and vLLM 0.28.0 only boots it with piecewise CUDA graphs. Then the settings. Rendering the prompt through the server showed that reasoning_effort in the chat-template arguments is a no-op for Gemma 4, so every "thinking" number the suite had recorded for this model was thinking-off. Turned on properly, thinking spent 4,000 to 8,000 tokens on a small parser task and, at temperature 0.2, degenerated into a literal loop on five of five samples. A per-request thinking_token_budget that vLLM implements but Google does not document bounds it; of three budgets tried, 512 cut thoughts short and 1,024 did not. Across seven sampling and thinking variants the only setting with every sample at 8/8 was thinking off at temperature 0, and its five replies were byte-identical. A second bank of 41 prompts across five task types (code with hidden tests, exact-answer math, JSON, tool calls, document extraction) under four recipes then showed the limit of that result: greedy decoding was flawless on 40 prompts and, on four re-runs, wrong the same way on one; Google's default temperature never scored zero on a task across its two samples but slipped on partial cases; budgeted thinking was perfect on math and lowest on code by one partial miss. Greedy is also repeatable only for one request at a time on a warm container: the first reply after every fresh boot differed, and four identical requests sent in parallel returned three different replies. The recipe that shipped is greedy, with a rule: retry a failed check at temperature 1.0, not at temperature 0 again.

Doug (freePerro) with Claude · 2 September 2026 · research note — every number is traceable to a run record under logs/ in the llm-stack repository; the previous note in this series is [A known state for every local model](https://squirrelglitch.com/research/known-state/).

1. Introduction

The starting question was operational: which Gemma is newest, and can it run on the single RTX 3090 that serves local models to an agent pipeline through a swapping proxy. The answer to the first half was quick. Gemma 4 (March 2026) is still the family; its last releases were the 12B Unified model in June, the QAT compressed-tensors checkpoints a day later, DiffusionGemma, and the MTP "assistant" drafters, one of which was refreshed on 20 August. Nothing newer exists under Google's Hugging Face organisation as of 2 September. Two of those were already installed. The one that was not, the 31B, cannot be served on this card by vLLM: Google's own QAT W4A16 checkpoint is 23.3 GB on disk because the embeddings and vision tower stay in bf16, the community 4-bit builds are 19 to 21 GB, and the reference RTX 3090 community repository records a single-card boot OOM on 24 GB Ampere regardless of KV-cache format [8].

So the "newest Gemma" for this hardware was a drafter for a model already in service. Setting it up (§3) was straightforward once its two traps were found. The second half of the day came from a hint: some of the settings might be different than assumed. They were. §4 to §8 report what the chat template, the serving engine and the model actually do under thinking, budgets and sampling, measured with the known-state tools built the previous day [12] plus two new ones. §9 is the broad test, §10 is what changed in the shipped recipe, and §11 compares all of it with what Google's documentation says.

Contributions:

1. A working, measured MTP lane for Gemma-4-12B on one 3090 (§3): drafter choice by acceptance length, depth by measurement, and the vLLM 0.28.0 boot fault with its workaround.
2. Ground truth for the thinking controls (§4), obtained by rendering prompts through the server rather than reading documentation.
3. The thinking-loop failure and its two fixes (§5, §6): a repetition penalty, and vLLM's undocumented-by-Google per-request thinking budget, with the budget size below which it hurts.
4. A 41-prompt, five-type hammer (§9) on which greedy decoding's one loss was a whole task, sampled decoding's losses were partial cases, and budgeted thinking was perfect on math and lowest on code.

2. Setup

One RTX 3090 (24 GB, Ampere sm_86, power-limited to 250 W) holds only the inference engine; the desktop is rendered by a second card. llama-swap fronts one docker run per model on 127.0.0.1:11440, swapping containers so that one model is resident at a time. Gemma 4 runs on the stock vllm/vllm-openai:v0.28.0 image with --reasoning-parser gemma4 and --tool-call-parser gemma4, the checkpoint's own chat template, a 65,536-token window and four concurrent sequences.

Target. google/gemma-4-12B-it-qat-w4a16-ct, Google's quantization-aware-trained 4-bit checkpoint in compressed-tensors format (group size 32, symmetric), served with --language-model-only so the vision path is not loaded. Weights occupy 9.07 GiB. The KV pool depends on the boot: 10.47 GiB (267,254 tokens) on the first boot, which compiles, and 12.17 GiB (310,753 tokens) on later boots that reuse the compile cache; the undrafted entry gets 12.98 GiB (331,344 tokens) on a warm boot, the drafter's weights and graphs accounting for the difference at the same 41 KiB per token.

Drafter. google/gemma-4-12B-it-qat-q4_0-unquantized-assistant, the QAT-pipeline version of the 4-layer MTP head that shares the target's embeddings and KV cache [3]. It is stored in bf16; what makes it "QAT" is the pipeline it was trained in.

Tools. bench.py (streamed decode speed on a fixed coding prompt), quality.py (hidden pytest sets), modeltest.py and refine.py (the thirteen-probe known-state suite and its recipe selector [12]), and two written for this note: sweep.py, which runs the bake-off code task under named recipe variants, and hammer.py, which runs a bank of 41 prompts across five task types with a scorer per type.

Sampling vocabulary. "Google default" means no sampling parameters in the request, so the checkpoint's generation_config applies: temperature 1.0, top-p 0.95, top-k 64. "House" means the recipe the suite had carried for this model since the previous sweep: temperature 0.2, top-p 0.95, presence penalty 1.0, thinking off, an 8,192-token output cap and a system prompt that demands one fenced code block. "Greedy" is temperature 0.

3. The drafter

Google ships two MTP drafters for the 12B: the plain gemma-4-12B-it-assistant and a QAT-pipeline variant. The QAT model card says a QAT target "must" be paired with a QAT drafter of the same precision [4]; the reference 3090 repository's profile notes record a lower acceptance length when the plain drafter is paired with a QAT 31B target [8]. Both were measured here, at three draft tokens per step, with bench.py on the same coding prompt, three samples each.

Table 1 — decode speed with and without the drafter (250 W, bench.py, three samples)

The QAT drafter gives about 1.7× decode; the plain one about 1.5×. Rejection sampling preserves the output distribution [14], not the bytes: at temperature 0 the drafted reply on record differs from the undrafted greedy reply in two of the model's own test lines, both scoring 8/8, though a later sequential greedy request on the drafted entry reproduced the undrafted reply exactly, so the variation is numerical rather than the drafter's (§7). The external eight-test scores on the three bench samples at temperature 0.2 were 0/8, 8/8, 8/8, the same bimodal pattern the frozen baseline had recorded for this checkpoint. One caveat belongs to the comparison: the undrafted entry runs full CUDA graphs and the drafted one piecewise (below), so Table 1 bundles the drafter's gain with whatever the piecewise mode costs, which was not isolated. The full known-state suite passed every probe with the drafter, including tools, the Anthropic-style messages endpoint, a 0.16 s cached second turn on a 6,116-token prompt and a 50 s swap.

Draft depth was swept under greedy decoding, four samples per depth, where the replies after the first request of each boot were byte-identical across depths and only the speed changed: end-to-end 117 tok/s at n=2 (acceptance 2.57), 126 at n=3 (3.1–3.2), 128 at n=4 (3.66). n=3 stayed.

The boot fault. With the drafter, vLLM 0.28.0 crashed during CUDA-graph capture of the speculator: Cannot copy between CPU and CUDA tensors during CUDA graph capture unless the CPU tensor is pinned. The piecewise captures had succeeded; the failure came in the speculator's full-graph capture (the log labels the step "prefill CUDA graphs (FULL)"), inside the drafter's compute_logits, which masks the checkpoint's two suppress_tokens (the <image|> and <audio|> end markers) by indexing the logits with a Python list, an unpinned host-to-device copy during capture. The target model applies the same mask with a cached device index, and its logits step runs outside graph capture, so only the drafter trips. Restricting the engine to piecewise graphs with --compilation-config '{"cudagraph_mode":"PIECEWISE"}' avoids the path; every drafted number in this note is measured with that flag. A patch would be a one-line change in gemma4_mtp.py; a flag was preferred to a mounted overlay.

Vision. The same target boots with vision enabled (--limit-mm-per-prompt '{"image":2,"audio":0}' in place of --language-model-only) and the drafter still attached, with no workaround. A 640×320 synthetic test card with two lines of text and two coloured shapes was transcribed exactly and described correctly at 285 prompt tokens. The cost is 10 % of the KV pool (278,589 tokens instead of 310,753 on warm boots) and about 3 % of decode speed (127–141 tok/s against 133–144). The shipped entry stays text-only with the one-line change documented beside it.

4. What the template does with thinking

Gemma 4's thinking is switched on by a <|think|> token at the start of the system turn, which the chat template inserts when enable_thinking is true [1, 2]. The suite's harness had been passing reasoning_effort inside chat_template_kwargs for its "thinking=low/medium/high" variants, the mechanism one other model family honours. Whether Gemma's template honours it can be settled without reading the template: the serving engine's /tokenize endpoint renders the chat template, and /detokenize turns the result back into text.

Table 2 — the rendered prompt for one system + user pair, per chat_template_kwargs

Three facts follow. reasoning_effort in the template arguments is a no-op: the prompt is byte-identical to thinking-off, so every "thinking" measurement the suite had recorded for this model was a thinking-off measurement, and the earlier relabelling of its recipe was correct. Thinking is off by default, and when off the template pre-fills an empty thought block <|channel>thought\n<channel|> so the model starts directly on the answer; Google's thinking notebook explains that this was added for the 12B, 26B and 31B "to stabilise model output by suppressing ghost thought channels" [2]. And only enable_thinking: true inserts <|think|>.

There is a second route that does work: vLLM 0.28.0 maps the OpenAI-style top-level reasoning_effort request field to enable_thinking = (effort != "none") in its chat-completion request handling [6]. The levels are not distinguished; low and high render the same prompt. The harness now has an explicit thinking: on mode and passes extra request fields through, so a recipe can say what it means.

5. Thinking, uncapped

With the switch found, the bake-off code task (a garage-door size parser, eight hidden tests, external to anything the model wrote) was run under seven recipe variants, five samples each, at the house 8,192-token output cap.

Table 3 — thinking × sampling on the code task (external tests passed of 8, five samples)

thinking offthinking on

Two findings. First, on this task thinking off with greedy decoding is the only variant with every sample at 8/8, and its five replies were identical to the byte. The bimodality that the previous sweep had attributed to temperature 0.2 is not the presence penalty's doing: removing the penalty leaves one zero-branch in five, and Google's default temperature drew two, differences that five samples cannot separate.

Second, thinking on is unusable at this cap. At temperature 1.0 the model reasoned for 4,129 to 8,191 tokens about a parser and hit the cap twice in five. At temperature 0.2 it hit the cap five times in five with 8,191 reasoning tokens each. The sweep did not keep reasoning traces, so a sixth sample was run with tracing on to see what fills those tokens: not long deliberation but a degenerate loop. The trace below is its tail, 19,556 characters in which 451 lines contain 336 distinct ones and a single line occurs 29 times:

    Let's use `r"['\"]"`? No.
    Let's use `r"['\"]"`? No.
    Let's use `r"['\"]"`? No.
    Let's use `r"['\"]"`? No.
    Let's use `r

This is not specific to this setup. A public issue on the 12B reports "infinite thinking loops" under llama.cpp [9]; a thread on Google's own developer forum reports coding agents that "almost always devolve into looping thoughts", resolved by the reporter with a repetition penalty of 1.08 over a 4,096-token window [10]; neither has a reply from Google. Applied here as vLLM's per-request repetition_penalty: 1.08 (vLLM's penalty has no window and counts prompt tokens as well as generated ones, so the thread's last-n setting does not transfer), the penalty does what the thread says: thinking at temperature 0.2 finished five of five with scores 8, 8, 7, 8, 8 at 2,049 to 3,266 reasoning tokens, and at temperature 1.0 finished five of five at 8, 8, 0, 6, 8. With thinking off the same penalty scored 0, 8, 0, 8, 8 at temperature 0.2, one more zero-branch than without it, which five samples cannot separate from noise.

6. The thinking budget

vLLM 0.28.0 carries a mechanism Google's documentation does not mention: a per-request thinking_token_budget. A ReasoningConfig derived from the reasoning parser's start and end markers (<|channel> and <channel|> for Gemma 4) tracks each request's reasoning section in the sampler and, once the budget is spent, forces the end marker so the model proceeds to its answer [6]. No server flag is needed beyond --reasoning-parser gemma4: vLLM 0.28.0's argument defaults build the reasoning config from the parser alone, the container log carries no "auto-initialization of reasoning token IDs failed" warning, and the requests confirmed the behaviour.

On a trivial prompt (15 × 37, budget 256, temperature 1.0) three recorded samples reasoned for 207, 150 and 233 tokens and answered 555; an earlier unrecorded trio had reasoned for 159, 256 and 249. On the code task:

Table 4 — thinking with a budget on the code task (external tests of 8)

Over its thirteen samples the 1,024 budget at Google's default sampling averages 7.0 of 8 and finished in twelve; 512 cuts the thought too early. The characteristic failure is visible in the replies: when the forced <channel|> lands mid-thought, the model sometimes keeps deliberating in the answer channel ("Actually, … Let's re-verify"), producing several code fences instead of one, and occasionally a loop that runs to the cap in the answer rather than the thought.

7. Choosing the recipe by measurement

The suite's recipe selector was rerun on the frozen entry with the control recipe, greedy, budgeted thinking, and budgeted thinking with the repetition penalty, five samples each, under its published rule: best mean wins, but a variant within the noise floor of the best displaces it only if it needs at least 25 % fewer tokens [12].

Table 5 — refine.py, code task, five samples per variant

Greedy won outright and was written into the spec. The re-frozen state: code 8/8 on all five at 135 tok/s median, the whole-file edit task 7/7 on all five, every contract probe passing. One caveat was recorded with it, and it grew under review. Greedy is repeatable only for identical requests sent one at a time to a warm container. In three of three cold boots on record the first reply differed (558 tokens twice, 935 once on the vision-enabled boot) before the 524-token reply appeared on every later sample. The failing cents task of §9, re-run four times sequentially on a warm container, gave the byte-identical reply each time. But four identical greedy requests for the code task sent in parallel returned three different replies (524 tokens twice, 548 and 808 once each), and a sequential request immediately after returned a fourth, byte-identical to the undrafted entry's greedy reply from two days earlier. Temperature 0 removes the sampling noise; it does not remove the numerical noise that batch shape and cache state introduce, and a near-tie between two tokens can go either way.

8. What Google's default temperature is for

Google's model card says to use temperature 1.0, top-p 0.95 and top-k 64 "across all use cases" [1]. On the single code task that setting was the worst thinking-off variant. Before treating that as a verdict, the broad test.

9. Forty-one prompts, four recipes

hammer.py runs a bank of 41 prompts in five types, each with an automatic scorer. Thirteen code tasks (a size parser, phone normalisation, money to cents, interval merging, Roman numerals, RFC 4180 fields, slugs, Luhn, run-length coding, word counts, date parsing, bracket matching, hex colours) each carry six to eight hidden pytest cases and a reference solution; a selftest runs every reference against its tests before any model is asked, so a wrong test cannot be blamed on a model. Twelve math word problems have exact numeric answers, scored on the last number in the reply. Six JSON tasks compare expected keys. Six tool-call tasks check the chosen function and its arguments, including one where the right answer is not to call a tool. Four questions are asked over a 365-word service bulletin. The four recipes were greedy (one sample per task, since its output is deterministic), the house recipe, Google's default, and thinking with the 1,024 budget at Google's default, two samples each: 287 requests, fifteen minutes. Thinking-off replies were capped at 4,096 tokens and thinking-on at 8,192; no thinking-off reply exceeded 1,599.

Table 6 — mean score per task type (1.00 = every hidden check passed)

Every recipe was perfect on JSON, tool calls and extraction, including the no-tool case. The differences are in code and math, and the per-task view (Figure 2) shows their shape.

Greedy's whole loss is one task. On the cents parser it wrote a three-group regular expression and unpacked the match into two names, so every input raises. It was perfect on the other twelve code tasks and on all twenty-eight non-code prompts. The hammer ran greedy once per task, so the reply was re-run four times sequentially on a warm container after review: byte-identical each time. That is what greedy decoding's risk looks like: when the single most likely path is wrong, an identical retry returns the same bug. Only a change in conditions, a parallel neighbour or a cold cache (§7), or a different temperature, gives a different draw.

Google's default never had a task mean of zero across its two samples. Its misses were 7/8 on cents and hex colours, 5/8 on dates, and one math reply that showed the correct working, "84 / 21 = 4", and then printed "404" on its final line. All four fell in the first sample of their task; the second samples were 41 of 41, which two samples per task cannot explain.

Budgeted thinking was the only recipe perfect on math and lowest on code, by one partial miss over greedy: on two code tasks the forced cut left the model deliberating in the answer, once with seven code fences and once looping to the cap, and two more replies scored 6/8 and 7/8. It also costs five times the tokens.

The house recipe had the most partial code misses (2/8, 4/8, 5/8, 7/8) and one arithmetic slip, consistent with the bimodality seen at temperature 0.2 throughout.

10. What shipped

The entry gemma-4-12b-mtp is the QAT 12B with the QAT drafter at depth 3, piecewise graphs, the checkpoint's canonical template, both Gemma 4 parsers, 64K context and four sequences. Its recipe is thinking off at temperature 0 with the house system prompt and an 8,192-token cap.

Table 7 — what to use, by need

What not to use, because it measured worse: temperature 0.2 with or without penalties; thinking without a budget; a 512 budget; presence or repetition penalties on thinking-off code; the plain non-QAT drafter. Google's default and greedy tied on the hammer, 0.98 to 0.98; greedy is the pick where a repeatable output matters, and Google's default is the equal choice where a retry is possible.

11. What Google's documentation says

For comparison, the official guidance, page by page. Sampling: the model card's only statement is the standardised configuration of temperature 1.0, top-p 0.95, top-k 64 for all use cases [1]; the technical report has no decoding guidance [3]; Google's own MTP notebook runs its examples with do_sample=False, which is greedy [13]. Thinking: enabled by <|think|> at the top of the system turn; when off, the model still emits the tags with an empty thought block [1]; the thinking notebook adds that the empty block was placed in the template for the three larger models to suppress ghost thought channels [2]. No page mentions a budget, effort levels or any length control; the examples set max_new_tokens=1024. Nothing warns about loops or low temperature; the only loop discussion is on Google's community forum and in a public issue, neither with a staff reply [9, 10]. Multi-turn: strip previous thoughts from history before the next user turn, "with the exception of tool call turns where thinking content should be preserved" [1, 2]; the canonical template enforces this if the client echoes the reasoning back on tool-call turns. MTP: the technical report describes the four-layer drafter that cross-attends to the target's KV cache, with no acceptance rates or recommended depth [3]; the assistant model card promises "up to 3×" [15]; the transformers notebook defaults to four draft tokens with an adaptive draft-length schedule [13]; the QAT card's one hard rule, pair a QAT target with a QAT drafter, is the one §3 confirmed [4]. Serving: Google's Cloud Run guide runs Gemma 4 on vLLM with both gemma4 parsers, --generation-config auto, prefix caching and chunked prefill, and enables thinking per request through chat_template_kwargs [11]; the entry here matches all of that. One page is stale: the prompt-structure reference still documents Gemma 3's turn tokens and says the system role is not supported.

12. Threats to validity

One model, one quantization, one card, one serving engine at one version. The code task in §5 to §7 is a single prompt; its bimodality is real but its ranking of recipes did not survive the 41-prompt test unchanged, which is the reason §9 exists. The hammer bank was written in a day and has two samples per sampled recipe; a 0.02 difference in Table 6 is one partial miss. The hidden tests were proven against reference solutions but the prompts are the authors' phrasing, and a model that reads a specification differently is scored as wrong. Greedy decoding was byte-stable within a warm container but produced a different first reply after every one of three cold boots, and the drafted greedy reply differs from the undrafted one; every determinism claim here was measured one request at a time with a warm prefix cache, while the entry serves an agent pipeline at four concurrent sequences, and the one parallel check made (§7) shows greedy replies diverging under exactly that load. The hidden tests were proven against reference solutions, but the prompts are the authors' phrasing: the cents prompt fixes the order of the dollar sign and the minus sign only by example, and both the house and the Google-default partial misses on that task fail exactly the "-$5.00" case, which is a specification miss scored as a model miss. Table 1 compares an undrafted entry on full CUDA graphs with a drafted entry on piecewise graphs; the piecewise cost was not isolated. The KV-pool figures depend on whether the boot compiled. sweep.py and refine.py disagreed on the budgeted-thinking variant (8, 8, 8 and 8, 6, 8, 8, 8 against 5, 0, 8, 8, 8) at temperature 1.0 with nothing else changed, which is the noise floor of five samples, not a bug found. The loop detector counts a repeated line six times; one house-recipe reply it flagged was a long but correct answer that restated a function. All four Google-default misses on the hammer landed in the first sample of their task, which is either chance at one in sixteen or a first-request effect the two-sample design cannot resolve.

13. Reproducibility

Everything is in the llm-stack repository: config/llama-swap.yaml holds the entry with the flags and the reasons in comments; specs/gemma-4-12b-mtp.yaml is the frozen known state with every finding in its quirks list; bin/sweep.py and bin/hammer.py with hammer/tasks.py are the two new tools, and hammer.py --selftest proves the task bank; PROCESS.md §11 is the day's method log; replies, reasoning traces, summaries and the program output of each run (run.log) are under logs/sweep, logs/hammer and logs/mtp-bench; the drafter, vision and boot figures of §3 are in logs/mtp-bench/decode-*.txt, acceptance-*.txt, boot-*.txt and logs/vision-test-result.json, re-measured after review; the greedy re-runs are logs/hammer/.../greedy_off/c03_cents-rerun.json and logs/mtp-bench/greedy-concurrency.json. The rendered prompts in Table 2 and the budget-256 samples are recorded in logs/gemma-template-render.json, from POST /upstream/<model>/tokenize and /detokenize on the proxy. The drafter fault is in vllm/model_executor/models/gemma4_mtp.py at the suppress_tokens mask; the budget mechanism is vllm/v1/sample/thinking_budget_state.py and vllm/config/reasoning.py.

References

1. Google DeepMind. Gemma 4 model card. https://ai.google.dev/gemma/docs/core/model_card_4
2. Google. Thinking mode in Gemma; and the thinking notebook in the Gemma cookbook. https://ai.google.dev/gemma/docs/capabilities/thinking
3. Gemma Team. Gemma 4 Technical Report. arXiv:2607.02770
4. Google. Gemma 4 QAT checkpoints (compressed-tensors) model card, "Assistant Compatibility". https://huggingface.co/google/gemma-4-12B-it-qat-w4a16-ct
5. vLLM. Gemma 4 usage guide (recipes). https://docs.vllm.ai/projects/recipes/en/latest/Google/Gemma4.html
6. vLLM. Reasoning outputs: automatic enable_thinking activation and the thinking token budget. https://docs.vllm.ai/en/latest/features/reasoning_outputs/
7. vLLM release v0.28.0. https://github.com/vllm-project/vllm/releases/tag/v0.28.0
8. club-3090 issue #814, official Gemma 4 31B QAT W4A16 on 4× 3090, and the gemma-4-31b README's single-card note. https://github.com/noonghunna/club-3090/issues/814
9. google-deepmind/gemma issue #727, gemma-4-12B-it infinite thinking loop. https://github.com/google-deepmind/gemma/issues/727
10. Google AI developer forum, "Gemma4: how to prevent looping output in thinking?" https://discuss.ai.google.dev/t/gemma4-how-to-prevent-looping-output-in-thinking/173205
11. Google Cloud. Run agents with Gemma 4 models on Cloud Run. https://docs.cloud.google.com/run/docs/run-gemma-on-cloud-run
12. Doug (freePerro) with Claude. A known state for every local model. https://squirrelglitch.com/research/known-state/
13. Google. Gemma cookbook: docs/mtp/mtp.ipynb and docs/capabilities/thinking.ipynb. https://github.com/google-gemma/cookbook
14. Leviathan, Kalman, Matias. Fast Inference from Transformers via Speculative Decoding. ICML 2023. arXiv:2211.17192
15. Google. Gemma 4 12B assistant (MTP drafter) model card. https://huggingface.co/google/gemma-4-12B-it-assistant