Speculative decoding: getting more tokens per forward pass

Language models write one token at a time. To produce the next token, the model runs a full pass through all of its billions of parameters — and it can’t start that pass until the previous token is finished, because the new token depends on everything written so far. The generation, also known as autoregressive generation, is therefore a strictly serial process, and that is the main reason it feels slow.

Speculative decoding is a way to speed this up by generating several tokens per compute pass instead of one, and to do it without changing a single thing about what the model outputs. The results are identical to ordinary decoding — just faster.

Speculative decoding has proven to be an effective technique for faster and cheaper inference from LLMs without compromising quality.^[1]

Memory vs Compute

To see where the opportunity is, it helps to know exactly where the time goes.

Each pass through the model has to read every one of its parameters out of the GPU’s memory. For a large model that’s tens of gigabytes, and moving those bytes from memory into the chip’s compute units is the slow part. The actual arithmetic — multiplying the numbers together — finishes almost instantly by comparison. So when you generate a single token, the expensive resource isn’t math; it’s memory bandwidth, and the arithmetic units sit mostly idle waiting for data to arrive.

Autoregressive: Memory vs Compute

The weights aren't cold — they live in the GPU's high-bandwidth memory (HBM) for the whole session. "Stream weights" means moving them from HBM into the tiny on-chip memory beside the compute cores, which is far too small to hold the model, so every forward pass re-streams the full set. That HBM→chip trip, once per token, is the bottleneck — not a disk load.

Here’s the consequence that makes everything else work: because the weights have to be loaded anyway, running the math for a handful of tokens in the same pass costs barely more wall-clock time than running it for one. The bytes are already on the chip; putting the idle arithmetic units to work on a few extra tokens is nearly free.

That’s the free lunch. The only catch is that normal decoding can’t take advantage of it — to run token 100 you need token 99, which needs token 98, and so on, so you’re forced back into computing them one at a time. Speculative decoding is the machinery that lets us fill a pass with several tokens even though we don’t yet know for certain what those tokens are.

Prefill vs Decode

Generation actually has two distinct phases, and they behave completely differently. When you hand the model a prompt like “The quick brown fox,” it reads every one of those tokens in a single parallel pass — none of them depend on the model’s own output, so the arithmetic units are genuinely busy. This is prefill. Only afterwards does the model enter decode, where it must produce the continuation one token at a time, spending a whole forward pass per token and reusing the cached keys and values from every earlier position.

Prefill & Autoregressive Decode

Every forward pass streams the weights (memory, warm) and then runs the math (compute, the green sliver). Prefill does the whole prompt in a single pass — for a short prompt that's about the wall-clock of generating one token — so it's one block, not one per prompt token. Decode then spends a full pass on every token, reusing the cached keys and values (faint green) from every earlier position. Collapsing that tower of decode passes back toward a single pass is exactly what speculative decoding chases.

The staircase is the whole story: prefill is one pass that swallows the whole prompt at once — for a short prompt, about the cost of generating a single token — while decode is a tower of narrow passes that each yield one token. Speculative decoding is, in effect, an attempt to make decode look more like prefill — to verify several candidate tokens in one parallel pass.

As a crude analogy: Speculative decoding tries to batch decode tokens like prefill batches prompt tokens.

Draft & Verify

The way speculative decoding works is by using a smaller, faster process/model to predict the next token(s) - known as draft tokens - and then using the larger, more accurate model to verify those predictions in a single parallel pass.

By predicting and verifying multiple tokens simultaneously, this technique shortens the path to results and makes AI inference faster and more responsive, significantly reducing latency while preserving output quality.^[2]

Two things are important to make this work:

The draft model needs to be fast and cheap to compute, it should be a lot more efficient than the larger model and not be too heavy on resources
The draft model needs to be reasonably accurate, it should be able to predict the a few of the next token(s) correctly most of the time. Otherwise we are just waisting compute.

Speculative Decoding — a draft model guesses, the target verifies

0 target passes · 0 draft passes · — tokens / verify · plain decode: 9 passes

A schematic, not a benchmark: the block sizes here are illustrative only and do not reflect the true ratio of memory to compute, or the real cost of any single pass — they are drawn to make the sequence legible, not to scale. What they show is the trick: a tiny draft model guesses 4 tokens ahead in a blink, and the expensive target model checks the whole batch in one pass, reusing its cached keys and values (faint green) for every earlier position. Everything up to the first wrong guess is kept, and the rejection itself yields the target's corrected token for free — 9 tokens from 4 target passes instead of 10.

Watch one round at a time: the draft model spends a handful of its own compute passes drafting multiple tokens ahead, and the target model spends exactly one expensive compute pass checking the whole batch once. It keeps every token up to the first wrong guess and overwrites that guess with its own choice, so a single target pass commits several tokens. The more often the draft model guesses right, the more tokens are verified and used on the expensive pass.

Autoregressive (Sequential) vs Parallel

The key insight thusfar is that speculative decoding allows us to verify multiple tokens in a single pass, instead of having to verify them sequentially. This is a big win when it comes to latency. As long as the drafting of tokens is relatively cheap this means we can get tokens back to the user faster.

When it comes to the drafting there are different approaches that have been tried and tested. Of the popular drafting strategies fall into two main camps: autoregressive and parallel.

Autoregressive (Sequential) Drafting

EAGLE3 and MTP (Multi-Token Prediction) rely on autoregressive drafting: they generate tokens sequentially (recursively for EAGLE3), which provides strong inter-token dependencies and often solid acceptance rates, but the sequential nature can limit speed for longer drafts.

Parallel (& Hybrid) Drafting

In contrast, two recent implementations, DFlash ^[3] and DSpark ^[4], emphasize parallel drafting.

DFlash uses a lightweight block diffusion model to predict an entire block of tokens in a single forward pass. This maximizes GPU utilization and minimizes drafting latency, making it ideal for high-throughput, structured tasks.
DSpark takes a hybrid approach, pairing a parallel backbone with lightweight sequential elements to preserve inter-token dependencies and reduce acceptance decay over longer sequences. It excels in adaptive, high-concurrency environments.

Overall, parallel methods trade a small degree of dependency precision for dramatically faster draft generation, pushing overall engine speedups significantly higher.

References

[1]Looking back at speculative decodingGoogle Research
[2]An introduction to speculative decoding for reducing latency in AI inferenceNVIDIA Developer
[3]DFlash: Block Diffusion for Flash Speculative DecodingarXiv
[4]DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive GenerationarXiv