Speculative decoding under load

A speculative drafter can make each user faster while making the serving system slower overall. Acceptance length held steady in our measurements; the cost of each decode step under load decided the outcome.

Committed

Proposed

Verification steps 0Tokens emitted 0Tokens per step –

Draft length K = 5. Accepted tokens turn green; the first rejection ends the block; the verifier adds one token of its own (amber).

Figure 1 · One speculative cycle. A small drafter proposes K tokens in one pass. The target model verifies them in a single forward pass and keeps the matching prefix plus one token it would have produced anyway. One expensive step emits several tokens. Illustration, not a model trace.

The problem with single-request benchmarks

Autoregressive decoding is expensive because every new token requires another forward pass. At low concurrency, the GPUs spend much of that pass waiting on memory movement rather than using their compute. Speculative decoding exploits that gap: a drafter proposes several tokens, the target verifies them together, and one expensive step produces multiple output tokens.

User concurrency changes the economics. As more requests enter the batch, draft verification costs scale. The draft model also has its own weights, which take memory that could otherwise be allocated to additional KV cache.

More accepted tokens per step do not necessarily mean more tokens per second across the system.

Concurrency changes the answer

Single-request benchmarks show a drafter’s potential. They do not predict its behaviour under production load. Published results and our own experiments show the same broad pattern: speedups shrink as batch size grows, with the exact outcome depending on model, drafter, workload and implementation.

In one comparison, a block-style drafter improved decode speed by roughly 40–50% over an MTP@3 baseline at concurrency 1. At concurrency 32, reported improvements for block drafters of the same general kind ranged from about 1% to 17%. The drafter families and benchmarks differ, so these numbers do not form a single curve.

At low concurrency the target has spare compute. At higher concurrency, verification competes with useful decoding work, longer proposals increase the work that may be rejected, and the drafter’s memory footprint reduces the number of requests that fit.

Baseline step1 token per request

Speculative stepup to K+1 tokens per request

Memory floor (weights, KV reads)Decode computeDraftingVerificationIdle compute

–

Per-user tok/s, baseline

–

Per-user tok/s, speculative

–

Aggregate tok/s, baseline

–

Aggregate tok/s, speculative

–

Aggregate ratio

Concurrent users 4

Figure 2 · Concurrency spends the idle compute speculation was trying to use. Drag the slider. At low concurrency the baseline step is memory-bound and the extra verification work is nearly free. As the batch fills, verification and drafting add real time to every step, and the aggregate ratio falls below 1. Drafting has a small fixed cost (reading the drafter’s weights) plus a per-request term that grows more gently than verification, because the drafter is much smaller. Illustrative model, not measured data; the measured crossover is in Figure 4.

Acceptance length is the numerator, step cost is the denominator

Mean acceptance length (MAL) is the number of tokens emitted per verification step, including the extra token the verifier produces. With conditional acceptance probability α and draft length K:

MAL(α, K) = (1 − α^K+1) / (1 − α)

MAL = 1 means no speculative benefit. But MAL is only the numerator. Throughput scales with MAL divided by the time each step takes, and that time grows with K:

Throughput ∝ MAL(α, K) / T(K)T(K) = t₀ [1 + (v + d) K]

Here t₀ is the ordinary decode-step time, and v and d are the incremental verification and drafting costs relative to t₀. This is a simplified model, not a GPU performance model, and v and d are not constant across concurrency.

At a measured conditional acceptance of about 0.74, raising K from 3 to 7 lifts modelled MAL from 2.69 to 3.50, about 30%. But the draft is longer and the step costs more. If the denominator grows faster than the numerator, throughput falls.

–

MAL (numerator)

–

Step cost T(K)/t₀ (denominator)

–

Relative throughput

–

Best K at this overhead

Draft length K 7 Per-token overhead v+d 0.12

Figure 3 · Optimise useful tokens per unit of step time, not acceptance length in isolation. α = 0.74 throughout. MAL keeps rising with K; the step cost rises with it. Raise the overhead slider, which is what concurrency does, and the best K moves left. MAL values follow the formula above; the overhead term is illustrative.

A measured crossover

We tested an off-the-shelf block drafter at K = 7 on a frontier open-weight MoE model served on a B200 node. The workload was a long-prompt agentic trace with prompts well over 100,000 tokens and a high prefix-cache hit rate. At each concurrency level the speculative and baseline configurations replayed the same conversations.

Figure 4 · Aggregate throughput at the two measured operating points. Speculative relative to baseline, same conversations replayed. The drafter wins at concurrency 10 and loses at concurrency 16, with acceptance length unchanged.

Metric	Concurrency 10	Concurrency 16
Aggregate throughput, speculative vs baseline	1.29×	0.91×
Aggregate computed tokens only (uncached prompt + output)	1.73×	0.80×
Per-user decode speed, speculative vs baseline	2.13×	1.41×
Mean acceptance length	3.07	3.07

Total tokens per second include cached prompt tokens, so they depend on how much cached history is replayed. Counting computed tokens only changes the apparent size of the effect; the crossover remains.

At concurrency 10, aggregate throughput rose 29% and per-user decode speed improved about 2.13×. At concurrency 16, per-user decode speed was still about 1.41× the baseline, yet aggregate throughput fell to 91%. The drafter also consumed a meaningful share of KV-cache headroom on every GPU, a capacity cost that matters most when the batch is already full.

A better drafter can still lose

Acceptance is one part of the cost. A drafter also changes memory capacity, step time and the surrounding serving system.

KV-cache capacity. A heavier drafter can leave too little memory for request KV caches. In one configuration we measured, a larger block drafter cut the available KV-cache tokens enough to limit the server's max supported users, which cancelled the per-user speed advantage.

Step time. In another measurement, even a block drafter’s lighter operating mode had a noticeably longer step than the model’s native MTP path, while acceptance declined at long context. A higher acceptance rate is not useful if obtaining it costs too much.

Integration and caching. Block size and KV-cache chunk alignment constrain how effectively external caching is used. On long-prompt agentic workloads, reduced prefix-cache capacity can cost more than speculation gains.

The model’s native MTP layer can also come out ahead. In one automated engine configuration search (a sweep over serving-engine settings such as scheduler behaviour and parallelism strategy), tuning the native MTP layer was the single largest improvement found.

The drafter is one lever among several

On a two-GPU H100 node serving an agentic workload, we compared two parallelism layouts with two drafting strategies.

Figure 5 · Relative decode throughput by parallelism layout and drafter. Indexed to the best configuration. Tensor parallelism beat data parallelism by 9–10% with either drafter; the block drafter beat MTP by 3–4% under either layout. The layout decision was worth more than the drafter decision. The DP engines also saw lower prefix-cache hit rates and each carried its own scheduling overhead.

Tensor parallelism will not win on every workload. Evaluate parallelism and speculation together, because each changes the step cost the other experiences.

The cheapest win may be a faster step

Every output token benefits when the underlying decode step gets cheaper. One single-stream step-time breakdown on a long-context request split as follows.

Figure 6 · Where the decode step goes. Plain decode about half, long-context attention about a quarter, speculation overhead under a fifth, the rest unattributed. Cutting step cost speeds up every token, accepted or not.

Work backwards from the end-to-end target

Given an end-to-end target, Amdahl’s law tells you how much faster decode has to get. Decode is only part of request time. If it accounts for a fraction f of end-to-end latency and gets x times faster, the overall speedup is:

G = 1 / [(1 − f) + f / x]

For one measured agentic workload, decode was about 83% of request latency. To reach a 1.5× end-to-end improvement, decode must become about 1.67× faster. A 1.67× decode speedup equals a 1.67× MAL gain only when step cost stays constant, and step cost changes with proposal length and concurrency.

Figure 7 · Theoretical end-to-end speedup under Amdahl’s law. Hover or tap a curve. A workload that spends more time in decode has more room for a decode optimisation to help. For the 83%-decode example, eliminating decode entirely would cap end-to-end speedup at about 5.9×.

End-to-end target	Decode = 70%	Decode = 83%	Decode = 93%
1.2×	1.31×	1.25×	1.22×
1.5×	1.91×	1.67×	1.56×
2.0×	3.50×	2.52×	2.16×

Required decode speedup to reach each end-to-end target. Real serving systems face additional constraints, including prefill contention and resource sharing.

Measure the ceiling before training a drafter

Open serving engines expose synthetic acceptance controls that allow experiments with a specified acceptance rate, without a trained drafter that achieves it. Used carefully, they estimate how much headroom speculation has on the real workload and serving stack. The practical questions are how much end-to-end performance is available to capture, and how much of that ceiling a realistic drafter can deliver. If a perfect drafter buys only a modest aggregate improvement, the opportunity cost of training, integrating and operating a more complex drafter may be too high.

What this means for inference engineering

In production, the questions are where speculative decoding pays off and at what cost.

Measure the baseline at real concurrency.Record aggregate throughput, per-user decode speed, inter-token latency, time to first token and KV-cache capacity.
Sweep proposal length.Evaluate several values of K at the concurrency levels that matter, using paired runs and the same request set.
Account for the entire step.Measure drafting, verification, attention, memory capacity, scheduler overhead and cache behaviour.
Estimate the ceiling.Use controlled synthetic-acceptance experiments to find how much headroom speculation has on the actual workload.
Optimise the whole configuration.Compare drafter, parallelism layout, caching strategy and decode implementation together.
Validate the service objective.Check end-to-end latency and tail behaviour, not just mean acceptance length or peak tokens per second.

The most useful drafter delivers the best end-to-end result within the system’s memory, concurrency and latency constraints, and that may not be the drafter that accepts the most tokens.

Run the head-to-head on your workload.

Share the traffic shape, models in use and latency target. PrimaLabs measures throughput, latency and cost side by side and returns the numbers in writing.

Book a demo

Frequently asked questions

Why does per-user speed go up while aggregate throughput goes down?

Per-user speed counts tokens emitted per unit of wall-clock time for one request; the drafter emits several per step, so each user sees faster output. Aggregate throughput counts all tokens across the batch per unit of GPU time. When the step grows slower for everyone to produce those extra tokens, the batch as a whole can emit fewer tokens per second than it did without speculation.

Is a longer draft always better?

No. Longer drafts raise acceptance length but also step cost, and the rejected tail is wasted work. The best K depends on the overhead at the concurrency you serve; Figure 3 shows how it moves.

Does this mean speculative decoding is not worth it?

It means it is worth it at a specific operating point, which you have to measure. On the workload above it was a clear win at concurrency 10 and a loss at 16; the right deployment keeps it where it pays.

Methodology. The measured crossover used an off-the-shelf block drafter at K = 7 on a frontier open-weight MoE model served on a B200 node, replaying a long-prompt agentic trace; speculative and baseline runs replayed the same conversations at each concurrency level. Ratios are reported rather than absolute throughput. Total tokens per second include cached prompt tokens. The parallelism comparison used a two-GPU H100 node on an agentic workload at K = 3 and is reported relative to the best configuration. The step-time breakdown is a single-stream measurement reported as shares. Figures 2 and 3 are illustrative models, not measurements; Figure 7 follows Amdahl’s law. Results compare the tested configurations and do not establish performance on every workload.

Speculative decoding under load.

TL;DR