Effect of scheduler step size on concurrency in hybrid-model serving.
TL;DR
- On a hybrid model, a cached prefix can only be resumed where the recurrent state was saved. On the LMCache versions we first ran, that rule limited each scheduler step to one cache block, about 1,600 tokens. Each engine saturated early, and the only way to scale was more replicas.
- Once LMCache 0.5.4 removed the limit, raising the step budget gave 3.1× the decode throughput and at least a 14.4× lower time to first token on the same GPU.
- Under the limit, the KV cache was 1.6–16% full and the GPU drew 332 W of its 600 W limit while requests queued. On a hybrid, check how many tokens each scheduler step carries before adding GPUs.
On our first tuned hybrid deployment, 94% of prompt tokens came from cache. Requests still waited an average of 45.5 seconds for their first token, because the engine was running one request at a time. That average leaves out the requests that timed out.
Hybrid models break the usual prefix cache
A full-attention layer keeps one key/value entry per token. If two requests share their first 10,000 tokens, the second reuses those entries and computes only the rest.
A Gated DeltaNet (GDN) or Mamba layer keeps a fixed-size recurrent state that summarises everything it has read. Skipping a 10,000-token prefix needs that state exactly as it was at token 10,000. There is no per-token history to slice, so the recurrent layers can reuse a prefix only where a saved state exists.
Hybrid models such as Qwen3.6, Qwen3.8 and Nemotron-3.5 interleave a few attention layers with many recurrent ones. vLLM stores their cache in blocks. On a hybrid, it enlarges the block so that one block of attention entries takes the same memory as one saved recurrent state. That is why blocks on these models are 1,500 tokens or more rather than the usual 16. In align mode, vLLM saves the recurrent state at the last token of a scheduler step when that token falls on a block boundary. A prefix can be resumed up to the last boundary where a state was saved.
A state exists only where a prefill step ended. If one step processes 8,192 tokens, the state at token 1,600 is never saved. So the scheduler’s step size limits what the cache can reuse.
One block per step
We served Qwen3.8-27B and Qwen3.6-35B-A3B on RTX PRO 6000 Blackwell GPUs, one replica per GPU, all attached to a shared LMCache server.
The rule that mattered came from LMCache 0.5.2. It required each step’s token budget, --max-num-batched-tokens, to be at least one block and less than two. Every step then advances exactly one block and leaves a saved state behind. Qwen3.8’s block was 1,600 tokens with its MTP head enabled (1,568 without), so vLLM’s default budget of 8,192 failed at startup. Three smaller fixes were also needed. We forced prefix caching on, because vLLM disables it when a cache connector is attached and LMCache then rejects the recurrent layers. We set the cache’s chunk size to a multiple of the block. We also removed a compile config, copied from a DeepSeek recipe, that crashed on the GDN layers.
With the budget held to one block:
- Long prompts prefill serially. An 8k prompt on Qwen3.8 takes ⌈8,192 ÷ 1,600⌉ = 6 steps, one after another.
- Each engine saturates early. On Qwen3.6 the block was 2,144 tokens, so the legal budget was 2,144 to just under 4,288 tokens. Each engine topped out around 16 concurrent requests. The two alternatives we tried,
mamba_cache_mode=alland a larger block, both failed. - Scaling means replicas, which moves the problem to the router. Our gateway sends requests with a shared prefix to the same replica unless replica loads differ by more than a set threshold. At our first threshold, bursts piled onto one replica while three sat idle. At a quarter of that, throughput scaled roughly linearly with replicas and P99 time to first token fell about 10×.
- Only whole chunks are reusable, so on Qwen3.8 a prompt shorter than 1,600 tokens got nothing from the shared cache.
The limit starved the scheduler
We expected a modest prefill penalty. We ran a diagnostic on Qwen3.8 with no speculative decoding: 95 agentic requests of about 78,000 tokens each, five concurrent clients and a step budget of one block (1,568 tokens). The engine’s metrics showed:
| Signal | Measured |
|---|---|
| Scheduler steps carrying one token or fewer | 88.1% of 47,265 (mean 19.8 of 1,568) |
| Requests running and waiting | 1–2 running, up to 4 waiting with reason “capacity”, 0 preemptions |
| KV cache in use | 1.6–16% of 1.63 M tokens |
| Time to first token | 40.0 s mean: about 34 s waiting in the queue, 6 s after it |
| GPU power | 332 W of 600 W |
The engine had KV room for about 20 such requests. The waiting requests were held back by how align mode schedules prefill. vLLM rounds each prefill chunk down to whole blocks. If less than a block of budget is left in the step, the request is skipped. The scheduler source lists this case as “insufficient budget for a block-aligned chunk”. With the budget equal to one block, a single decode token from a running request leaves less than a block free, so no prefill can run in that step. Most steps carried one decode token and nothing else.
On a full-attention model, an empty KV cache with a queue usually points at the router or a request limit. On a hybrid, check how many tokens each scheduler step carries.
The rule changed, and our recipe did not
LMCache 0.5.4 dropped the upper bound. Its alignment check now fails only when the budget is below one block. Its documentation says larger steps are handled safely when the cache server runs with --separate-object-groups. The limit came from a rule in the cache connector. Hybrid models do not require it.
Our first benchmark on 0.5.4 still used 1,568, because that number lived in the recipe. We then raised the budget to 16,384 and changed nothing else on the engine. On one RTX PRO 6000, in AIPerf runs with the same agentic traces:
See the numbers as a table
| Budget 1,568 | Budget 16,384 | Change | |
|---|---|---|---|
| Concurrent requests offered | 4 | 8 | |
| Concurrent requests achieved | 1.0 | 7.68 | |
| Decode throughput | 42 tok/s | 130 tok/s | 3.1× |
| Mean time to first token | 45.5 s* | 3.16 s | at least 14.4× lower |
| Cache hit rate | 94.4% | 93.5% |
* Leaves out 4 timed-out requests, so the real figure is higher.
The engine with the one-block budget ran one of the four requests it was offered, so offering eight would not have helped it. Four of its 70 requests timed out and are not in its averages. Its real mean time to first token was higher than 45.5 s, so the 14.4× improvement is a lower bound.
Larger steps have a cost. A state is saved only at the end of a step, so a 16k step leaves fewer places where a later request can resume. On this agentic traffic, where each turn’s prefix is the previous conversation, the hit rate moved by about one point. Traffic with many short shared prefixes may lose more, so measure it.
What this means for inference engineering
- Read the scheduler before adding GPUs. A mostly empty KV cache, a queue and a GPU below its power limit mean the engine has spare room and nothing scheduled into it.
- Check the cache connector’s rules on the versions in use. A recipe records the constraint that held when it was written.
- Fix the router before adding replicas. Packing bursts onto one replica wastes the others.
- Re-measure the hit rate after raising the step budget. Larger steps leave fewer places to resume. On our agentic traffic the hit rate moved by about one point, and traffic with many short shared prefixes may lose more.
Once the step budget was fixed, the next question was which host tier to run, and what it does to speculative decoding on a hybrid model. That is the subject of a companion post.
Run the head-to-head on your workload.
Share the traffic shape, the model and the latency target. PrimaLabs measures the caching and speculative-decoding options on your traffic and returns the numbers in writing.
Book a demoFrequently asked questions
Does this affect every hybrid model?
Any model with recurrent layers needs saved states to reuse a prefix. Whether that limits the batch size depends on the engine and the cache connector version.
Is the block size a property of the model?
Partly. vLLM sizes it so attention entries and recurrent states line up in memory, so it depends on the model, the engine version and the options. On Qwen3.8 it changed from 1,568 to 1,600 tokens when we enabled the MTP head. Read it from the engine’s startup log.
Does LMCache still have this limit?
Not since 0.5.4. Its alignment check now fails only when the budget is below one block. Check the rule on the version in use, because a recipe records the constraint that held when it was written.
What are the signs that an engine is hitting this limit?
Look at how many tokens each scheduler step carries. In our diagnostic, 88.1% of steps carried one token or fewer. The KV cache was 1.6–16% full, and up to four requests waited with reason “capacity” while the GPU drew 332 W of 600 W.
Methodology. Qwen3.8-27B-FP8 and Qwen3.6-35B-A3B-FP8 ran on RTX PRO 6000 Blackwell (96 GB), one replica per GPU, on vLLM 0.26.0–0.27.1 with LMCache 0.5.2–0.5.4. Load came from AIPerf with agentic traces. Prompts averaged about 78k tokens, and about 82.5k in the eight-request run at the 16,384 budget, with about 95% theoretical prefix reuse. Tables compare runs with the same dataset and seed. Offered concurrency is stated where it differs. Engine figures come from vLLM Prometheus metrics.
What this does not establish. Each cell is one run, so treat a few percent as noise. The 1,568 run lost 4 of its 70 requests to timeouts, so its averages understate the wait. The two budgets were offered different concurrency, 4 and 8, but the engine with the one-block budget ran only one request at a time. The hit-rate cost of larger steps was measured on agentic traffic only. Traffic with many short shared prefixes may lose more. The LMCache rule we describe was on specific releases and has changed since. This is two models on one GPU type, so re-measure before reusing any number here.
Acknowledgements. The step-budget diagnosis and the 1,568 to 16,384 benchmarks in sections 03 and 04 are Thomas Randall’s work.
References.
- vLLM documentation: cache configuration (
mamba_cache_mode) and automatic prefix caching - “Hybrid models as first-class citizens in vLLM”, PyTorch blog, November 2025
- Marconi: prefix caching for the era of hybrid LLMs, arXiv:2411.19379