Engineering · Inference systems

A comparison of host cache tiers for hybrid-model serving.

We find that draft-token acceptance in production ranged from 0.04% to about 80% on the same model, depending on the host cache tier.
A benchmark study of LMCache, SGLang HiCache and vLLM native CPU offload in production configurations of Qwen3.8-27B and Nemotron-3.5-Lightning.
0.04%vs80%
speculative-decoding acceptance with LMCache and with vLLM’s native offload
21×
lower time to first token on SGLang from changing one memory flag
94–95%
cache hit rate with vLLM’s native tier from 1 to 32 concurrent requests

TL;DR

  • The host tier determines whether speculative decoding works. On Nemotron-3.5-Lightning, multi-token-prediction acceptance fell to 0.04% with LMCache attached and stayed at about 80% with vLLM’s native CPU offload. Patching the rollback in the engine did not fix the LMCache case.
  • Tuning mattered more than the engine. The memory split in SGLang’s recipe for Qwen3.8 left the attention cache short for our long prompts, and changing one flag cut time to first token 21×. With both engines tuned, SGLang gave a 47% lower time to first token and vLLM with LMCache decoded each request 11% faster.
  • Features draw from the same memory. On SGLang, the EAGLE drafter took 10% of the attention cache and halved the requests that could run at once.
  • With vLLM’s native tier, the hit rate held at 94–95% up to 32 concurrent requests, as long as a session returns to the same engine.

With only the GPU prefix cache, our Nemotron engine accepted about 80% of its draft tokens. With LMCache attached as a host tier, it accepted 278 of 718,845. We ran three host tiers on hybrid models: vLLM with LMCache, SGLang HiCache and vLLM’s native CPU offload.

01

Three host tiers

A host tier keeps KV cache in CPU memory, so a returning prefix does not have to be recomputed. On a hybrid model, a resumable prefix needs the recurrent state as well as the attention entries, and each tier handles that differently. Our companion post covers the scheduler limit on the LMCache versions we first ran. The vLLM with LMCache runs on Qwen3.8 here use the raised 16,384-token step budget.

vLLM with LMCache. Several replicas share one cache server. We tested that directly: replica B retrieved a prefix that only replica A had ever prefilled. Eight replicas share one host-memory tier instead of having eight. The cost is version sensitivity. The alignment rule, the chunk rule and the memory layout all changed between releases. Our MoE hybrid and dense hybrid needed opposite cache-server alignment flags. On the releases we ran, engines attached to the shared server sometimes stopped admitting work under sustained load. The KV cache sat almost full with nothing running, and only restarting the engine cleared it. We did not confirm the connector as the cause, but test it at production concurrency before relying on it.

SGLang HiCache. SGLang’s hierarchical cache for hybrids (HiMambaRadixCache) keeps two host pools, one for attention entries and one for recurrent states. On Qwen3.8 it needed --hicache-mem-layout page_first_direct, because the default layout stops the recurrent pool from starting. The configured host size applies to each pool, so 256 GB reserves 512 GB of address space. Actual use was about 28 GB.

The setting that mattered most was the GPU memory split between recurrent states and attention entries. SGLang’s published recipe for this model sets --mamba-full-memory-ratio 4.59. That reserved 42 GB for recurrent states that averaged about 1% use and left 9 GB for attention. With four concurrent requests and only this flag changed:

Ratio 4.59Ratio 1.87
Attention cache capacity302,657 tokens589,937 tokens
Mean time to first token31.6 s1.47 s
Decode throughput48 tok/s96 tok/s
Cache hit rate94.9%94.7%

The hit rate did not change. Requests had been waiting for attention-cache space. At a ratio of 0.5, the engine took eight concurrent requests before queueing, against four at 1.87.

We then compared SGLang at ratio 0.5 with vLLM and LMCache at the 16k budget. Neither engine won outright. SGLang’s time to first token was 47% lower, and vLLM decoded each request about 11% faster. Section 02 sets the two side by side.

vLLM native CPU offload. vLLM’s built-in OffloadingConnector keeps a host-memory tier inside each engine process. It needs no separate server and has no chunk-size rule. It also shares nothing across replicas. We moved to it because of speculative decoding.

02

SGLang HiCache against vLLM with LMCache, on identical traffic

Held equal. Qwen3.8-27B-FP8 on one RTX PRO 6000, one engine each with no tensor parallelism. The same agentic trace dataset, the same entries and seed, and the same maximum context length. AIPerf sent load directly to each engine, with no gateway in between. On the SGLang side, HiMambaRadixCache kept a host pool for attention entries and one for recurrent states. On the vLLM side, the engine attached to an LMCache server.

An earlier four-request run still had vLLM’s one-block step budget, so it measured that limit rather than the engines, and we leave it out here.

Eight concurrent requests: both engines tuned

vLLM at a budget of 16,384; SGLang at a memory ratio of 0.5.

vLLM + LMCacheSGLang HiMambaRadixCacheDifference
Requests decoding at once7.68 of 87.93 of 8about equal
Mean time to first token3.16 s1.69 sSGLang 47% lower
Prefill throughput14,007 tok/s14,788 tok/sSGLang +6%
Decode throughput130 tok/s121 tok/svLLM +8%
Decode per request (decode throughput ÷ requests decoding)16.9 tok/s15.2 tok/svLLM +11%
Prefill plus decode, per GPU14,137 tok/s14,909 tok/sSGLang +5%
Cache hit rate93.5%91.5%vLLM 2 points higher

The per-request row uses one definition for both engines. AIPerf’s own per-user figure for vLLM was 22.0 tok/s. That figure runs higher than this division on every run where we have both, so comparing it with SGLang’s 15.2 would overstate the gap.

Not held equal. Three differences remain:

  • Run length. vLLM ran for 2,400 s and SGLang for 1,200 s. Rates are comparable; request counts are not.
  • Output length. 17.8% of vLLM’s requests stopped early at an end-of-sequence token. SGLang’s workload forced every request to its full output length.
  • Cache hit rate. SGLang’s run found 2 points less in cache, so it computed about 31% more of each prompt: about 7,000 tokens a request against 5,350. Prompts in these runs averaged about 82,500 tokens. Its prefill and first-token wins came despite that extra work.

Neither engine won on every line. SGLang gave the faster first token on more uncached work. vLLM decoded each running request about 11% faster and produced 8% more decode tokens on the same GPU. We did not measure SGLang across two GPUs, the setup where vLLM reached the highest per-request speed in our tests.

03

On SGLang, speculative decoding halved the requests that could run

SGLang’s EAGLE drafter worked on Qwen3.8. It accepted 2.78 tokens per verification step out of a possible 4, and cut the time between tokens from 43.0 ms to 36.4 ms. We still turned it off. With four concurrent requests and the memory ratio at 1.87:

No speculative decodingEAGLE
Attention cache capacity589,937 tokens530,141 tokens
Request limit11546
Requests decoding at once3.84 of 42.07 of 4
Decode throughput96 tok/s73 tok/s
Mean time to first token1.47 s19.6 s

The drop in concurrency was the reason. Without EAGLE, 3.84 of the 4 offered requests were decoding at any moment; with EAGLE, 2.07 were. The other two waited in the queue, so mean time to first token rose from 1.47 s to 19.6 s, 13× longer.

The cause was memory: EAGLE took 10% of the attention cache and cut the engine’s request limit from 115 to 46. A limit of 46 is still far above 4, so the limit alone did not hold requests back. Section 01 showed that attention-cache space was already the constraint on this model, where each request carries about 78,000 tokens of context. We did not isolate how much of the drop each memory change caused.

Each running request spent about 15% less time between tokens, but only about half as many requests could run. Total decode throughput fell by a quarter, from 96 to 73 tok/s.

SGLang’s prefill chunk size did not matter the way vLLM’s step budget did. Raising it from 2,048 to 16,384 tokens changed neither decode nor prefill throughput. At a 94.7% hit rate, a 78,000-token prompt leaves about 4,100 tokens to compute, so the chunk size is rarely the limit. vLLM’s one-block step budget, covered in the companion post, hurt because it blocked prefill whenever a decode token shared the step.

04

With LMCache attached, speculative decoding stopped working on Nemotron

Nemotron-3.5-Lightning-30B-A3B is a Mamba-2 and mixture-of-experts hybrid with a built-in multi-token prediction (MTP) head. Without a host tier, the engine accepted about 80% of draft tokens. With LMCache attached, it accepted 0.04%: 278 of 718,845.

When a draft token is rejected, the engine has to undo its effect on both the attention entries and the recurrent state. Our first reading, following vLLM issue #53505, was that the LMCache connector rolled back only the attention entries and left the two out of step. Section 05 rules that out: acceptance was near zero before LMCache had served anything from cache. The same engine with the same MTP head kept about 80% acceptance under vLLM’s native offload, so on this model the failure follows the connector.

On one B200 with NVFP4 weights and three draft tokens:

Host tierMTP acceptanceDecode
None (GPU prefix cache only)about 80%885 tok/s
LMCache0.04%not benchmarked
vLLM native CPU offload78.9–80.4% from 1 to 32 concurrent requests839 tok/s

The native tier cost about 5% of decode in these runs and needed one engine fix, described in section 05. On a low-reuse coding corpus, the GPU cache alone served 5.6% of prompt tokens. The host tier served 39.3% of the tokens the GPU missed, for 43% in total. Section 06 shows how it held on high-reuse agentic traces.

A separate draft model avoids the problem. With DSpark, which keeps its own cache and never writes to the main model’s state, acceptance with LMCache was 72%.

0%25%50%75%100% MTP, no host tier 79.9% MTP + LMCache 0.04% MTP + native CPU offload 78.9–80.4% DSpark + LMCache separate drafter 72%
Figure 1 · Draft acceptance by host tier, Nemotron-3.5-Lightning on one B200. NVFP4 weights, three draft tokens. The native-offload bar spans the range measured from 1 to 32 concurrent requests.
05

Patching the rollback did not fix LMCache

We tried to fix the LMCache case in the engine before switching tiers. Each row adds one change to the same Nemotron NVFP4 engine with three MTP draft tokens:

ConfigurationMTP acceptance
LMCache, stock vLLM 0.30.00.04%
+ backported guard on the Mamba state copy0.18%
+ vLLM #48375, which makes the Mamba cache manager honour dropped draft blocks0.27%
Both patches, with vLLM native CPU offload instead of LMCacheabout 80%

Neither patch moved acceptance off zero with LMCache attached. The same patches with the native tier kept it at 80%. The upstream fix was validated on that setup. The full scheduler fix, vLLM #54165, was still open when we ran this.

Acceptance was near zero from the start of the run, before the tier had served much from cache. The vLLM fixes deal with reusing a cached prefix after a rejected draft. Our failure appeared once the connector was attached, whether or not it had retrieved anything, so the vLLM fixes we tested do not address it.

Three practical notes from the switch:

  • The backports are no longer needed. We dropped them when we moved to vLLM 0.31.0. Acceptance did not change, and decode speed on the 10k-token workload matched within noise.
  • The drafter changes the chunk rule. With DSpark instead of MTP, LMCache’s chunk size had to go from 2,192 to 8,768 tokens. The chunk must be a multiple of the engine’s block, so read the block from the startup log after every change to the drafter.
  • The gateway adds to time to first token only. Through our gateway, the native-offload engine decoded at 816 tok/s against 839 direct in short runs, a gap inside run-to-run noise. Time to first token rose by 0.47 s.
06

The hit rate held under load

The 43% total in section 04 came from a low-reuse coding corpus, so it reflects how little that corpus repeats. To test the native tier at production concurrency, we replayed agentic traces with subagents through our gateway, 900 seconds per level. These traces have about 97% theoretical prefix reuse. The engine was Nemotron NVFP4 with MTP and a native host tier on one B200:

Concurrency offeredAchievedDecode tok/s per GPUAIPerf active decode per user, tok/sMean time to first tokenHit rate, achieved / theoretical
10.863915930.70 s95.2% / 97.0%
86.881,2572101.11 s95.2% / 97.1%
1614.01,7691411.10 s95.0% / 97.2%
3228.41,918821.46 s94.3% / 97.0%

Decode throughput rose 4.9× from one to 32 concurrent requests, and the hit rate fell by less than one point. The per-user column is AIPerf’s active decode rate, counted only while a request is decoding. That is why it exceeds the per-GPU figure at one request, where the GPU sits idle between turns. At one concurrent request, the engine’s counters show the GPU cache serving 89.6% of prompt tokens by itself. The host tier recovered 47.5% of what it missed (2.47M of 5.19M tokens), against 39.3% on the low-reuse corpus. Together that is 94.5%, slightly under AIPerf’s 95.2% in the table.

05001,0001,5002,000 0100200300400500600 AIPerf active decode per user, tok/s Decode tok/s per GPU c=32 · 94.3% hit · 1.46 s c=16 · 95.0% hit · 1.10 s c=8 · 95.2% hit · 1.11 s c=1 · 95.2% hit · 0.70 s
Figure 2 · Throughput against per-user speed with a native host tier. Nemotron-3.5-Lightning NVFP4 with MTP and a vLLM native CPU tier on one B200, agentic traces through our gateway. Labels give the achieved hit rate and mean time to first token at each point.

The tier lives inside each engine, so it only helps when a session’s next turn returns to the same engine. Through the gateway, our cache-correctness suite could not confirm reuse, because a replayed request could land on a different engine and find nothing cached. Routing that returns a session to the same engine matters for every per-engine tier.

07

What this means for inference engineering

  1. Test speculative decoding with the cache tier attached. Acceptance measured without the tier does not predict acceptance with it.
  2. Give memory to the pool that fills. On SGLang, the recipe’s split favoured a pool we barely used.
  3. Check which memory pool a feature draws from before turning it on. On SGLang, EAGLE halved the requests running at once.
  4. Pair a per-engine host tier with sticky routing. A returning session that lands on another engine finds nothing cached.
  5. Check the cache connector’s rules on the versions in use. A recipe records the constraint that held when it was written.

Run the head-to-head on your workload.

Share the traffic shape, the model and the latency target. PrimaLabs measures the caching and speculative-decoding options on your traffic and returns the numbers in writing.

Book a demo

Frequently asked questions

Why not turn the host tier off?

On Nemotron, the native tier kept acceptance at about 80% and cost about 5% of decode. It served 39.3% of the tokens the GPU cache missed on a low-reuse corpus and 47.5% on agentic traces. With LMCache attached, acceptance fell to 0.04%, and the GPU prefix cache alone kept about 80%.

Which tier should I start with?

For speculative decoding on a Mamba hybrid, vLLM native offload. For one cache shared across many replicas, LMCache, tested at production concurrency. For the lowest time to first token on one engine, SGLang HiCache with a tuned memory split.

Does speculative decoding work with a host tier on a hybrid model?

It depended on the tier. On Nemotron-3.5-Lightning, MTP acceptance was about 80% with vLLM’s native CPU offload and 0.04% with LMCache attached. A separate draft model, DSpark, kept 72% acceptance with LMCache.

Is the memory split in SGLang’s Qwen3.8 recipe right for long prompts?

It was not for ours. At the recipe’s ratio of 4.59, 9 GB was left for attention entries and mean time to first token was 31.6 s. At 1.87 it was 1.47 s with about the same hit rate.

Methodology. Qwen3.8-27B-FP8 ran on one RTX PRO 6000 Blackwell (96 GB) on SGLang 0.5.14 with HiCache, and on vLLM 0.26.0–0.27.1 with LMCache 0.5.4 at a 16,384-token step budget. Nemotron-3.5-Lightning-30B-A3B (NVFP4) ran on one B200 on vLLM 0.30.0 with backported fixes, then on 0.31.0 without them. Load came from AIPerf with agentic traces, from guidellm, and from workloads shaped like Artificial Analysis’s for decode speed. Qwen prompts averaged about 78k tokens, and about 82.5k in the section 02 eight-request runs, with about 95% theoretical prefix reuse. The section 06 sweep replayed agentic traces with subagents through our gateway, 900 s per concurrency level, with about 97% theoretical reuse. Tables compare runs with the same dataset and seed. Offered concurrency is stated where it differs. Per-request decode speed in section 02 is decode throughput divided by the number of requests decoding, so both engines use the same definition. AIPerf’s own per-user figure runs higher than this division, so we don’t mix the two. The exception is the single-engine sweep in section 06, which reports AIPerf’s active decode per user and labels it as such. Engine figures come from vLLM and SGLang Prometheus metrics.

What this does not establish. Each cell is one run, so treat a few percent as noise. The single-stream decode figures in sections 04 and 05 come from short runs of the 10k-token workload. The gateway figure is from four requests. Later 24-request runs through the gateway measured 751 to 772 tok/s. The SGLang and vLLM comparisons are single-GPU and do not test cross-replica sharing. On SGLang, the EAGLE run changed two memory settings at once, and we did not isolate how much of the drop each caused. The host tier’s 47.5% recovery figure is from the one-request run only. Through the gateway, our cache-correctness suite could not confirm reuse, so the section 06 hit rates depend on a session returning to the same engine. The LMCache problems we describe were on specific releases, and several rules have changed since. This is two models on two GPU types, so re-measure before reusing any number here.

Acknowledgements. The vLLM with LMCache runs in section 02 are Thomas Randall’s work.

References.

Application-specific adaptive inference. Owned, sovereign, and measured in writing.
PrimaLabs Engineering · 20 Nov 2026