Engineering · Inference systems

The TP step‑time model.

Split the weights and inherit a synchronisation that never shrinks.

A production study of weight sharding, collective cost, and tokens per dollar on Nemotron-3.5-Super-VL.

Dipanjan Sengupta and Thomas Randall · PrimaLabs Engineering · 5 Nov 2026 · 8 min read

0.92×

Output speed, TP=2 against TP=1, single stream at 1k input

1.32×

Output speed, TP=4 against TP=1, same model and same checkpoint

3.19

Tokens per step, unchanged at every parallelism level

TL;DR

The same model and checkpoint ran 8% slower on two B200s than on one, and 32% faster on four. Tokens per step did not move.
Sharding cuts the weight read, but never by more than (1 − 1/n) of the step. Communication adds a cost that barely shrinks. The sweet spot is where the two cross, and on this model it sits just above TP=2.
Parallelism is one lever among several. Work out the ceiling before you rent the GPU, benchmark at the concurrency you serve, and decide whether you are optimising latency or tokens per dollar, because they point to opposite ends of the range.

Adding a GPU is supposed to make a model faster. On Nemotron-3.5-Super-VL, going from one B200 to two made single-stream decoding slower, from 522 tokens per second to 482. Four brought it back up, to 689.

The dip is reproducible. Tensor parallelism (TP) does two things to every decode step, in opposite directions, and the two scale differently with the number of GPUs. Knowing which one is winning at a given rank count tells you where the sweet spot is, and whether the second GPU is worth renting.

The cost of splitting

With one request at a time, each decode step handles about three tokens (that's speculative decoding at work). That is very little compute for a model this size, so the GPU spends most of the step reading weights out of memory. Step time ends up being roughly the time it takes to read the weights once.

Splitting the model across GPUs shrinks that read: with n GPUs, each one reads 1/n of the weights. The catch is that the GPUs have to swap results at every layer before they can move on, and that exchange takes about the same time whether the step is big or small.

Switch between one, two and four GPUs below. The amber block is the share of weights each GPU reads; the red line under the cards is the exchange between them.

Tensor parallel

Figure 1 · One decode step, sharded. Each card is one GPU. The weight block shrinks as 1/n; the collective beneath it appears the moment there is more than one rank and then stays in the same few-millisecond range. The timeline shows the two stacked against the dashed single-GPU step.

Weight saved against communication added

Put the two forces on one axis. Below zero is the weight read that sharding removes from the step. Above zero is the communication it adds back. The diamond is the net effect on step time.

Figure 2 · Opposing forces, and the net. At TP=2 the red bar is taller than the green one, so the net lands above zero and the step gets longer. At TP=4 the green bar has grown while the red one has not, so the net falls below zero. Step times are measured; the communication figures are derived from them by subtracting the weight saving.

At TP=2 the two forces nearly cancel. On this model, communication wins by 0.71 ms.

The engine's own counters show nothing else in play. Tokens per step holds at 3.13 to 3.23 across all three configurations: speculative acceptance depends on the content being generated, and sharding the model does not change what the drafter proposes. Every difference in output speed traces back to the step.

Server-side decode counters on a realistic prompt, B200.
Metric	TP=1	TP=2	TP=4
Decode step time	7.42 ms	8.13 ms	5.17 ms
Tokens per step (MTP)	3.19	3.13	3.23
Raw decode	431 tok/s	385 tok/s	625 tok/s

The break-even point

Put the two costs side by side and you get a simple test. Splitting across n GPUs only helps if the weight-read time it saves is bigger than the time the GPUs spend talking to each other:

(1 − 1/n) · T > C(n)T = single-GPU step time (7.42 ms) · C = per-step collective cost

The saving has a hard limit, and it grows slowly as you add GPUs. With two GPUs you can save at most half the step, 3.71 ms here. If the exchange between them costs close to 4 ms, two GPUs will never beat one, however well you tune them.

Drag the slider to change the communication cost and see which setups come out ahead.

Comm cost C

Figure 3 · The crossover. Each bar is the most that setup can save; the dashed line is the communication cost. A setup only beats one GPU if its bar clears the line. When communication is cheap, even two GPUs come out ahead. At 4.4 ms, which is roughly what we measured, two GPUs fall short and four clear it.

The sweet spot moves with the load

Everything so far is a batch-1 story, and batch 1 is the regime where the model is read once and barely used. Add concurrent requests and both forces change.

At ten concurrent streams the step carries roughly ten times the tokens. The weights are still read once, but now amortised across the whole batch, and the matmuls are large enough that the step becomes compute-bound rather than bandwidth-bound. Sharding now splits real arithmetic instead of trimming a memory read, so its benefit grows with the batch rather than hitting a ceiling. Meanwhile the communication cost, roughly fixed per step, is spread over ten times as much useful work.

Both forces move the same way, so the ordering changes. At c=10 every TP level helps, including the one that was a loss at batch 1: TP=2 goes from 0.92× to 1.32× against TP=1, and TP=4 reaches 1.69×.

A TP choice made on single-stream benchmarks is the wrong choice for a loaded server, and the reverse: the configuration that looks worst in a demo is not the one that looks worst in production.

The price of speed

So far we have only looked at speed. Cost tells a different story. On rounded provider pricing, two GPUs cost about 2.1× as much per hour as one, and four cost about 6×. Four costs more than four times one because providers price whole nodes, not individual cards.

Compare each setup against one GPU on both speed and tokens per dollar, and the best choice flips.

Workload

Figure 4 · Speed against efficiency. Both panels compare against one GPU (1.00) for the selected workload. On every workload, the fastest setup is also the least cost-efficient.

Output speed from Artificial Analysis's prompt set, their metric definition. Tokens per dollar is speed divided by rounded hourly cost, indexed to TP=1.
Workload	TP2 speed	TP4 speed	TP2 tok/$	TP4 tok/$
1k input, c1	0.92×	1.32×	0.44×	0.22×
10k input, c1	0.85×	1.30×	0.41×	0.22×
100k input, c1	1.07×	1.24×	0.51×	0.21×
1k input, c10	1.32×	1.69×	0.63×	0.28×

One GPU gives the most tokens per dollar on every workload, four to five times more than four GPUs. That holds whether you count tokens per stream or in total, since every setup ran at the same concurrency.

Four GPUs make sense when speed is what you are selling: an interactive agent where someone is watching the tokens come in, or a contract with a latency target. For batch work, where cost per token is what counts, paying 6× for 1.3× the speed is a bad deal.

Two GPUs are hard to justify at all. With one request at a time they are slower than one GPU and less than half as cost-efficient. Nothing in this data makes them the right pick.

What this means for inference engineering

The instinct when a model is too slow is to give it more hardware. That instinct is right often enough to be dangerous, because the cases where it fails look the same from the outside: same model, same engine, more GPUs, worse numbers. Working out the ceiling first costs an afternoon with a calculator.

Start at TP=1 and make the case for leaving. It wins tokens per dollar on every workload here. The burden of proof belongs to the larger configuration.
Work out the ceiling before you rent anything. The most TP=n can save is (1 − 1/n) of your single-GPU step. If that is close to your collective latency, the configuration cannot win, and you know it without running it.
Do not assume the curve is monotonic. It dips here. You cannot hill-climb TP by trying the next rank up and stopping when it gets worse; TP=2 would have told you to stop one step before the configuration that wins.
Benchmark the regime you serve. Single-stream numbers get published because they are easy to run and they flatter. They are also where TP looks worst, since there is no batch to amortise the collective over. A single-stream sweep points a loaded server at the wrong configuration, and a concurrency sweep does the same to an interactive product.
Measure step time, not just output speed. With speculative decoding, output speed is step time multiplied by tokens per step. If acceptance is unchanged across your arms, every difference is step time.
Name the axis before choosing. Latency and tokens per dollar point to opposite ends of this range, TP=4 on one and TP=1 on the other, by four to five times. A benchmark table that reports only one axis has made the choice for you.
Price per node, not per GPU. Four B200s cost about six times one here, which is most of why TP=4's efficiency looks as poor as it does.
Re-measure when the model, the fabric or the hardware changes. Where the forces cross depends on weight size, what the collective costs on your interconnect, and how cleanly the model shards. A hybrid Mamba-Transformer MoE with 512 experts behaves differently from a dense transformer of the same parameter count, and a slower interconnect moves the crossing to the right.

On this model, one GPU was the right choice for everything except interactive traffic, the one place where latency is the product.

Run the head-to-head on your workload.

Share the traffic shape, the model and the latency target. PrimaLabs measures step time, output speed and cost per parallelism layout on your traffic rather than a benchmark suite, and returns the numbers in writing.

Book a demo

Frequently asked questions

Does TP=2 ever beat TP=1?

In this data, at 100k input (1.07×) and at concurrency (1.32×). But the 100k figure is within single-run noise, and at concurrency TP=4 is faster while TP=1 is still cheaper. There is no workload here where TP=2 is the configuration you would choose.

Would a faster interconnect fix TP=2?

It lowers C and moves the crossing left, so in principle yes. But TP=2's benefit is capped at half the weight read, 3.71 ms here, by the factor of two itself. A faster fabric has to bring the collective below that number, not merely improve it.

Why is the curve not monotonic?

Because the two forces scale differently. The saving follows 1 − 1/n, which rises steeply from n=1 to n=2 and then flattens; the cost appears in full the moment n exceeds one. So the first rank you add pays nearly the entire communication bill while collecting only half the available saving.

Does speculative decoding change the conclusion?

Not here. Tokens per step was flat at 3.13–3.23 across all three configurations, so acceptance contributes nothing to the differences. It does make step time matter more, since output speed is the product of the two.

Methodology. NVIDIA Nemotron-3.5-Super-VL-120B-A12B, NVFP4 checkpoint, on vLLM 0.31.0 with in-checkpoint MTP speculative decoding at K=5: a hybrid Mamba-Transformer MoE with 512 experts, sharded to 128 per rank at TP=4 with expert parallelism enabled. B200 SXM6 192GB, one GPU at TP=1, two at TP=2 and four at TP=4, all within a single node. Output speed uses Artificial Analysis's published prompt set and metric definition: answer tokens divided by answer-phase time, excluding the reasoning segment and time to first token. The same client and script ran every arm; a few runaway-reasoning samples were excluded. Step time comes from the engine's /metrics decode counters on a realistic prompt, not a client-side estimate. Hourly figures are rounded provider pricing and are quoted only as ratios.

What this does not establish. One run per cell, so treat a few percent as noise; the TP=2 deficit and the cost ratios are far larger than that, but the 100k TP=2 figure of 1.07× is inside it. Communication cost is derived, not timed: step times are measured and the collective cost is what remains after subtracting the weight saving. Expert parallelism is confirmed at TP=4 but cannot be confirmed from the TP=2 record, so part of the gap between the two could be configuration rather than rank count. This is one model on one generation of hardware; a denser model or a slower fabric moves the crossing, so re-measure before reusing any number here.