Brief
Calibrated router for disaggregated LLM serving shows goodput gains in measured tests
A preprint reports a router that estimates completion time per instance using prompt length, predicted output length, KV cache pressure, and SLO class. Tested on eight NVIDIA A40 GPUs, it achieved the highest mean goodput across three bursty traces.
The study compares a calibrated router against round robin, least loaded, and a length heuristic. On eight NVIDIA A40 GPUs running vLLM engines, it achieved mean goodput of 0.864 versus 0.835 to 0.847 for the baselines. It beat round robin and the length heuristic on all three traces, and least loaded on two; on the third it trailed by 0.003, within run to run noise. Hardware calibration was crucial: simulator derived constants cost 4.5 goodput points and about 40 percent of the tail latency advantage. Benefits grew with decode pool size and traffic heterogeneity but vanished with three instances. Under extreme scarcity, greedy cost minimization concentrated requests, while blind spreading performed better. With calibrated costs, the router matched round robin goodput using six GPUs instead of seven.
Our reading
Our reading is that the measured gains are modest and depend on hardware calibration, so the approach's practical advantage may be limited to larger, more heterogeneous decode pools.
Source details and supporting facts
Each line is stated by the page named above it.
Stated by arXiv
- The router estimates the additional completion time on each instance using exact prompt length, predicted output length, post admission KV cache pressure, and SLO class.
- The study validates the policy on eight NVIDIA A40 GPUs, each running a vLLM engine, with NIXL transferring KV caches between pools.
- Across three mixed, bursty arrival traces, the calibrated router achieves the highest mean goodput at 0.864, compared with 0.835 to 0.847 for round robin, least loaded, and a length heuristic.
- Hardware calibration matters: simulator derived constants cost 4.5 goodput points and roughly 40 percent of the tail latency advantage.
- Benefits grow with decode pool size and traffic heterogeneity but disappear in pools with three instances.
- Under extreme scarcity, greedy cost minimization concentrates requests on the cheapest scored instance, and blind spreading performs better.
- With calibrated costs, the learned router matches the goodput of round robin using six GPUs instead of seven.
Sources
- arXivText stored 16 September 2026
How this story was checked. Written from the 1 page listed above, stored 16 September 2026; claims checked against that stored text on 16 September 2026.
What that means
- 7 of 7 reported statements were confirmed against the page that carries them; the rest were removed rather than published.
- Figures in the text were required to appear in the stored source text: yes. Identifiers: yes.
- The check reads stored text only: no claim rests on a fresh look that did not happen.
- Where the reporting was silent, the text says so instead of filling the gap.