Brief
Preprint scheduler cuts agentic LLM workflow tail latency by up to 3.50×
A new arXiv paper proposes releasing model turns only when tail risk is low, instead of the moment they are ready. On real software-engineering traces it matched eager release under light load and cut P95 workflow flow time under contention by up to 3.50×.
Agentic LLM workflows interleave model turns with tool interactions, so their end-to-end completion time depends on when ready turns are released, not only on inference speed. Most runtimes release each turn the moment it becomes ready.
The paper's authors say that under contention this eager release piles up released but unfinished work that can no longer be reordered, which raises tail latency. Their method decides which ready turn to release next and how much unfinished work to keep in flight, using a mean-CVaR objective, online estimates of turn work, and a budget that adapts to observed queue pressure.
On real agent execution traces from software engineering tasks across multiple LLMs and workflow arrival rates, the method performed comparably to eager release under light load and cut the P95 of workflow flow time under contention, reaching up to a 3.50× speedup. The work is a preprint and has not been peer-reviewed.
Our reading
Our reading is that the reported speedup is promising, but it comes from a single preprint's own evaluation, so the 3.50× figure is preliminary.
What to do or watch
Watch for peer review and independent replication before treating the 3.50× P95 reduction as established, since it comes from a single preprint's own evaluation. The unresolved question is how much queue contention is required before the tail-aware release policy beats eager release, and whether that gap holds on workloads other than the software-engineering traces tested here.
Source details and supporting facts
Each line is stated by the page named above it.
Stated by arXiv
- Agentic LLM workflows interleave model turns with tool interactions, so end-to-end completion time depends on when ready turns are released, not only on inference speed.
- Most runtimes release each turn immediately upon readiness; under contention this eager release can accumulate released but unfinished work and increase tail latency.
- The proposed method uses a mean-Conditional Value-at-Risk objective, online estimates of turn work, and a released-work budget that adapts to observed queue pressure.
- On real agent execution traces from software engineering tasks across multiple LLMs and arrival rates, the method performed comparably to eager release under light load and cut the P95 of workflow flow time under contention by up to 3.50×.
Sources
- arXivText stored 13 September 2026
How this story was checked. Written from the 1 page listed above, stored 13 September 2026; claims checked against that stored text on 14 September 2026.
What that means
- 4 of 4 reported statements were confirmed against the page that carries them; the rest were removed rather than published.
- Figures in the text were required to appear in the stored source text: yes. Identifiers: yes.
- The check reads stored text only: no claim rests on a fresh look that did not happen.
- Where the reporting was silent, the text says so instead of filling the gap.