Seven requests have finished. The eighth is still typing. On a naive LLM server, those seven seats stay locked until the eighth is done.
That is where the money goes when you serve a model at scale: not a model that thinks too slowly, a GPU that sits half empty waiting for it.
Why the seats are locked at all: as a model writes, it keeps a running set of intermediate calculations for every token (roughly, every word piece) it has seen so far, so it never redoes that math for the whole conversation on each new word. That store is the KV cache. It sits in GPU memory, one chunk per live conversation, growing with every token.
The counterintuitive part: the ceiling here is memory, not compute. And fixed batching wastes what memory is left: wait for 8 requests, run them together, wait for all 8 to finish, repeat. The one person asking for an essay holds the other seven, whose caches keep occupying memory long after they are done.
The fix reads as trivial and is not: recheck the batch after every single token, so whoever finished leaves and whoever is queued takes their slot right then. Engines like vLLM ship this as continuous batching, next to PagedAttention, which hands out that memory in small pages as a reply grows instead of reserving one big block up front.
Don't wait for the whole table to finish eating. Reseat the chair the second it opens up.
If you build agents, every tool call in the loop pays this tax.
Quick check before you scroll: Why is "wait for the batch to fully finish, then start the next batch" bad for GPU utilization?
Full breakdown + the answer: frankduah.me/learnings/2026-09-17-serving-llms-in-production-latency-cost-scale
New here? I post a bite-size AI / ML concept like this every day - follow me for the daily drop, and it compounds fast. Why I do it: https://lnkd.in/gK8knHDH
#LLMInference #vLLM #MLOps #AI #LLM #AIAgents #MachineLearning
The answer
Different requests finish generating at different times (some responses are short, some long), so with fixed batching the GPU sits idle waiting on the slowest request in the batch instead of immediately reusing freed-up capacity for new work - continuous batching fixes this by swapping requests in and out per step.