Today's concept: model serving, batch vs real-time vs streaming
That "instant" ChatGPT-style reply isn't one prediction dropped on you all at once, it's a live stream of hundreds of tiny ones, arriving token by token while the model is still thinking.
Model serving is just "how does a trained model actually answer requests in production." Three shapes:
1) Batch = a cron job. A job spins up, chews through millions of queued inputs, writes results to storage, shuts down. Nobody is waiting, so latency is irrelevant: you optimise throughput and cost per prediction. Amazon SageMaker's Batch Transform is literally this: read from S3, run the dataset, write to S3, terminate.
2) Real-time = a REST call (one HTTP request, one answer back). A prediction in milliseconds from a server kept warm 24/7: fraud checks, a recommendation on page load. You pay for that warmth whether traffic shows up or not.
3) Streaming = a connection that keeps talking, pushing partial results out as they're produced. For LLMs that's server-sent events (a one-way HTTP stream) rather than a websocket: "stream": true on the Messages API.
Anthropic's Message Batches API puts a price tag on the tradeoff: submit asynchronously, pay 50% less, most batches finish in under an hour (results within 24 hours; a batch can expire unfinished when demand is heavy). Right home for nightly agent evals or embedding a corpus, wrong home for anything a human is watching.
The bit that trips people up: "batch" names two different things. Batch serving is offline and async. Continuous batching (a headline vLLM feature, beside PagedAttention) is a scheduler packing concurrent live requests into the same GPU step, retiring and admitting them per iteration, fully compatible with real-time and streaming.
Most of what's trending in inference sits on that seam: prefix and KV caching (Anthropic bills cache reads at 0.1x input, and time-to-first-token drops), speculative decoding (a small draft model proposes tokens the big one verifies in one pass). All of it exists because reading your prompt and emitting tokens one at a time bottleneck differently, which is why serving teams quote time-to-first-token and tokens-per-second separately.
For agents: stream so the loop feels alive, real-time for tool steps, batch for the bulk nobody watches.
Quick definitions: - REST: an HTTP API style where each URL maps to a resource and the method (GET, POST) says what to do with it.
Quick check before you scroll: Why not just use a real-time endpoint for everything, including scoring 50 million rows overnight?
Full breakdown + the answer: frankduah.me/learnings/2026-08-23-model-serving-batch-vs-real-time-vs-streaming
New here? I post a bite-size AI / ML concept like this every day, follow me for the daily drop, and it compounds fast. Why I do it: https://lnkd.in/gK8knHDH
#ModelServing #AI #LLM #AIAgents #MachineLearning
The answer
A real-time endpoint is provisioned and billed continuously whether it's busy or not, and it's built for one-request-at-a-time responses, grinding through 50 million rows that way is slow and expensive. Batch only spins up compute when the job runs, blasts through everything at high throughput, then shuts down, which is far cheaper per prediction at that scale.