Two apps take exactly 12 seconds to answer. One feels broken, the other feels fine, and the model is identical in both.
The only difference is when the first word shows up.
An AI answer has two speeds, not one. Time to first token is how long you stare at nothing before anything appears, and most of that gap is the model reading your prompt before it writes a single word (which is why a long prompt hurts the wait even when the answer itself is short). The second speed is how fast the rest arrives once it has started.
Streaming means sending each piece as it is generated instead of holding the finished answer back. A restaurant serving each course as it is ready instead of keeping every plate until the last dish is done: same meal, same total time, only one of them leaves you staring at an empty table.
Why the empty table costs so much: Jakob Nielsen's response time limits (1993, built on research going back to the 1960s) say a user's train of thought survives about 1 second of delay, and attention is gone past 10. Delivered in one lump, a 12 second answer breaks that every time. Streamed, the first word lands inside the 1 second window and the other 11 seconds turn into reading rather than waiting. The part I keep coming back to is that nothing got faster. The wait just moved to the side of the line where people stay.
Agents are the hard case. When one stops to run a search or call another service, no words are being produced, so there is nothing to stream, and the honest substitute is showing the user which step is running.
Quick definitions: - rest: an HTTP API style where each URL maps to a resource and the method (GET, POST) says what to do with it.
Quick check before you scroll: A support bot generates a 400 token answer in 8 seconds. After switching on streaming, what does the user's wait become, what stays exactly the same, and which safety check gets harder?
Full breakdown + the answer: frankduah.me/learnings/2026-08-31-streaming-responses-time-to-first-token-and-why-the-wait-is
New here? I post a bite-size AI / ML concept like this every day, follow me for the daily drop, and it compounds fast. Why I do it: https://lnkd.in/gK8knHDH
#StreamingResponses #AI #LLM #AIAgents #MachineLearning
The answer
The wait drops to the time to first token, typically well under a second. The total generation time is unchanged at 8 seconds; streaming changes when the words appear, not how fast they are produced. The check that gets harder is any output filter that needs the complete answer, such as a policy or toxicity screen on the full text: it can no longer run before the first words are on screen, so it has to run on chunks as they arrive or on the buffered end with the ability to retract.