A transformer is a neural network (a big pile of adjustable numbers that turns an input into an output) that takes a sequence of tokens, which are small chunks of text, and outputs a score for every possible next token. It fires once per token it generates, so a 500 token answer means 500 separate runs of the same network.
Inside, a transformer is a stack of identical layers. Each layer has two jobs: attention, where every token pulls in information from the other tokens it can see, and a small feed-forward network (plain layers of numbers) that processes each token on its own afterwards. Chat models like GPT and Claude generate text by running that stack, picking one token from the scores, tacking it onto the input, and running the whole thing again.
"Stateless" describes what the model sees, not what the server does. Inference servers typically keep the intermediate attention values (the key/value cache) for tokens they already processed, so they don't redo that work for every new token, and some APIs, Anthropic's prompt caching among them, let you reuse that work across separate requests when the start of your prompt is identical. The catch for agent code: changing anything near the top of the prompt, such as a timestamp in the system message, breaks the match and you pay full price for the whole context again.
Quick check before you scroll: A model writes a 200 token reply. How many times did the transformer run, and where does it "remember" what it already wrote?
The answer
About 200 times, once per new token. It remembers only because each token it produced was appended to the input for the next run.
New here? I post a bite-size AI / ML concept like this every day. Follow me for the daily drop, and it compounds fast. Why I do it: https://lnkd.in/gK8knHDH