How Do LLMs Generate a Response?
Root Concept
An LLM grows its answer one chunk at a time: it re-reads everything so far, scores what could come next, picks one, adds it, and goes around again — with no plan for how the answer ends.
CodePLU Goal
Upgrading Human Mental Models
Learn how to think in Workflows
Concept Development By codeplu.com
The writing loop: read, score, pick, add — then read it all again
How Does an AI Actually Write Its Answer?
Imagine writing a sentence under a strange rule: you may add only one word at a time, you must re-read the whole sentence before choosing each next word, and you are never allowed to go back and change anything. No outline, no draft, no editing. Somehow the sentence still has to make sense when you reach the end.
That is exactly how an AI writes. It does not compose an answer and then show it to you. It grows the answer: read everything so far, score every chunk that could come next, pick one, add it — then read everything again, including the chunk it just added. Round and round, a few hundred times for a single paragraph.
This one mechanism explains an astonishing amount of everyday AI behaviour: why answers appear word by word, why the same question gives differently worded replies, why an AI sometimes rambles or paints itself into a corner, and why it never takes back a sentence it has already started. Build the loop in the playground above and the rest of this concept clicks into place.
What Happens on Each Trip Around the Loop?
Why does the answer appear word by word?
Because it is genuinely being made word by word. That typing effect is not a decorative animation designed to look thoughtful — each chunk is sent to your screen the moment it is chosen, while the rest of the answer does not exist yet. When the text pauses slightly, real work is happening: the entire model is running again to choose the next chunk. This also explains why long answers take longer in direct proportion to their length, and why asking for a shorter reply genuinely gets you a faster one. There is no finished paragraph waiting behind a curtain, being revealed slowly for effect.
Where does the randomness come from?
After scoring every possible next chunk, the model does not always take the top scorer. It usually samples — leaning heavily toward the likely chunks while leaving room for less obvious ones. That small deliberate randomness is why asking the identical question twice gives two differently worded answers. Most tools expose it as a setting called temperature: low temperature sticks to the safest chunks and produces predictable, sometimes repetitive text, while higher temperature takes more chances and produces livelier but riskier text. Many beginners read variation as unreliability. It is simply a dial, usually set to a middle value on purpose.
Why can't the AI plan the whole answer first?
Because there is nowhere for a plan to live. Each trip around the loop produces exactly one chunk, chosen from what exists so far, with no separate outline held aside. The model has absorbed the shape of well-organised writing, so answers usually flow sensibly — but it is improvising in a very well-practised way rather than following an agenda. This is why it can begin a sentence it cannot gracefully finish, and why tasks that constrain the ending are surprisingly hard for it. It commits to the beginning long before it knows the end.
How does the model know when to stop?
Stopping is just another choice. Among all the chunks it can pick sits a special one meaning 'the answer is finished', and when that becomes the likeliest option, generation ends. Because the model learned this from countless examples of complete answers, it usually stops somewhere sensible. Two other things can cut it off: the tool imposes a maximum length, or the shared token budget runs out — which is what has happened when an answer stops abruptly mid-sentence. If that occurs, simply asking it to continue usually works, because everything written so far is still in view.
Why does it never take back a mistake?
Once a chunk is added it becomes part of the input for every later choice, so a wrong turn is not merely left in place — it actively shapes what follows. This is how a small early error grows into a confidently wrong paragraph: the model keeps writing text consistent with what it already said. It also explains why 'work through it step by step' improves accuracy, since the reasoning chunks written early become material read later, like showing your working on paper. And it is why the fix for an answer that has gone off the rails is a fresh, clearer prompt rather than an argument.
Real World Example
Why Does an AI Struggle to End a Sentence on a Given Word?
Ask an AI for a sentence about the sea that ends with the word 'orange'. Follow the loop and watch why this simple-sounding task is genuinely awkward for it.
The first chunk is chosen with no ending in mind
It reads your request and picks something that fits a sentence about the sea: 'The'. That is the entire decision. Nothing anywhere holds a plan for how this sentence will arrive at 'orange' — there is only this chunk, chosen because it fits what came before it.
Each new chunk narrows what is still possible
'The' becomes 'The sun', then 'The sun sank', then 'The sun sank into the'. Every addition is sensible on its own, and every addition quietly shrinks the set of endings that could still work. The loop has no way to check the destination while doing this.
Sometimes it lands, sometimes it cannot
From 'turning the water a deep shade of' the constraint is easy to meet. But had it begun 'Fishermen returned at dusk with their nets', there may be no graceful path to 'orange' left at all. It cannot rewind to try a different opening, so it improvises from where it is.
The failure looks like carelessness but is structural
You may get a clumsy ending, or the word 'orange' appearing in the middle instead. It is not ignoring your instruction or being lazy. It committed to a beginning before knowing the end, exactly as the loop forces it to.
How to work with the loop instead of against it
Ask it to plan out loud first — 'suggest three possible endings, then write the sentence' — and that plan becomes written chunks it can read while composing. You have given the loop somewhere to keep an outline, and the same constrained task suddenly becomes easy.
FAQs
Final Words
An AI answer is grown, not written: read everything so far, score what could come next, pick one chunk, add it, and go around again. In the playground you built that loop including the edge that makes it a loop — the new chunk becoming part of the input.
Almost every quirk you will meet traces back to it. Streaming is real work happening live. Variation is a randomness dial. Rambling and awkward endings come from having no outline. And a wrong turn stays in view, shaping everything after it, which is why the fix is a fresh prompt rather than an argument. With this loop in hand you have the full picture of how a language model turns your words into an answer.
Continue This Track
This concept is part 4 of How Large Language Models Work.
What Are Large Language Models?
An LLM is not a search engine and not a database. Find out what it really is — a machine that learned the patterns of language from an enormous amount of text, and writes new text from them.
How Does a Large Language Model Work?
Inside an LLM there are no rules and no stored facts — only billions of tuned numbers and one clever trick: every word looks at the words around it to settle what it means.
How Does an LLM Turn Your Words Into Tokens?
An AI never sees your letters. Follow your sentence as it is chopped into tokens, turned into numbers, predicted over, and stitched back into text — and see why tokens explain AI's oddest failures.
How Do LLMs Generate a Response?
An AI does not write an answer — it grows one, adding a single chunk of text at a time and re-reading everything before each choice. That one fact explains streaming, randomness, rambling, and mistakes it cannot take back.
