How Does an LLM Turn Your Words Into Tokens?
Root Concept
An LLM never sees your letters — your text is chopped into tokens, turned into numbers, predicted over, and stitched back into words. Tokens are the unit everything runs on.
CodePLU Goal
Upgrading Human Mental Models
Learn how to think in Workflows
Concept Development By codeplu.com
Text in, tokens through the middle, text out — the token pipeline of an LLM
What Happens to Your Words Inside an LLM?
Imagine a factory that can only handle bricks from its own fixed catalogue. You bring in a sculpture; before anything can happen, it must be broken down into catalogue bricks. At the end, bricks are glued back together and handed to you as a shape again. Everything the factory actually does happens at brick level — it never touches your original sculpture at all.
That is what happens to your sentence inside a large language model. Your text is chopped into small chunks called tokens, each token is swapped for a number, and all the machinery works on those numbers. When the answer is ready, the numbers are turned back into chunks and the chunks are glued into text on your screen.
This sounds like a technical detail, but it quietly explains a lot of AI's real-world behaviour: why tools charge by tokens, why long chats start forgetting the beginning, and why an AI that can write a poem about strawberries cannot reliably count the letter r inside the word. In the playground above you will connect this journey stage by stage.
How Does the Token Pipeline Work?
What exactly is a token?
A token is a chunk of text — often a whole common word, sometimes a piece of a longer one, and it usually carries its leading space along with it. In English a token averages roughly four characters, so a rule of thumb is that a thousand tokens is around seven hundred and fifty words. Everyday words like 'water' or 'the' are single tokens because they appear constantly. Longer or rarer words get split: 'unbelievable' might arrive as three chunks, something like un / believ / able. Many beginners assume one token equals one word. It does not, and that gap is exactly where the surprising behaviour comes from.
Why must tokens become numbers?
The model is, underneath, an enormous pile of arithmetic — multiplication and addition on a colossal scale. Arithmetic needs numbers, and letters are not numbers. So every chunk in the model's catalogue has an ID, and your tokenised sentence becomes a list of IDs before anything else happens. This is the crucial consequence: the model never sees your letters. It sees a sequence of chunk IDs and works with the patterns between them. When we say an AI 'reads' your message, what really happens is closer to recognising a familiar sequence of catalogue numbers.
How does the model choose what comes next?
With your tokens loaded, the model does the thing it was trained to do: score every chunk in its catalogue for how well it would continue the sequence. Not one guess — a likelihood for tens of thousands of possible chunks at once. One of them is then chosen and added to the sequence, and the process repeats to build the answer piece by piece. That piece-by-piece construction has consequences of its own, from why answers appear word by word on your screen to why the same question can come back worded differently — and those belong to the concept of how LLMs generate responses.
Why do tokens explain the limits you keep bumping into?
Almost every practical limit in AI tools is measured in tokens, not words. A model's context window is a token budget for everything it can hold at once — your instructions, the documents you paste, the conversation so far, and the answer being written. Exceed it and the oldest material falls away, which is precisely why a very long chat starts forgetting what you said at the beginning. Pricing works the same way: services bill per token in and per token out, so a rambling prompt costs real money. Once you see limits as a token budget, these behaviours stop looking like bugs.
Why does AI get confused counting letters in a word?
Ask an AI how many times the letter r appears in 'strawberry' and you may get a confident, wrong answer. Now you know why: the model was never given letters. It received two or three chunks — perhaps str / aw / berry — and inside a chunk the individual letters are simply not visible to it. Asking it to count letters is like asking someone to count the bricks inside a wall they have only ever seen as finished rooms. This is also why spelling puzzles, rhyming edge cases, and reversing words trip these systems up. The practical fix is easy: spell the word out with spaces or dashes — s t r a w b e r r y — and each letter becomes its own token that the model can actually see and count.
Real World Example
What Happens to "How Many R's in Strawberry?"
Follow the exact sentence from the playground through every stage, and watch the mistake become inevitable rather than mysterious.
Your text is chopped into chunks
The sentence is split into tokens: 'How', ' many', ' r', "'s", ' in', ' straw', 'berry', '?'. Notice that the fruit did not survive as one piece — and that the exact split differs between AI tools, since each has its own catalogue of chunks.
Chunks become a list of numbers
Each chunk is swapped for its ID — something like 2118, 1523, 436, 885, 306, 15113, 15717, 30. From this moment on there is no sentence and no spelling, only this list of numbers moving through the arithmetic.
The model scores what should come next
It has seen countless questions of this shape and knows the answer is a small number word. So 'two' and 'three' both score highly, plus a short explanation to follow. Nowhere in this process does anything actually inspect the letters of the word.
Chunks are chosen and glued back into text
A chunk is picked, added, and the process repeats until a full reply exists. The numbers are converted back into chunks, the chunks joined into readable text, and you see: 'There are two r's in strawberry.' Fluent, confident — and wrong, because the word has three.
The fix follows directly from the pipeline
Ask again with the word spelled out — 'count the r's in s t r a w b e r r y' — and each letter is now its own token, visible to the model. It answers three. Nothing about the AI improved; you simply handed it bricks the right size for the job.
FAQs
Final Words
Text in, tokens through the middle, text out. In the playground you connected that whole journey: your words chopped into chunks, chunks swapped for numbers, the model scoring what fits next, chunks chosen one at a time, and everything stitched back into readable text at the end.
Keep the brick factory in mind, because it demystifies so much: token budgets explain forgetful long chats, token billing explains why bloated prompts cost more, and the invisibility of letters inside chunks explains the strawberry problem — along with the simple fix of spelling words out. Next comes the bigger question of what these models actually are, and how the piece-by-piece choosing turns into a flowing answer.
Continue This Track
This concept is part 3 of How Large Language Models Work.
What Are Large Language Models?
An LLM is not a search engine and not a database. Find out what it really is — a machine that learned the patterns of language from an enormous amount of text, and writes new text from them.
How Does a Large Language Model Work?
Inside an LLM there are no rules and no stored facts — only billions of tuned numbers and one clever trick: every word looks at the words around it to settle what it means.
How Does an LLM Turn Your Words Into Tokens?
An AI never sees your letters. Follow your sentence as it is chopped into tokens, turned into numbers, predicted over, and stitched back into text — and see why tokens explain AI's oddest failures.
How Do LLMs Generate a Response?
An AI does not write an answer — it grows one, adding a single chunk of text at a time and re-reading everything before each choice. That one fact explains streaming, randomness, rambling, and mistakes it cannot take back.
