Pointer Walk

How an LLM Generates Text

One prompt in: tokens, vectors, layers, scores, one token out. Then the whole thing runs again for the next token.

AI and Machine LearningBeginner10 min read

One job: guess the next piece

Ask a language model a question and it writes paragraphs. It is tempting to imagine it thinking of an answer and then typing it out.

That is not what happens. A language model does exactly one thing: given some text, it estimates what piece of text is likely to come next. Then that piece is added to the text, and it does the same thing again.

Everything it produces, from an essay to a working program, is that one step repeated.

The stages of one step

From prompt to the next tokenInteractive
  1. 1Text
  2. 2Tokens
  3. 3Vectors
  4. 4Layers
  5. 5Scores
  6. 6Pick one
The464 capital3139 of286 France4881 is318

Chance of each next token

92%
" Paris"
3%
" a"
2%
" the"
1%
" located"

Start. The model receives the prompt as plain text.

Pass
1
Tokens in the text
5
  1. Text

    Your prompt, as you typed it.

  2. Tokens

    The text is cut into pieces from a fixed list, and each piece becomes a number. See Tokenization.

  3. Vectors

    Each token number is swapped for a long list of numbers that stands for its meaning. See Embeddings.

  4. Layers

    The vectors pass through a deep stack of layers. In each layer the tokens exchange information with each other and are transformed. See Attention.

  5. Scores

    The result is turned into a chance for every token in the list: tens of thousands of numbers that add up to 100%.

  6. Pick one

    One token is chosen according to those chances. See Temperature and Sampling.

Then the whole thing runs again

One pass produces one token. To get the next one, the new token is stuck on the end of the text and everything is run again from the top.

Example

Writing three tokens

Input: The capital of France is → output: Paris

Input: The capital of France is Paris → output: .

Input: The capital of France is Paris. → output: a special "end" token. Stop.

This explains several things you can observe from outside.

  • Text appears a bit at a time. You are watching the loop run.
  • Long answers take longer and cost more. Every token is a full pass through the model.
  • It can write itself into a corner. Each token is chosen without knowing what will come after. Once written, it is part of the input and cannot be taken back.

When does it stop?

  • The model picks a special end token that it learned means "finished".
  • It reaches a maximum length set by whoever is calling it.
  • It produces a stop sequence the caller asked it to halt on.

It predicts what is likely, not what is true

The model's training goal was to make its predictions match real text. "Paris" follows "The capital of France is" in millions of documents, so it gets a very high chance. That is why models are right so often.

But the mechanism is the same when the text is about something the model knows little about. It still produces chances, still picks a token, and the result still reads fluently. A confident, well-written, wrong answer is called a hallucination, and it is a direct consequence of how generation works.

Key takeaways

  • A language model does one thing: estimate which token comes next.
  • One step: text, tokens, vectors, layers, scores, pick one.
  • The chosen token is added to the text and the whole process runs again.
  • That loop is why answers stream out piece by piece and why long answers cost more.
  • It produces what is likely given its training, which is usually but not always what is true.

Your progress

Check your progress

0%

  1. Read the page50% of this page

    When you have gone through the page and its animations, tick it off.

  2. Take the quiz50% of this page

    8 questions, from easy to hard. Take it as often as you like: the questions change, and only your best score counts.

Saved on this device only. See all your progress