The word "it"
Read these two sentences.
The animal did not cross the street because it was too tired.
The animal did not cross the street because it was too wide.
In the first, "it" is the animal. In the second, "it" is the street. You worked that out without effort, using the last word and what you know about animals and streets.
A language model starts with each word as a separate list of numbers. The numbers for "it" are the same in both sentences. To understand the sentence, the model has to let "it" gather information from the other words. The mechanism that does this is called attention.
Every word looks at every other word
Each token asks: which other tokens matter to me?
It compares itself with every token in the text and gives each a score.
The scores are turned into shares that add up to 100%
A high share means "I am paying a lot of attention to you".
The token updates itself with a blend of the others
It takes a lot from the tokens it attends to strongly and little from the rest.
After this step, the numbers for "it" are no longer just "it". They carry a strong trace of whichever noun it attended to.
"it" attends most to "animal" (62%). Streets do not get tired, so "it" must be the animal. Only the last word changed, and the heavy line moved to a different noun.
Switch the last word. Nothing else in the sentence changes, and the thick line jumps from "animal" to "street".
Nobody programmed the rule
There is no line of code saying "pronouns look at nouns". The scores come from numbers inside the model that were adjusted during training, by showing it enormous amounts of text and nudging it whenever it predicted the next word badly. Attending to the right noun turned out to help prediction, so the model learned to do it.
The price: everything looks at everything
Each token looks at every token. With n tokens, that is n × n comparisons.
8 × 8 = 64 pairs. Each square is one token looking at one other token. Double the prompt and the grid is twice as wide and twice as tall: four times the squares.
Example
Double the prompt, four times the work
1,000 tokens means 1,000 × 1,000 = 1,000,000 pairs.
2,000 tokens means 4,000,000 pairs.
10,000 tokens means 100,000,000 pairs. Ten times the text, a hundred times the pairs.
This is one of the main reasons models have a limit on how much text they can take in at once, why long prompts are slower and cost more, and why a great deal of research goes into making attention cheaper.
Why it changed everything
Older models read text one word at a time, passing along a summary as they went. By the end of a long paragraph, the beginning had faded.
With attention, any word can look directly at any other word, however far apart. A name on page one is one step away from a pronoun on page ten. And because every token can do its looking at the same time, the work can be done in parallel on modern hardware. This design is the Transformer, and it is the "T" in GPT.
Key takeaways
- Attention lets each token gather information from the other tokens in the text.
- Each token scores every other token, and updates itself with a weighted blend of them.
- This is how "it" comes to mean the animal in one sentence and the street in another.
- The patterns are learned from data, not written as rules.
- The work grows with the square of the length: double the prompt, four times the pairs.