Models do not read words
A language model works with numbers. Before it can do anything with your text, the text has to be cut into pieces and each piece replaced by a number. Those pieces are tokens.
What should a piece be?
- One token per word? There are millions of words, plus names, typos and new slang every week. Any word not on the list could not be handled at all.
- One token per character? The list is tiny and can spell anything. But every sentence becomes hundreds of tokens, and the model has to rebuild every word from letters.
Modern models use something in between: pieces of words. Common words get a token of their own. Rare words are built from a few smaller pieces. Nothing is ever impossible to write, and ordinary text stays short.
Building the pieces: merge the most common pair
The token list is not written by hand. It is learned from text, with a method called byte-pair encoding (BPE).
Start with single characters
Every character is a token.
Count every pair of neighbouring tokens
Which two sit next to each other most often?
Merge that pair into one new token
Everywhere it appears. The token list grows by one.
Repeat
The next most common pair may include the token just made, so pieces grow into longer pieces.
Here it is running for real on one small sentence.
Training text
Merges learned, in order
none yet
Start. Start with the sentence split into single characters: 35 tokens. A model with only these would need a great many tokens for every text.
- Tokens in this text
- 35
- Characters
- 35
- Merges learned
- 0
Look at what came out. Nobody told the algorithm what a word is, yet:
- "the", which appears four times, ended up as a single token.
- "cat", which appears twice, did too.
- "rat" appears once and stayed in two pieces:
randat. - "chatter", a word it never saw, was still handled, as six small pieces.
That is the whole idea: how often something appears decides how big a piece it gets.
A real tokenizer does the same thing on a vast amount of text, and keeps merging until the list has tens of thousands of tokens.
Tokens are not words
A few things follow from this that surprise people.
Example
Roughly how many tokens?
In English, a token is about three quarters of a word on average, or about four characters. A 750-word essay is roughly 1,000 tokens.
Common words are one token. Long or unusual words are several. Other languages, code and numbers usually take more tokens for the same amount of text.
- A space belongs to the token after it.
" cat"and"cat"are different tokens. - Capital letters matter.
"The"and"the"are different tokens. - Numbers get chopped up. A long number may become several tokens with no regard for its digits' meaning.
Why you should care
| Because | What it means for you |
|---|---|
| Limits are counted in tokens | A model's maximum input is a number of tokens, not words or pages |
| Prices are counted in tokens | You pay for tokens in and tokens out |
| The model sees pieces, not letters | Asking it to count letters or spell backwards is harder for it than it looks |
| Rare words cost more | Jargon, names and non-English text use up the limit faster |
Key takeaways
- A model reads tokens: pieces of text, each replaced by a number.
- Tokens are usually pieces of words: common words are one token, rare words are several.
- Byte-pair encoding builds the list by repeatedly merging the most common pair of neighbours.
- In English, a token is about three quarters of a word.
- Limits and prices are measured in tokens, and the model does not see individual letters.