Pointer Walk

Tokenization

Models do not read words. A sentence is split into characters, then the most common pairs are merged into tokens.

AI and Machine LearningBeginner8 min read

Models do not read words

A language model works with numbers. Before it can do anything with your text, the text has to be cut into pieces and each piece replaced by a number. Those pieces are tokens.

What should a piece be?

  • One token per word? There are millions of words, plus names, typos and new slang every week. Any word not on the list could not be handled at all.
  • One token per character? The list is tiny and can spell anything. But every sentence becomes hundreds of tokens, and the model has to rebuild every word from letters.

Modern models use something in between: pieces of words. Common words get a token of their own. Rare words are built from a few smaller pieces. Nothing is ever impossible to write, and ordinary text stays short.

Building the pieces: merge the most common pair

The token list is not written by hand. It is learned from text, with a method called byte-pair encoding (BPE).

  1. Start with single characters

    Every character is a token.

  2. Count every pair of neighbouring tokens

    Which two sit next to each other most often?

  3. Merge that pair into one new token

    Everywhere it appears. The token list grows by one.

  4. Repeat

    The next most common pair may include the token just made, so pieces grow into longer pieces.

Here it is running for real on one small sentence.

Building tokens by merging pairsInteractive

Training text

thecatsatonthematandthecatatetherat

Merges learned, in order

none yet

Start. Start with the sentence split into single characters: 35 tokens. A model with only these would need a great many tokens for every text.

Tokens in this text
35
Characters
35
Merges learned
0

Look at what came out. Nobody told the algorithm what a word is, yet:

  • "the", which appears four times, ended up as a single token.
  • "cat", which appears twice, did too.
  • "rat" appears once and stayed in two pieces: r and at.
  • "chatter", a word it never saw, was still handled, as six small pieces.

That is the whole idea: how often something appears decides how big a piece it gets.

A real tokenizer does the same thing on a vast amount of text, and keeps merging until the list has tens of thousands of tokens.

Tokens are not words

A few things follow from this that surprise people.

Example

Roughly how many tokens?

In English, a token is about three quarters of a word on average, or about four characters. A 750-word essay is roughly 1,000 tokens.

Common words are one token. Long or unusual words are several. Other languages, code and numbers usually take more tokens for the same amount of text.

  • A space belongs to the token after it. " cat" and "cat" are different tokens.
  • Capital letters matter. "The" and "the" are different tokens.
  • Numbers get chopped up. A long number may become several tokens with no regard for its digits' meaning.

Why you should care

BecauseWhat it means for you
Limits are counted in tokensA model's maximum input is a number of tokens, not words or pages
Prices are counted in tokensYou pay for tokens in and tokens out
The model sees pieces, not lettersAsking it to count letters or spell backwards is harder for it than it looks
Rare words cost moreJargon, names and non-English text use up the limit faster

Key takeaways

  • A model reads tokens: pieces of text, each replaced by a number.
  • Tokens are usually pieces of words: common words are one token, rare words are several.
  • Byte-pair encoding builds the list by repeatedly merging the most common pair of neighbours.
  • In English, a token is about three quarters of a word.
  • Limits and prices are measured in tokens, and the model does not see individual letters.

Your progress

Check your progress

0%

  1. Read the page50% of this page

    When you have gone through the page and its animations, tick it off.

  2. Take the quiz50% of this page

    7 questions, from easy to hard. Take it as often as you like: the questions change, and only your best score counts.

Saved on this device only. See all your progress