Eight words that come up everywhere
Read anything about AI and the same handful of terms appears, usually without explanation. Here are eight of them. Each one has a plain definition, an example, and where possible an animation of the thing actually happening. Each links to a full page if you want more.
1. Token
A piece of text, as the model sees it. A model does not read letters or words. Text is cut into pieces from a fixed list, and each piece is replaced by a number.
Common words are one token. Rare words are several. In English a token is about three quarters of a word.
Training text
Merges learned, in order
none yet
Start. Start with the sentence split into single characters: 35 tokens. A model with only these would need a great many tokens for every text.
- Tokens in this text
- 35
- Characters
- 35
- Merges learned
- 0
Why it matters: limits and prices are counted in tokens. Full page: Tokenization.
2. Context window
The most text a model can be given at one time, measured in tokens. Your instructions, the whole conversation, any documents and the reply all have to fit inside it.
A model keeps nothing between requests. What looks like memory is the app sending the entire conversation again with every message.
Watch what is actually sent to the model each time.
Why it matters: when a chat gets too long, the oldest parts are dropped and the model can no longer see them. Full page: Context Windows and Memory.
3. Embedding
A list of numbers that stands for the meaning of a piece of text. Texts that mean similar things get similar numbers, so "similar meaning" becomes "nearby position", which a computer can measure.
QHow do I reset my password?
1Steps to recover access to your account0.99
2Forgot your login? Start here0.97
3How to choose a strong password0.69
4Reset your router to factory settings0.07
5Our refund policy-0.81
6Pasta for two in 20 minutes-0.87
Start. Every text has been turned into an arrow. Texts about similar things point in similar directions. The pink arrow is the question.
Why it matters: it is how search finds the right document even when no words match. Full page: Embeddings and Semantic Search.
4. Temperature
A setting that controls how adventurous the model's word choices are. At each step the model has a chance for every possible next token. Low temperature makes the favourite win almost every time. High temperature gives unlikely words a real chance.
What was picked, run by run
Start. Temperature 0.7. The bars are the chance of each word coming next. Now run the same prompt 16 times and see what gets picked.
- Runs
- 0 / 16
- Different words picked
- 0
Why it matters: low for answers that should be consistent, higher for variety. Full page: Temperature and Sampling.
5. Top-p
A second setting for the same choice, which cuts off the unlikely end. Sort the possible next tokens from most to least likely. Keep adding them until their chances total p. Throw the rest away.
Example
Top-p of 0.9
The next word could be "blue" (65%), "clear" (20%), "grey" (11%), or one of thousands of unlikely words sharing the last 4%.
The first three add up to 96%, which passes 90%. Only those three can be picked. "cheese" is impossible, however many times you ask.
Why it matters: it stops one bizarre word from derailing a long answer. Temperature reshapes the chances. Top-p removes the tail.
6. Hallucination
A confident, fluent answer that is wrong or made up. A model generates text that is likely given its training. When it lacks the facts, the same process still produces a plausible-sounding reply. Nothing inside it flags the difference.
A question whose answer is in your documents, not in the model.
Why it matters: fluency is not evidence. Check anything important, and prefer setups where the model answers from documents it has been given.
7. RAG
Retrieval-augmented generation: look things up first, then answer. Before the model replies, the app searches your documents for relevant passages and puts them in the prompt. The model answers from what it has just read.
A question whose answer is in your documents, not in the model.
Why it matters: it is the standard way to make a model answer correctly about your own, current information. Full page: RAG or Fine-Tuning.
8. Fine-tuning
Training an existing model further on your own examples. It changes the model itself, so that it reliably writes in a certain style, follows a certain format, or performs a narrow task.
It teaches behaviour, not facts. For facts, use RAG.
What is the model getting wrong?
All eight in one table
| Word | In one line |
|---|---|
| Token | A piece of text, the unit a model reads and writes |
| Context window | The most text a model can take in at once |
| Embedding | Numbers that place a text's meaning in space |
| Temperature | How much chance unlikely words are given |
| Top-p | A cut-off that removes the least likely words |
| Hallucination | A confident answer that is not true |
| RAG | Fetching relevant text and giving it to the model before it answers |
| Fine-tuning | Further training to change how a model behaves |
Key takeaways
- Tokens are the pieces a model reads. The context window is how many it can read at once.
- Embeddings turn meaning into positions, so similar texts can be found.
- Temperature and top-p control how the next token is picked.
- A hallucination is likely-sounding text that is not true.
- RAG supplies facts at answer time. Fine-tuning changes behaviour through training.