The model does not know your business
A language model learned from a huge amount of public text, up to a certain date. It has never seen your company's handbook, your product's manual, or anything that happened after it was trained.
Ask it about those and it will still answer, fluently and confidently.
A question whose answer is in your documents, not in the model.
There are two well-known ways to close this gap, and they are constantly confused. They fix different problems.
- RAG gives the model information at the moment it answers.
- Fine-tuning changes how the model behaves.
RAG: let it read before it answers
Retrieval-augmented generation is an open-book exam. Before the model answers, the app looks up relevant passages in your documents and puts them in the prompt.
A question whose answer is in your documents, not in the model.
Retrieve
Search your documents for the passages most related to the question. This is usually a search by meaning, using embeddings.
Augment
Add those passages to the prompt, with an instruction such as "answer using only the text below".
Generate
The model writes its answer from what it was just given.
The model itself is untouched. Nothing was trained.
Good at
- Answers stay current: update a document and the next answer reflects it
- Can show the source of each answer
- Access can be controlled per user, by controlling what is retrieved
- No training needed
Where it bites
- Only as good as the search: if the right passage is not retrieved, the answer is wrong
- Retrieved text takes up room in every prompt
- Does not change the model's tone, format or skills
Fine-tuning: more practice
Fine-tuning takes an already trained model and trains it some more, on your own examples. Each example is an input and the exact output you want.
Example
What the examples look like
Input: a customer email. Output: a reply in your company's voice, in three short paragraphs, signed off in your standard way.
A few hundred to a few thousand of those, and the model learns to write that way without being told each time.
This adjusts the numbers inside the model. The result is a new version of the model that has absorbed a skill or a style.
Good at
- Consistent tone, format or behaviour without long instructions in every prompt
- Can teach a narrow task very well, such as classifying your kind of ticket
- Shorter prompts, so faster and cheaper per request
Where it bites
- Needs good training examples, which take effort to collect
- A poor way to teach facts: they are not reliably recalled, and changing one means training again
- The model is frozen at whatever it was shown
One question separates them
Does the model lack facts, or lack a skill?
- It gives wrong or outdated information about your world → it lacks facts. Facts change, so they are looked up: RAG.
- It knows enough but responds in the wrong format, tone or manner → it lacks a skill. Behaviour is trained: fine-tuning.
What is the model getting wrong?
| RAG | Fine-tuning | |
|---|---|---|
| Changes | What the model is given | The model itself |
| Good for | Facts, documents, fresh or private data | Style, format, a specialised task |
| To update | Edit a document | Train again |
| Can cite sources | Yes | No |
| Effort to start | Build a search over your documents | Collect examples and run training |
A question whose answer is in your documents, not in the model.
Key takeaways
- A model only knows its training data. It will answer about anything else anyway.
- RAG retrieves relevant passages and puts them in the prompt, so the model answers from them.
- Fine-tuning trains the model further on examples, changing how it behaves.
- Facts change, so they are looked up. Behaviour is trained.
- Try a better prompt first, then RAG, then fine-tuning.