Pointer Walk

RAG or Fine-Tuning

Does the model lack facts, or lack a skill? Facts change, so they are looked up. Behaviour is trained.

AI and Machine LearningIntermediate9 min read

The model does not know your business

A language model learned from a huge amount of public text, up to a certain date. It has never seen your company's handbook, your product's manual, or anything that happened after it was trained.

Ask it about those and it will still answer, fluently and confidently.

The model on its ownInteractive
You
App
Your documents
Model

A question whose answer is in your documents, not in the model.

There are two well-known ways to close this gap, and they are constantly confused. They fix different problems.

  • RAG gives the model information at the moment it answers.
  • Fine-tuning changes how the model behaves.

RAG: let it read before it answers

Retrieval-augmented generation is an open-book exam. Before the model answers, the app looks up relevant passages in your documents and puts them in the prompt.

With retrieval (RAG)Interactive
You
App
Your documents
Model

A question whose answer is in your documents, not in the model.

  1. Retrieve

    Search your documents for the passages most related to the question. This is usually a search by meaning, using embeddings.

  2. Augment

    Add those passages to the prompt, with an instruction such as "answer using only the text below".

  3. Generate

    The model writes its answer from what it was just given.

The model itself is untouched. Nothing was trained.

Good at

  • Answers stay current: update a document and the next answer reflects it
  • Can show the source of each answer
  • Access can be controlled per user, by controlling what is retrieved
  • No training needed

Where it bites

  • Only as good as the search: if the right passage is not retrieved, the answer is wrong
  • Retrieved text takes up room in every prompt
  • Does not change the model's tone, format or skills

Fine-tuning: more practice

Fine-tuning takes an already trained model and trains it some more, on your own examples. Each example is an input and the exact output you want.

Example

What the examples look like

Input: a customer email. Output: a reply in your company's voice, in three short paragraphs, signed off in your standard way.

A few hundred to a few thousand of those, and the model learns to write that way without being told each time.

This adjusts the numbers inside the model. The result is a new version of the model that has absorbed a skill or a style.

Good at

  • Consistent tone, format or behaviour without long instructions in every prompt
  • Can teach a narrow task very well, such as classifying your kind of ticket
  • Shorter prompts, so faster and cheaper per request

Where it bites

  • Needs good training examples, which take effort to collect
  • A poor way to teach facts: they are not reliably recalled, and changing one means training again
  • The model is frozen at whatever it was shown

One question separates them

Does the model lack facts, or lack a skill?

  • It gives wrong or outdated information about your world → it lacks facts. Facts change, so they are looked up: RAG.
  • It knows enough but responds in the wrong format, tone or manner → it lacks a skill. Behaviour is trained: fine-tuning.
RAG or fine-tuning?Interactive

What is the model getting wrong?

RAGFine-tuning
ChangesWhat the model is givenThe model itself
Good forFacts, documents, fresh or private dataStyle, format, a specialised task
To updateEdit a documentTrain again
Can cite sourcesYesNo
Effort to startBuild a search over your documentsCollect examples and run training
With retrieval (RAG)Interactive
Setup
You
App
Your documents
Model

A question whose answer is in your documents, not in the model.

Key takeaways

  • A model only knows its training data. It will answer about anything else anyway.
  • RAG retrieves relevant passages and puts them in the prompt, so the model answers from them.
  • Fine-tuning trains the model further on examples, changing how it behaves.
  • Facts change, so they are looked up. Behaviour is trained.
  • Try a better prompt first, then RAG, then fine-tuning.

Your progress

Check your progress

0%

  1. Read the page50% of this page

    When you have gone through the page and its animations, tick it off.

  2. Take the quiz50% of this page

    7 questions, from easy to hard. Take it as often as you like: the questions change, and only your best score counts.

Saved on this device only. See all your progress