Searching by meaning
Type "How do I reset my password?" into an old-style search box and it looks for pages containing those words. It will find "Reset your router", because the word "reset" matches. It may miss "Steps to recover access to your account", which is the right answer and shares none of the words.
To search by meaning, the computer needs a way to tell that two pieces of text are about the same thing even when the words differ. That is what embeddings are for.
Text becomes a position
An embedding model reads a piece of text and outputs a list of numbers, typically several hundred to a few thousand of them. That list is the text's embedding, also called a vector.
The numbers are not random. The model was trained so that texts with similar meanings get similar numbers.
Think of the list as coordinates. Two numbers give a point on a map. Three give a point in a room. A thousand give a point in a space we cannot picture but can still do arithmetic in. Every text lands somewhere in that space, and texts about the same thing land close together.
Example
A map of meaning
"dog" lands near "puppy" and "bark". All three are far from "invoice".
"How do I reset my password?" lands near "I cannot log in" and "recover my account", even though they have no words in common.
Measuring closeness: the angle between arrows
Draw each embedding as an arrow from the centre to its point. Two texts are similar when their arrows point in nearly the same direction.
The usual measure is cosine similarity: the cosine of the angle between the two arrows.
| Angle between arrows | Cosine similarity | Meaning |
|---|---|---|
| 0 degrees: same direction | 1 | Same meaning |
| 90 degrees: at right angles | 0 | Unrelated |
| 180 degrees: opposite | -1 | As different as can be |
Here is a question and six documents. To make it drawable they have been flattened onto a circle and placed by hand. The similarity scores are the real formula applied to those arrows.
QHow do I reset my password?
1Steps to recover access to your account0.99
2Forgot your login? Start here0.97
3How to choose a strong password0.69
4Reset your router to factory settings0.07
5Our refund policy-0.81
6Pasta for two in 20 minutes-0.87
Start. Every text has been turned into an arrow. Texts about similar things point in similar directions. The pink arrow is the question.
The winner shares no key word with the question. Two documents that do share a word, "reset" and "password", are ranked lower, because they are about routers and password strength.
How a search works
Ahead of time: embed every document
Run each document through the embedding model and store its vector.
When a question arrives: embed the question
With the same model.
Find the nearest vectors
Compare the question's vector with the stored ones and take the closest few.
Return those documents
The nearest arrow wins.
question = embed("How do I reset my password?")
scores = [(cosine_similarity(question, vector), text) for text, vector in stored]
best = sorted(scores, reverse=True)[:3]With millions of documents, comparing against every one is too slow. A vector database stores the vectors in a structure that finds near neighbours quickly without checking them all.
Where you will meet them
- Semantic search: find documents by meaning, as above.
- RAG: fetch relevant passages to give a language model before it answers. See RAG or Fine-Tuning.
- Recommendations: "more like this" is "nearest vectors to this".
- Grouping: cluster thousands of support tickets by topic without reading them.
- Inside the language model itself: the first thing a model does with each token is look up its embedding.
Key takeaways
- An embedding is a list of numbers that represents the meaning of a piece of text.
- Texts with similar meanings get vectors that point in similar directions.
- Cosine similarity measures the angle between two vectors: 1 is the same direction, 0 is unrelated.
- Semantic search embeds the question and returns the nearest stored vectors.
- The best match may share no words with the question.