Pointer Walk

Embeddings and Semantic Search

Search by meaning: text becomes an arrow, and the nearest arrow wins even when no words match.

AI and Machine LearningBeginner9 min read

Searching by meaning

Type "How do I reset my password?" into an old-style search box and it looks for pages containing those words. It will find "Reset your router", because the word "reset" matches. It may miss "Steps to recover access to your account", which is the right answer and shares none of the words.

To search by meaning, the computer needs a way to tell that two pieces of text are about the same thing even when the words differ. That is what embeddings are for.

Text becomes a position

An embedding model reads a piece of text and outputs a list of numbers, typically several hundred to a few thousand of them. That list is the text's embedding, also called a vector.

The numbers are not random. The model was trained so that texts with similar meanings get similar numbers.

Think of the list as coordinates. Two numbers give a point on a map. Three give a point in a room. A thousand give a point in a space we cannot picture but can still do arithmetic in. Every text lands somewhere in that space, and texts about the same thing land close together.

Example

A map of meaning

"dog" lands near "puppy" and "bark". All three are far from "invoice".

"How do I reset my password?" lands near "I cannot log in" and "recover my account", even though they have no words in common.

Measuring closeness: the angle between arrows

Draw each embedding as an arrow from the centre to its point. Two texts are similar when their arrows point in nearly the same direction.

The usual measure is cosine similarity: the cosine of the angle between the two arrows.

Angle between arrowsCosine similarityMeaning
0 degrees: same direction1Same meaning
90 degrees: at right angles0Unrelated
180 degrees: opposite-1As different as can be

Here is a question and six documents. To make it drawable they have been flattened onto a circle and placed by hand. The similarity scores are the real formula applied to those arrows.

Search by meaning: the nearest arrow winsInteractive
123456Q

QHow do I reset my password?

1Steps to recover access to your account

2Forgot your login? Start here

3How to choose a strong password

4Reset your router to factory settings

5Our refund policy

6Pasta for two in 20 minutes

Start. Every text has been turned into an arrow. Texts about similar things point in similar directions. The pink arrow is the question.

The winner shares no key word with the question. Two documents that do share a word, "reset" and "password", are ranked lower, because they are about routers and password strength.

How a search works

  1. Ahead of time: embed every document

    Run each document through the embedding model and store its vector.

  2. When a question arrives: embed the question

    With the same model.

  3. Find the nearest vectors

    Compare the question's vector with the stored ones and take the closest few.

  4. Return those documents

    The nearest arrow wins.

python
question = embed("How do I reset my password?")

scores = [(cosine_similarity(question, vector), text) for text, vector in stored]
best = sorted(scores, reverse=True)[:3]

With millions of documents, comparing against every one is too slow. A vector database stores the vectors in a structure that finds near neighbours quickly without checking them all.

Where you will meet them

  • Semantic search: find documents by meaning, as above.
  • RAG: fetch relevant passages to give a language model before it answers. See RAG or Fine-Tuning.
  • Recommendations: "more like this" is "nearest vectors to this".
  • Grouping: cluster thousands of support tickets by topic without reading them.
  • Inside the language model itself: the first thing a model does with each token is look up its embedding.

Key takeaways

  • An embedding is a list of numbers that represents the meaning of a piece of text.
  • Texts with similar meanings get vectors that point in similar directions.
  • Cosine similarity measures the angle between two vectors: 1 is the same direction, 0 is unrelated.
  • Semantic search embeds the question and returns the nearest stored vectors.
  • The best match may share no words with the question.

Your progress

Check your progress

0%

  1. Read the page50% of this page

    When you have gone through the page and its animations, tick it off.

  2. Take the quiz50% of this page

    7 questions, from easy to hard. Take it as often as you like: the questions change, and only your best score counts.

Saved on this device only. See all your progress