Pointer Walk

Context Windows and Memory

Models do not remember your chat. Each message is sent with the whole conversation so far, and the app is what remembers.

AI and Machine LearningBeginner7 min read

"It remembers what I said"

You tell a chatbot your name. Three messages later it uses it. It certainly looks as if the model remembered.

It did not. A language model has no memory between requests. Every time it is called, it starts from nothing, reads the text it is given, and produces a reply. Then it forgets everything.

So how does the conversation hold together?

The app remembers. The model re-reads.

Two messages in one chatInteractive
You
Chat app
saved: nothing
Model
remembers: nothing

Watch what is actually sent to the model each time.

  1. You send a message

    The chat app saves it in its own storage.

  2. The app sends the model the whole conversation so far

    Every message from both sides, in order, as one long piece of text.

  3. The model reads all of it, from the top, and writes a reply

    It has never seen any of it before.

  4. The app saves the reply and waits for your next message

    Then the cycle repeats, with a longer conversation each time.

Your name is not stored in the model. It is in message 1, and message 1 is sent again every single time.

The proof: open a new chat

If the model itself remembered you, it would remember you everywhere.

The same question in a new chatInteractive
You
Chat app
saved: nothing
Model
remembers: nothing

Watch what is actually sent to the model each time.

Same model, same question. The only difference is the text it was handed.

Two messages in one chatInteractive
Scenario
You
Chat app
saved: nothing
Model
remembers: nothing

Watch what is actually sent to the model each time.

The context window

The text a model is given in one request is called its context. There is a hard limit on how much it can take, measured in tokens. That limit is the context window.

Everything has to fit inside it together:

  • the instructions the app gives the model behind the scenes,
  • the whole conversation so far,
  • any documents or files you attached,
  • and room for the reply.

Example

Why long chats drift

A long conversation eventually no longer fits. The app has to do something: drop the oldest messages, or replace them with a short summary.

From that moment the model genuinely cannot see what you said at the start. It has not become forgetful. Those words are simply no longer in the text it is given.

Two more things follow from "everything is re-sent every time".

  • Cost and speed. The tenth message of a chat makes the model read all nine before it. Long conversations cost more per message than short ones.
  • Attention is spread thin. A model can lose track of details buried in the middle of a very long context, even when they technically fit.

What "memory" features really are

Some assistants do recall things from earlier conversations. This is still not the model remembering. It is the same trick with one more step:

  1. The app notes down facts worth keeping

    For example "the user's name is Sam" or "prefers short answers".

  2. It stores those notes in an ordinary database

    Outside the model.

  3. At the start of each new chat, it pastes the relevant notes into the context

    The model reads them like any other text.

The model's knowledge from training is fixed at the time it was trained. Anything newer or more personal reaches it one way only: by being written into the context.

Key takeaways

  • A language model keeps nothing between requests. It starts from zero every time.
  • The chat app stores the conversation and sends the whole thing with every new message.
  • The context window is the maximum amount of text the model can be given at once.
  • When a conversation outgrows it, old messages are dropped or summarized, and the model can no longer see them.
  • Memory features work by saving notes outside the model and pasting them back into the context.

Your progress

Check your progress

0%

  1. Read the page50% of this page

    When you have gone through the page and its animations, tick it off.

  2. Take the quiz50% of this page

    6 questions, from easy to hard. Take it as often as you like: the questions change, and only your best score counts.

Saved on this device only. See all your progress