Pointer Walk

Temperature and Sampling

The same prompt gives a different answer. Temperature reshapes the odds before the next token is drawn.

AI and Machine LearningBeginner8 min read

Same prompt, different answer

Ask a chatbot the same question twice and you often get two different answers. The model did not change in between. So where does the difference come from?

At each step, a language model does not output a word. It outputs a chance for every possible next token. Then one token is drawn at random according to those chances, a bit like spinning a wheel where likely tokens have bigger slices.

That draw is where the variety comes from. And one setting controls how even or uneven the slices are: temperature.

What temperature does

For the prompt "The sky is", the model might give these raw scores: high for "blue", a bit lower for "clear" and "grey", low for "cheese".

Temperature is a number the scores are divided by before they are turned into chances.

  • Below 1: the gaps between scores are stretched. The favourite gets an even bigger share.
  • Exactly 1: the scores are used as they are.
  • Above 1: the gaps are squeezed. Unlikely words get a real chance.

Pick a temperature and watch the same prompt run 16 times.

"The sky is ___" at temperature 0.7Interactive
Temperature
78.4%
blue
14.1%
clear
6.0%
grey
1.1%
falling
0.3%
green
0.1%
cheese

What was picked, run by run

Start. Temperature 0.7. The bars are the chance of each word coming next. Now run the same prompt 16 times and see what gets picked.

Runs
0 / 16
Different words picked
0

Example

The same scores at four temperatures

At 0.2, "blue" has a 99.7% chance. Every run says the sky is blue.

At 0.7, "blue" has 78%, "clear" 14%, "grey" 6%.

At 1.0, "blue" has 65%, "clear" 20%, "grey" 11%, and "falling" appears at 3%.

At 1.5, "blue" is down to 50% and even "cheese" has almost a 2% chance.

The arithmetic, in full:

python
import math

def chances(scores, temperature):
    scaled = [math.exp(score / temperature) for score in scores]
    total = sum(scaled)
    return [value / total for value in scaled]

Choosing a temperature

TemperatureBehaviourGood for
0 to 0.3Focused and repeatableExtracting data, classifying, code, anything with one right answer
0.5 to 0.8Mostly sensible, some varietyGeneral chat, explanations, summaries
0.9 to 1.2Varied, sometimes surprisingBrainstorming, fiction, names and slogans
HigherIncreasingly randomRarely useful: text starts to fall apart

Low temperature does not make a model more correct. It makes it more consistent. If the most likely answer is wrong, a low temperature gives you that wrong answer every time.

Top-p: cutting off the tail

Even a small chance comes up eventually. Over a long answer, one very unlikely word can derail the whole thing. Top-p, also called nucleus sampling, removes that risk.

  1. Sort the tokens from most to least likely

    As always.

  2. Add up their chances from the top until the total reaches p

    With p = 0.9, keep going until you have covered 90%.

  3. Throw away everything below that line

    Those tokens cannot be picked at all.

  4. Draw from what is left

    Rescaled so it adds up to 100% again.

Example

Top-p 0.9 on the sky

At temperature 1: "blue" 65% + "clear" 20% = 85%. Not yet 90, so add "grey" at 11%: 96%. Stop.

Only those three words can be chosen. "falling", "green" and "cheese" are cut, however many times you run it.

Temperature reshapes all the chances. Top-p removes the unlikely end entirely. They are often used together, and a common piece of advice is to adjust one and leave the other at its default.

Key takeaways

  • A model outputs a chance for every next token, and one is drawn at random.
  • Temperature divides the scores first: low makes the favourite dominate, high evens things out.
  • Low temperature gives consistent output, not necessarily correct output.
  • Top-p keeps only the most likely tokens that together cover a share p, and discards the rest.
  • Use low settings for tasks with one right answer and higher ones for variety.

Your progress

Check your progress

0%

  1. Read the page50% of this page

    When you have gone through the page and its animations, tick it off.

  2. Take the quiz50% of this page

    7 questions, from easy to hard. Take it as often as you like: the questions change, and only your best score counts.

Saved on this device only. See all your progress