Same prompt, different answer
Ask a chatbot the same question twice and you often get two different answers. The model did not change in between. So where does the difference come from?
At each step, a language model does not output a word. It outputs a chance for every possible next token. Then one token is drawn at random according to those chances, a bit like spinning a wheel where likely tokens have bigger slices.
That draw is where the variety comes from. And one setting controls how even or uneven the slices are: temperature.
What temperature does
For the prompt "The sky is", the model might give these raw scores: high for "blue", a bit lower for "clear" and "grey", low for "cheese".
Temperature is a number the scores are divided by before they are turned into chances.
- Below 1: the gaps between scores are stretched. The favourite gets an even bigger share.
- Exactly 1: the scores are used as they are.
- Above 1: the gaps are squeezed. Unlikely words get a real chance.
Pick a temperature and watch the same prompt run 16 times.
What was picked, run by run
Start. Temperature 0.7. The bars are the chance of each word coming next. Now run the same prompt 16 times and see what gets picked.
- Runs
- 0 / 16
- Different words picked
- 0
Example
The same scores at four temperatures
At 0.2, "blue" has a 99.7% chance. Every run says the sky is blue.
At 0.7, "blue" has 78%, "clear" 14%, "grey" 6%.
At 1.0, "blue" has 65%, "clear" 20%, "grey" 11%, and "falling" appears at 3%.
At 1.5, "blue" is down to 50% and even "cheese" has almost a 2% chance.
The arithmetic, in full:
import math
def chances(scores, temperature):
scaled = [math.exp(score / temperature) for score in scores]
total = sum(scaled)
return [value / total for value in scaled]Choosing a temperature
| Temperature | Behaviour | Good for |
|---|---|---|
| 0 to 0.3 | Focused and repeatable | Extracting data, classifying, code, anything with one right answer |
| 0.5 to 0.8 | Mostly sensible, some variety | General chat, explanations, summaries |
| 0.9 to 1.2 | Varied, sometimes surprising | Brainstorming, fiction, names and slogans |
| Higher | Increasingly random | Rarely useful: text starts to fall apart |
Low temperature does not make a model more correct. It makes it more consistent. If the most likely answer is wrong, a low temperature gives you that wrong answer every time.
Top-p: cutting off the tail
Even a small chance comes up eventually. Over a long answer, one very unlikely word can derail the whole thing. Top-p, also called nucleus sampling, removes that risk.
Sort the tokens from most to least likely
As always.
Add up their chances from the top until the total reaches p
With p = 0.9, keep going until you have covered 90%.
Throw away everything below that line
Those tokens cannot be picked at all.
Draw from what is left
Rescaled so it adds up to 100% again.
Example
Top-p 0.9 on the sky
At temperature 1: "blue" 65% + "clear" 20% = 85%. Not yet 90, so add "grey" at 11%: 96%. Stop.
Only those three words can be chosen. "falling", "green" and "cheese" are cut, however many times you run it.
Temperature reshapes all the chances. Top-p removes the unlikely end entirely. They are often used together, and a common piece of advice is to adjust one and leave the other at its default.
Key takeaways
- A model outputs a chance for every next token, and one is drawn at random.
- Temperature divides the scores first: low makes the favourite dominate, high evens things out.
- Low temperature gives consistent output, not necessarily correct output.
- Top-p keeps only the most likely tokens that together cover a share p, and discards the rest.
- Use low settings for tasks with one right answer and higher ones for variety.