termEngineering & architecture

Sampling Temperature

Sampling temperature adjusts how strongly a generative model favors higher-scoring next tokens when sampling an output.

Download Markdown (.md)

01What it means

Temperature is a decoding parameter that rescales token scores before sampling. Lower positive values concentrate probability on favored tokens; higher values spread it more broadly. Some interfaces expose zero as a special setting for greedy selection. The term describes output sampling, not factual confidence, and differs from temperature scaling used to calibrate predictive probabilities.[1][2]

Token probability concentration Lower temperature concentrates choices; higher temperature spreads them
  1. 01Greedy / zeroSpecial deterministic selection where supported
  2. 02LowerProbability concentrates on favored tokens
  3. 03ModerateBroader alternatives enter sampling
  4. 04HigherDistribution spreads; variation and failure risk rise

Temperature controls sampling variation, not factual confidence. Supported ranges and zero behavior depend on the model interface.

02Why it matters

The setting affects variation between responses and can change task quality. A writing assistant may benefit from varied candidates, while extraction requires consistent behavior. Neither a low setting nor repeated agreement establishes that an answer is correct.

03When to use it

Tune temperature when the chosen model exposes the control and evaluation shows that response variation matters. Follow model-specific guidance: supported ranges, defaults, and recommended settings vary. Use validation and deterministic code for guarantees rather than relying on a sampling parameter.

04How it works

  1. Define whether the task values variation, repeatability, or another measurable quality.
  2. Record the model version, prompt, supported temperature range, and other decoding controls.
  3. Compare a small set of settings across representative inputs, collecting multiple responses for each.
  4. Assess correctness, repetition, useful diversity, and downstream failures rather than judging writing style alone.
  5. Deploy the evaluated configuration and retest when model versions or task requirements change.

05A practical example

Example

A product team generates alternative onboarding headlines. It compares several sampling settings on the same briefs, reviewing whether candidates remain accurate and distinct. The selected setting provides useful alternatives, while a separate check rejects unsupported promises; temperature is not used as that check.

06Where it breaks

The same numeric setting is not directly comparable across models. Top-p filtering and other decoding choices interact with sampling. Greedy selection can still vary across serving conditions, and reducing variation can preserve a consistently wrong answer.

Common mistakes

  • Describing temperature as a creativity percentage or a truthfulness score.
  • Promising identical outputs merely because temperature is zero.
  • Changing several decoding settings together without isolating effects.
  • Confusing sampling temperature with probability calibration.

07Sources

  1. GenerationHugging Face · Official documentationGeneration
  2. Content generation parametersGoogle Cloud · Official documentationContent generation parameters
View saved topics

Start typing to search titles, definitions, categories, and frameworks.