LLM Sampling: Temperature, Top-p and Why Hallucinations Happen

Ask a museum assistant bot when Thursday late opening ends and you may hear 8pm. Ask again and you may hear 9pm. Neither answer is a typo. The model produced a probability for each plausible time, and a sampler rolled weighted dice. The same dice explain both creative variety and confidently stated falsehoods, so this lesson treats them together.

From scores to a choice

At each step the network outputs one raw score, called a logit, per vocabulary token. Softmax turns the scores into probabilities. Temperature divides the scores before softmax. A temperature below 1 sharpens the distribution (the favourite gets even more probability); above 1 flattens it (unlikely tokens get a real chance). Top-p, or nucleus sampling, keeps only the smallest group of top tokens whose probabilities add up to p, renormalises, and samples from that group, cutting off the long tail of nonsense.

The scores below are invented for a toy question, but the arithmetic is the real thing. The seed makes the random draws repeatable; yours may differ slightly across Python versions.

import math
import random

# Made-up scores ("logits") a model might give to candidate next words after:
# "The museum late opening on Thursdays ends at ..."
logits = {"8pm": 3.0, "9pm": 2.2, "7pm": 1.5, "late": 0.6, "midnight": -0.5}

def softmax_with_temperature(logits, temperature):
    scaled = [v / temperature for v in logits.values()]
    peak = max(scaled)
    exps = [math.exp(s - peak) for s in scaled]
    total = sum(exps)
    return {tok: e / total for tok, e in zip(logits, exps)}

def top_p_filter(probs, p):
    """Keep the smallest set of top tokens whose probabilities add up to >= p, then renormalise."""
    kept, running = {}, 0.0
    for tok, pr in sorted(probs.items(), key=lambda kv: kv[1], reverse=True):
        kept[tok] = pr
        running += pr
        if running >= p:
            break
    total = sum(kept.values())
    return {tok: pr / total for tok, pr in kept.items()}

for t in (0.2, 1.0, 2.0):
    probs = softmax_with_temperature(logits, t)
    print(f"temperature {t}: " + "  ".join(f"{k}={v:.2f}" for k, v in probs.items()))

print()
nucleus = top_p_filter(softmax_with_temperature(logits, 1.0), 0.9)
print("top-p 0.9 keeps:", {k: round(v, 2) for k, v in nucleus.items()})

rng = random.Random(42)                      # fixed seed so the run is repeatable
for t in (0.2, 1.0, 2.0):
    probs = softmax_with_temperature(logits, t)
    draws = rng.choices(list(probs), weights=list(probs.values()), k=1000)
    counts = {tok: draws.count(tok) for tok in logits}
    print(f"1000 draws at temperature {t}: {counts}")

temperature 0.2: 8pm=0.98  9pm=0.02  7pm=0.00  late=0.00  midnight=0.00
temperature 1.0: 8pm=0.56  9pm=0.25  7pm=0.12  late=0.05  midnight=0.02
temperature 2.0: 8pm=0.38  9pm=0.26  7pm=0.18  late=0.12  midnight=0.07

top-p 0.9 keeps: {'8pm': 0.6, '9pm': 0.27, '7pm': 0.13}
1000 draws at temperature 0.2: {'8pm': 978, '9pm': 20, '7pm': 2, 'late': 0, 'midnight': 0}
1000 draws at temperature 1.0: {'8pm': 561, '9pm': 262, '7pm': 114, 'late': 49, 'midnight': 14}
1000 draws at temperature 2.0: {'8pm': 384, '9pm': 247, '7pm': 184, 'late': 102, 'midnight': 83}

At temperature 0.2 the favourite wins about 98 percent of draws in this toy, so the answer is nearly deterministic. At 1.0 the model sometimes says 9pm or 7pm, in proportion to how plausible it found them. At 2.0 even "midnight" appears regularly. Top-p 0.9 removes the two least likely options entirely, so they can never be chosen however many times you sample.

Which settings exist, their ranges and their defaults vary by provider and model, so check the documentation. A common recommendation is to adjust temperature or top-p, not both at once, so that you can tell which one changed the behaviour.

Why hallucination happens

A hallucination is fluent output that is wrong or unsupported by any source: a function that does not exist in the library, a citation to a paper nobody wrote, a policy your company never had. It is tempting to see this as a glitch. It follows from how generation works.

• The objective is plausible text, not true text. Training rewards predicting likely continuations. A confident, well-formed wrong answer scores well on plausibility.
• Facts are stored as blurry patterns. Frequent facts are well represented; rare ones (an obscure API flag, a small company's return policy) are weak, and the model fills the gap with something shaped like the truth.
• Answering is the default. In training data, questions are usually followed by answers, not by "I do not know", so a model needs extra tuning to abstain, and that tuning is imperfect.
• Errors compound. Each token is conditioned on the ones before it, including wrong ones. One invented detail makes the next invented detail more likely, so a single bad choice early can snowball.
• Sampling can pick an unlikely token. Higher randomness raises the chance of a wrong turn, but at any temperature, the most likely continuation can simply be wrong.

What actually reduces hallucination

Here is what grounding plus abstention looks like as a prompt. The wording is only a starting point, and you still have to test it, including with questions the passages cannot answer.

Answer the question using ONLY the passages between <passages> tags.
After each claim, quote the exact words from the passage that support it.
If the passages do not contain the answer, reply exactly:
"I cannot find that in the provided documents."

<passages>
{{retrieved_passages}}
</passages>

Question: {{question}}

Common mistakes and the symptom each one causes

• Setting temperature to 0 and calling it fixed. Symptom: the same wrong answer, delivered consistently. Fix: ground and verify.
• High temperature on extraction or JSON tasks. Symptom: fields with creative spellings and an occasional parse failure. Fix: low temperature, plus validation.
• Changing temperature and top-p together. Symptom: you cannot tell which knob fixed or broke the behaviour. Fix: change one at a time against a test set.
• Trusting generated citations or URLs. Symptom: a reference that looks perfect and leads nowhere. Fix: check that sources exist and say what is claimed.
• Testing with one sample. Symptom: it "works" once and fails one run in ten. Fix: run each test input several times.

How this shows up in real projects and interviews

Almost every production design that uses a model for facts also has a grounding or verification step, because hallucination is the failure customers notice first. Interviewers ask "what is temperature?" as a warm-up and then "how would you reduce hallucinations in a support bot?". A strong answer separates sampling (a knob) from knowledge (missing or stale), proposes retrieval from approved documents, abstention rules, output validation, an evaluation set that includes unanswerable questions, and human review for risky cases. Avoid promising that any technique eliminates hallucination. The next lesson turns to the practical side: calling models through APIs and keeping latency and cost under control.