how sampling works

What you’re actually looking at

The final layer of a transformer emits one raw score — a logit — for every token in its vocabulary. Logits are unbounded real numbers; on their own they aren’t probabilities. To turn them into probabilities that sum to 1, you run them through softmax:

p_i = exp(logit_i) / Σ_j exp(logit_j)

Exponentiating makes every value positive and blows up the gaps: a logit that’s a couple points higher than its neighbors ends up owning a lopsided share of the probability mass. That’s the tall bar you see for left.

Temperature: dividing before you exponentiate

Temperature T is a single number you divide the logits by before the softmax:

p_i = exp(logit_i / T) / Σ_j exp(logit_j / T)

That’s the entire mechanism — no randomness lives here, just a rescale. But it does two opposite things depending on which side of 1 you’re on:

• T < 1 divides by something small, stretching the logit gaps wider. Softmax then concentrates even harder on the leader. The distribution gets peaky and confident.
• T > 1 shrinks the gaps, so softmax spreads mass toward the tail. The long shots wake up and the output gets more surprising.
• T = 1 is the raw distribution, untouched.

Drag temperature to 0.1 and the model becomes nearly greedy — one bar swallows the chart. Push it to 2.0 and the tail floods with mass.

temperature only rescales — the ordering of tokens never changes, no matter how hot it gets.

Top-k and top-p: cutting the tail

Even a well-shaped distribution has a long tail of low-probability tokens. Any single one is unlikely, but collectively they carry enough mass that, sampled often enough, the model will eventually blurt out something incoherent. Truncation sampling removes that tail before the roll. Two ways to draw the line:

• Top-k keeps a fixed count — the k highest tokens — and zeroes the rest. Simple, but blunt: k=5 is stingy when the model is genuinely uncertain across 40 good options, and wasteful when it’s certain of just one.
• Top-p (nucleus sampling) keeps a fixed probability mass instead: sort tokens high to low, walk down accumulating probability, and stop the moment the running total crosses p. When the model is confident, that’s one or two tokens; when it’s unsure, the nucleus widens to include many. The cutoff adapts to the shape of the distribution — which is exactly why it was proposed (Holtzman et al., 2019) as a fix for top-k’s rigidity.

After either cut, the survivors are renormalized so their probabilities sum back to 1 — that’s why the kept bars grow when you tighten a knob: they’re dividing up the mass the cut tokens gave back. In the playground, cut tokens don’t vanish; they go ghosted and hatched, tagged with which knob did the cutting, so you can see the mass that got thrown away. Set both, and a token is cut if either rule rejects it.

Myths worth retiring

• “Temperature 0 is deterministic.” Mostly, with an asterisk. You can’t literally divide by zero, so implementations special-case T = 0 to mean “skip sampling, take the argmax” (greedy decoding). That’s deterministic in principle — but the same prompt can still vary run to run because of floating-point non-associativity, GPU kernel scheduling, and batching. Determinism is a property of the whole stack, not just the temperature field.
• “Higher temperature = more creative.” It’s more random, which isn’t the same thing. Past a point you’re not unlocking creativity, you’re just sampling from the incoherent tail.
• “Top-p and top-k are alternatives you pick between.” They stack. Many production configs apply a generous top-k as a hard ceiling and let top-p do the adaptive trimming underneath it.

The takeaway: sampling settings don’t change what the model knows — the logits are fixed the instant the forward pass ends. They change how much of that knowledge you let leak into the roll.