how sampling works

what you’re looking at

The final layer of a transformer emits one raw score — a logit — for every token in its vocabulary. Logits are unbounded real numbers; on their own they aren’t probabilities. To turn them into probabilities that sum to 1, you run them through softmax:

p_i = exp(logit_i) / Σ_j exp(logit_j)

Exponentiating makes every value positive and widens the gaps: a logit that’s a couple points higher than its neighbors ends up owning a lopsided share of the probability mass. That’s the tall bar you see for left.

temperature: dividing before you exponentiate

Temperature T is a single number you divide the logits by before the softmax:

p_i = exp(logit_i / T) / Σ_j exp(logit_j / T)

That’s the entire mechanism — no randomness lives here, just a rescale. But it does two opposite things depending on which side of 1 you’re on:

• T < 1 divides by something small, stretching the logit gaps wider. Softmax then concentrates even harder on the leader. The distribution gets peaky and confident.
• T > 1 shrinks the gaps, so softmax spreads mass toward the tail. Low-probability tokens gain share and the output gets more surprising.
• T = 1 is the raw distribution, untouched.

Drag temperature to 0.1 and the model becomes nearly greedy — the leading token takes almost all of the mass. Push it to 2.0 and the tail carries a large share.

top-k and top-p: cutting the tail

Even a well-shaped distribution has a long tail of low-probability tokens. Any single one is unlikely, but collectively they carry enough mass that, sampled often enough, an incoherent token will eventually be drawn. Truncation sampling removes that tail before the draw. Two ways to set the cutoff:

• Top-k keeps a fixed count — the k highest tokens — and zeroes the rest. The count ignores the shape of the distribution: k=5 keeps too few tokens when the model is uncertain across 40 good options, and too many when it is certain of one.
• Top-p (nucleus sampling) keeps a fixed probability mass instead: sort tokens high to low, walk down accumulating probability, and stop the moment the running total crosses p. When the model is confident, that’s one or two tokens; when it’s unsure, the nucleus widens to include many. The cutoff adapts to the shape of the distribution — which is exactly why it was proposed (Holtzman et al., 2019) as a fix for top-k’s rigidity.

After either cut, the survivors are renormalized so their probabilities sum back to 1 — that’s why the kept bars grow when you tighten a knob: they’re dividing up the mass the cut removed. In the playground, cut tokens don’t vanish; they go ghosted and hatched, tagged with which knob did the cutting, so you can see the mass that got thrown away. Set both, and a token is cut if either rule rejects it.

myths worth retiring

• “Temperature 0 is deterministic.” Mostly. You can’t literally divide by zero, so implementations special-case T = 0 to mean “skip sampling, take the argmax” (greedy decoding). That’s deterministic in principle — but the same prompt can still vary run to run because of floating-point non-associativity, GPU kernel scheduling, and batching. Determinism is a property of the whole stack, not just the temperature field.
• “Higher temperature = more creative.” It’s more random, which isn’t the same thing. Past a point you’re not adding creativity; you’re sampling from the incoherent tail.
• “Top-p and top-k are alternatives you pick between.” They stack. Many production configs apply a large top-k as a hard cap and let top-p do the adaptive trimming underneath it.

Sampling settings don’t change what the model knows — the logits are fixed the instant the forward pass ends. They change which part of the distribution the sample can come from.