How the next token is chosen

final selection is a dot product, a softmax, and a draw

Everything so far produced numbers. Turning them into a word takes one more stage, and it is the stage most people never see.

On the France question, the last position scores every token the model could say next. Paris scores +5.0 and Marseille +2.0.

After the last block, exactly one position matters: the final one, the position that has to say what comes next. Its finished thought state h is compared against a second stored table that plays the embedding lookup in reverse: every token in the vocabulary has a stored vector of its own in that table, its candidate vector, written u_A for candidate A, and the score for that candidate is the dot product of the two — multiply h against u_A slot by slot, add up the results, one number out. Fifty thousand entries in the table means fifty thousand scores. Those raw scores are logits, and the table is the unembedding table. In most large models it is its own set of numbers, as large again as the embedding table below the stack; some smaller models save the space by reading one table both ways.

Read that as: the logit for candidate A is how closely the model’s final thought state matches the identity of that word. It is the same arithmetic the two machines in the block were running.

Logits are unbounded and hard to compare: one might be 14.2 and another −3.8, and neither number means anything on its own. To choose among them they have to become proportions, and the step that does it is softmax: it keeps the scores in the same order, turns them into fractions adding up to one, and exaggerates the gaps between them along the way. What comes out is a probability distribution across every token the model could say next.

Softmax does two things to a list of scores, and both are worth seeing on the France question. Here are four candidates and their logits:

Token          Logit (ℓ)   Softmax probability (p)
──────────────────────────────────────────────────────────────────
"Paris"          +5.0      93.5%  [███████████████████████████████]
"Marseille"      +2.0       4.7%  [█]
"London"         +1.0       1.7%  [░]
"Apple"          -2.0       0.1%  [░]

First, it suppresses negative scores. Each logit is raised to a power of e before the shares are taken, and a negative score like −2.0 becomes about 0.135, so a candidate the thought state points away from ends up with a share near zero rather than a negative one. Second, it amplifies small leads. The exponential curve steepens fast, so a gap of only three logits between Paris at +5.0 and Marseille at +2.0 becomes a probability advantage of about twenty to one, because e cubed is roughly 20. That is the number to remember about softmax: three points of logit is a twenty-to-one lead. Small linear differences in the score turn into wide margins in the draw.

For our C-40 question with the policy present, the distribution comes out just as lopsided: 45 takes almost all the weight, a few plausible alternatives take slivers, and the rest of the vocabulary shares what is left. With the policy absent it is flatter and centered somewhere else, with 30 on top and 14 and 45 and 60 holding real shares behind it.

Then one token is drawn from the distribution. That draw is sampling, and it is the one place in the pass where chance is put in on purpose.

Temperature governs the draw. The number divides the logits before softmax runs, which sounds like a technicality and behaves like a dial on how adventurous the model is. Turn it down and the gaps between scores get exaggerated further, so the leading token takes even more of the weight and the long shots are squeezed out. Turn it up and the gaps flatten toward an even spread, so tokens that were barely in contention get a real chance of being drawn, and at a high enough setting even a token the pass scored near zero can come up. At temperature 0 there is no draw at all: the highest-scoring token wins every time, which is called greedy decoding. That removes the random draw and nothing else. The same prompt can still come back different across runs, deployments and model versions, because the arithmetic of the pass itself can vary on a busy server, where the size of the batch your request is computed in changes with the load.

That last line is worth having in your hands, because it is the most common mistake made with this dial:

Temperature governs the draw, not the ranking. It can make a model bolder or duller about picking a low-scoring token, and it never changes which token the pass scored highest. Turned high enough, it raises every long shot together, the wrong ones with the one you needed, so it cannot single out a fact the pass scored near zero. When the fact you needed was never near the top of the distribution, no temperature setting will find it.

So a wrong answer that stays wrong at temperature 0 is not a randomness problem, and turning the dial down is not the fix. You have learned something instead: that answer sat at the top of the distribution on every one of those runs, and the repair has to change what the model is drawing from, not how it draws.

That is the whole trip, and it is worth seeing laid out now that you have walked it:

the autoregressive pass repeats for every token

We now have one token. 45, drawn from a distribution built by a complete pass through every number in the model.

The answer is not finished, so the whole thing runs again.

The chosen token is appended to the text, and the model makes another pass to produce the token after that. This pass runs one new position, the token just written, through every block, and at every block that position’s attention reads everything before it: your policy, the customer’s question, and the one token written so far. Then a third pass for the token after that. This is what autoregressive means, and the arithmetic is unforgiving: a hundred-token answer is a hundred passes, each one attending to everything that came before it.

Module 1.01 said the prompt is read once and each new token reuses work stored for the earlier ones. Here is that work. When a request arrives, the model runs the whole prompt through the stack in one pass, every position at once, which is the run the full-pass figure above draws. That first pass is the prefill. Each token after it costs one decoding step: one new position, the token just drawn, climbs through every block, and at each block its attention reads the per-position numbers attention compares against and copies from, the keys and values, at every earlier position. Those keys and values were computed when the earlier positions went through that same block, and computing them again for every new token would repeat the prefill each time.

A serving engine keeps them instead, in the KV cache: at every block it stores the keys and values of every position processed so far, and each decoding step appends the new position’s own pair. The newest position still attends to all sixty-eight positions of the prompt, your policy and the customer’s question, and to every token of the answer written so far; what is not repeated is the arithmetic that produced their keys and values. The cache holds exactly what the text would rebuild: recompute from the same text and the same keys and values come out. It saves time and adds nothing the text does not already determine, and it changes no weight. It is inference state, kept while this answer is being written and dropped when the request ends. Some serving systems also keep the cache for a prompt’s opening text and reuse it when a later request begins with exactly the same text; that reuse is keyed on identical text and holds only what that text determines. Anything a product appears to remember about you across sessions is a separate system that saved your text and placed it in a later prompt.

That is why a model’s own written output is the only working memory it can add to: the cache holds nothing the text does not determine, and it hands earlier work forward only partway. A block’s attention reads the keys and values that the same block computed, so a result an earlier position worked out high in the stack reaches later positions only in the blocks at that height and above. A result written out as a token enters every later position at the bottom of the stack, as part of the text, and every block above can build on it. Anything the model needs to keep, it has to say.

two kinds of number: weights and activations

Look back at what the pass touched, because it divides cleanly in two.

You ran a model this morning on one prompt, and tonight you run the same model on a different one.

One set is the weights, fixed the day training ended and untouched since. The tokenizer’s entries, the embedding table, every block’s stored patterns, and the unembedding table that turns the thought state into vocabulary scores: on the order of a hundred billion numbers sitting in a file, a few hundred gigabytes that would fill a laptop’s whole drive, identical for you and for everyone else querying that model today. Nothing in an inference pass writes to them. The model you run tonight computes exactly what it computed this morning.

The other set is the activations: the vectors built from your text for this run and discarded when the answer ends. Your thirteen positions and their residual streams are activations, and so is the thought state at the last position that they end in. They are yours alone, and they last one request.

That text can arrive from more places than you might expect. You typed some of it. Someone may have pasted a document above it, a retrieval system may have gone looking and dropped three paragraphs in, or a tool may have run and returned its result a moment ago. Above all of it may sit a system prompt you never see. None of it arrives labeled. By the time the model reads it, it is one flat sequence of text, and the activations are built from all of it at once.

Two sets of numbers: the weights are stored and shared by everyone; the activations are yours alone and last one request. When someone says a model “knows” something, ask which set they mean. The answer changes what you should do about it.

Our three tokens now sort themselves. ␣days came out of the weights applying a rule to the activations. 45 came out of the activations, carried across by a rule in the weights. And 30 came out of the weights alone, filling a slot the activations had nothing to say about.

Hold the three failure modes beside that. A language model computes one thing, over and over: the next token, and it goes wrong in three ways. It confabulates: asked for something it does not hold, it produces a fluent answer shaped exactly like a true one. It has a grounding gap, having read every description of water ever written without ever being wet. And it carries systemic bias, reproducing the skews of the text it learned from.

On your first day, watching for three separate failures is exactly the right habit. Underneath, they are one mechanism seen from three angles, and you have just watched that mechanism produce 30 without hesitating. What we have not done is the arithmetic. Every comparison in this module, every “lands hard against” and “nothing matched,” is one operation repeated a few hundred billion times per word, and it is the same operation every time. Set that count against something you can picture: one word costs about as many multiply-and-add steps as there are stars in the Milky Way.

what you can do with this today

Ask the model to write out intermediate steps. The model makes one pass for every token it writes. Work done inside the stack at one position reaches later positions through attention only partway up, while a result written out as text enters every later pass at the bottom of the stack. The model’s written output is the only working memory it can add to: when a task needs intermediate results, ask the model to write them out.

Ask for the length the task needs. A hundred-token answer is a hundred passes, each one attending to everything before it. A serving engine’s KV cache saves recomputing positions already processed, but every new token is still a full pass through every block. The length of an answer is compute you pay for, token by token, so ask only for what the task requires.

Put essential facts into the request. The weights are fixed at training, identical for every user, and nothing in an inference pass writes to them. The activations are built from the text of this one request and discarded when it ends. A fact your agent must use belongs in the text of the request, where it becomes activations, rather than being hoped for in the weights.

Before you go on, put the habit to work once. Take an answer a model gave you today, pick one token in it, and say which set supplied it: the weights everyone shares, or the activations built from your prompt. In our next lesson we do the arithmetic behind that call, and after that we build the three sources of the thought state one at a time.