Token by token

A language model writes one token, and then it starts over. The token it just wrote is added to the text, and the whole machine runs again from the first character to produce the one after that. A hundred-word answer is a few hundred separate acts of writing, each one a complete trip through every number the model has.

Every one of those trips has the same shape, and it is worth stating before we take one apart. At each step the model reads the entire prompt, updates one central numerical state, compares that state against every word in its vocabulary, and draws the next token from the comparison. Read, update, compare, draw. The state in the middle is a long list of numbers we will call the thought state, written h, and everything the model has read, remembered, or worked out has to be in it, because that list is the only thing the vocabulary is ever compared against.

We take one of those trips now, start to finish, and we bring two questions along.

The first is one you could ask any model, cold: “What is the capital of France?” Seven tokens in, and the answer, Paris, comes back at once. Nothing in the prompt says Paris. The fact arrives out of the model’s own stored numbers, because the pairing of France with Paris was in the training text millions of times, and we will watch the machinery hand it over.

The second question is built so that the same route is closed. Imagine you run a company called Aperture Instruments, which sells a digital camera called the C-40. A customer writes in asking how long they have to send one back, and you have pasted your returns policy above their question so the model can see it:

Returns policy, rev. 2026-02: Aperture Instruments accepts returns
of the C-40 within 45 days of delivery. Units returned after day 30
incur a 15% restocking fee. Opened lens kits are non-returnable.

How many days do I have to return the C-40?

The company is invented, and that is exactly what makes it useful. No model has ever read a word about the C-40, so nothing can arrive out of the model’s memory without us noticing it arrive. Paris comes from memory; 45 has to come from the page. Two questions, one machine, and by the end of this part you will be able to say, for any token in an answer, which of the two supplied it.

The answer we want to the second question is You have 45 days to return it. Three tokens of that sentence are worth following, because each one gets into the answer by a different route:

• the s in “days” settles where it stands. English puts an s on a plural, the number is right there, and no other part of the text has anything to say about it.
• 45 exists nowhere near where it is needed. It sits in the policy line, eleven positions back, and something has to carry it forward.
• whatever appears in place of 45 when you delete the policy comes from neither of those routes. Nothing in the text holds it, so the model supplies it out of its own expectations about what a returns policy says — the same route that supplies Paris, now filling a slot it has no fact for.

Same machinery all three times. Three different places the token came from, and by the end of this module those three places will have names, because they are the three sources the thought state is built from. Getting that difference into your hands is what the rest of Part 1 is for, and we start by watching the machinery run.

One thing to settle before we do. What follows is a great deal of apparatus for a single word, and the volume of it is part of the lesson: the cost of one token is the thing almost nobody sees. You are not expected to hold all of it at once, and nothing later in this course depends on you having memorized any of it. Treat it as a frame of reference. Once you can name the stages, you can say which stage produced a given answer, and being able to say that is what every module after this one is built on. If it feels overloaded on the way through, that is the machinery being honest about its size, not you falling behind.

from text to numbers

The model receives text: your policy, the customer’s question, all of it as one run of characters with nothing marking where one ends and the other begins. Before any arithmetic can happen that text has to become numbers, which takes two steps.

First the text is cut into chunks. A fixed lookup table holds every chunk the model is allowed to recognize: between fifty thousand and two hundred thousand entries, roughly the headword count of a large dictionary, assembled once from the training text and never touched since. Your text gets spelled out of those entries and only those. Words that appeared constantly earned an entry of their own. Anything rare has to be built out of smaller scraps. The table is the tokenizer, and the chunks it produces are tokens.

The customer’s question comes apart like this:

["How"]["␣many"]["␣days"]["␣do"]["␣I"]["␣have"]
["␣to"]["␣return"]["␣the"]["␣C"]["-"]["40"]["?"]

Thirteen tokens. Read the end of that line closely. “C-40” is not one thing to the model but three separate tokens, C and - and 40, because Aperture Instruments and its camera did not exist when the table was built and nothing in it spells the product directly. Whatever the model says about the C-40, it assembles on the spot out of three fragments that mean nothing on their own.

Second, each token is looked up in another stored table and comes back as a list of numbers — 4,096 of them in a common mid-sized model, about eight thousand in a large one; we say eight thousand throughout this part. Thirteen tokens in, thirteen lists out. Each list also carries a mark of where its token sat in the order, so that 40 arriving tenth is distinguishable from the same 40 arriving second.

A list of numbers used this way is a vector, and the France question shows what one looks like from the outside:

Position 1 ("What"):    [  0.20, -0.50,  1.10, ... ]   (about 8,000 numbers)
Position 6 ("France"):  [  1.80,  0.40, -0.90, ... ]   (about 8,000 numbers)
Position 7 ("?"):       [  0.10,  0.30, -0.20, ... ]   (about 8,000 numbers)

Each slot tracks some abstract feature the model discovered during training. A positive value marks the presence of a trait, a negative value marks its absence or its opposite, and no slot has a name you could read off. The numbers above are made up to show the shape; what is exact is that every position starts with a vector of this kind, and that two tokens playing similar roles in text come out with vectors pointing similar ways. Module 1.04 works the arithmetic of “pointing similar ways” by hand.

Each slot in the token order is a position, and a position keeps its own vector for the whole trip. Thirteen positions, side by side, each carrying eight thousand numbers. That grid is everything the model has of your text: a block of numbers built a moment ago and thrown away when the answer ends. The characters, the words, and the policy as a document are gone; only the numbers made from them remain.

the same machine, forty to a hundred and twenty times

Now the numbers travel. Each position’s vector is handed upward through a stack of identically shaped machines, forty of them in a small model and up to about a hundred and twenty in a large one. Each machine is a block, and they are where all the work happens.

A block reads the vector arriving at a position and adds to it. It computes a vector of its own, the same length, and adds that in slot by slot, so what leaves the block is the sum of what arrived and what the block contributed. The next block up receives that sum, reads it, and adds its own contribution to it. Then the next. All the way to the top. The vector under construction at a position is the thought state h we named at the start, and it is built purely by addition: every block merges something in, and nothing is ever taken off.

Two consequences follow from adding rather than replacing, and both matter later.

Everything written stays legible. A block working near the top of the stack can still read what block 2 put there, because block 2’s contribution is part of the total that arrived. When a low block works out that 40 is part of a product name rather than a quantity, that finding is available seventy blocks later, while the answer is being composed.

And a later block can argue with an earlier one. Contributions can be negative, so a block that disagrees with what was written below it adds a contribution pointing the other way and cancels part of it. Nothing is erased, but a note can be crossed out by a later note.

The running vector each position carries is the residual stream. Picture a noticeboard per position, with every block posting to it and nobody taking anything down: early blocks post what a token is, middle blocks post what it means here, and blocks near the top post something close to a prediction.

This is where our easy token settles. The s in “days” needs a plural after a number, the number is right there in the same phrase, and no other part of the text bears on it at all. The blocks that handle it are low in the stack, the finding never gets contradicted higher up, and by the time the answer is being composed it has been sitting on the noticeboard, unchallenged, for a long climb. Most tokens in most sentences are like this. They are the reason a model’s prose is fluent even when its facts are wrong.

three tokens, three sources

The other two tokens cannot settle in place. Each of them reaches the answer by a different route, and you are about to watch both.

Take 45 first. It is in your text, but it is posted on the wrong board: it sits at the position holding 45 in the policy line, eleven positions behind the place where the answer is being written. Positions keep their own vectors, so nothing carries it forward on its own. Something has to go and get it.

That is what the first machine in every block is for. At every position it forms a short vector describing what that position is looking for, and another short vector describing what that position has to offer. Compare the first against the second at every other position and you get a score per position, saying how strongly these two line up. Where the score is high, the content of the matching position gets copied forward and added to the thought state here.

So the position composing the answer forms something that amounts to a number of days, attached to this product, that matches strongly against the policy position holding 45, and 45 gets pulled across into the answer’s noticeboard. The machine that does this is attention, and this is the only way information ever moves between positions. Every block runs dozens of these side by side, each looking for something different. Everything attention adds to the thought state, summed over every block, is the attention context, written h_ctx: the part of h that was pulled in from earlier positions.

Now delete the policy and ask again. The position composing the answer forms the same request, compares it against every position in the sequence, and finds nothing that matches: the customer’s question mentions the C-40 but says nothing about how long anything lasts. No number gets copied, because there is no number to copy.

What does the pass do with nothing to copy? It keeps running: every stage is arithmetic, and arithmetic has no channel for raising an empty result. The slot gets filled instead by the second machine in the block, which works at one position at a time: it compares the thought state against tens of thousands of stored patterns and adds a blend of whatever those patterns are paired with. Those patterns were fixed during training, and among them is a great deal of general knowledge about how returns policies read. So a number arrives: 30, usually, because thirty days is what most of the internet’s return policies say. It arrives in exactly the same way, at exactly the same fluency, as 45 did when the document was there. This machine is the MLP, and roughly two-thirds of a model’s numbers live in it. Everything the MLP adds to the thought state, summed over every block, is the parameter memory, written h_prior: the part of h that was injected from the frozen weights, whether or not the context asked for it.

Run the France question through the same two machines and you can see the memory term doing honest work rather than filling a gap. Attention at the last position gathers France together with capital; that is h_ctx, and it names the question without answering it. The MLP then recognizes that pattern in the thought state and adds what was stored with it, a contribution pointing toward Paris; that is h_prior, and it is the answer. Same machine that supplied 30. The difference is that this time the stored pattern was the fact that was asked for.

And the s in “days”? It settled in place, which in these terms means the position’s own starting vector, plus what the neighboring number contributed, already leaned toward it before anything far away or anything from memory had a say. The starting vector is the baseline, written h_initial: what the token itself brought to the position when it came out of the lookup table.

Three tokens, three sources, and now one equation:

  h_initial   the baseline — the current token's own embedding
+ h_ctx       the attention context — pulled in from earlier positions
+ h_prior     the parameter memory — injected by the MLP from the frozen weights
─────────────────────────────────────────────────────────────────────────────
= h           the thought state at this position

That is bookkeeping rather than theory: the lookup table, the attention machines, and the MLPs are the only three things that ever add to a thought state, so group the contributions by who made them and the sum falls out. Module 1.05 builds each of the three terms by hand. For now, hold the picture: the s is mostly baseline, 45 is context, 30 and Paris are memory.

Sit with what the dashed path means. Both numbers reach the same slot by machinery running at the same fluency, and the finished sentence reads the same either way. The model has no step at which it could notice the difference, because noticing would require comparing what it produced against something it does not have.

choosing the word

Everything so far produced numbers. Turning them into a word takes one more stage, and it is the stage most people never see.

After the last block, exactly one position matters: the final one, the position that has to say what comes next. Its finished thought state h is compared against a stored table that runs the embedding lookup backwards: every token in the vocabulary has a stored vector of its own in that table, its candidate vector, written u_A for candidate A, and the score for that candidate is the dot product of the two — multiply h against u_A slot by slot, add up the results, one number out. Fifty thousand entries in the table means fifty thousand scores. Those raw scores are logits, and the table is the unembedding table.

ℓ(A) = h · u_A

Read that as: the logit for candidate A is how closely the model’s final thought state matches the identity of that word. It is the same arithmetic the two machines in the block were running, and module 1.04 works it by hand.

Logits are unbounded and hard to compare: one might be 14.2 and another −3.8, and neither number means anything on its own. To choose among them they have to become proportions, and the step that does it is softmax: it keeps the scores in the same order, turns them into fractions adding up to one, and exaggerates the gaps between them along the way. What comes out is a probability distribution across every token the model could say next.

Softmax does two things to a list of scores, and both are worth seeing on the France question. Here are four candidates, with illustrative logits:

Token          Logit (ℓ)   Softmax probability (p)
──────────────────────────────────────────────────────────────────
"Paris"          +5.0      93.5%  [███████████████████████████████]
"Marseille"      +2.0       4.7%  [█]
"London"         +1.0       1.7%  [░]
"Apple"          -2.0       0.1%  [░]

First, it suppresses negative scores. Each logit is raised to a power of e before the shares are taken, and a negative score like −2.0 becomes about 0.135, so a candidate the thought state points away from ends up with a share near zero rather than a negative one. Second, it amplifies small leads. The exponential curve steepens fast, so a gap of only three logits between Paris at +5.0 and Marseille at +2.0 becomes a probability advantage of about twenty to one, because e cubed is roughly 20. That is the number to remember about softmax: three points of logit is a twenty-to-one lead. Small linear differences in the score turn into wide margins in the draw.

For our C-40 question with the policy present, the distribution comes out just as lopsided: 45 takes almost all the weight, a few plausible alternatives take slivers, and the rest of the vocabulary shares what is left. With the policy absent it is flatter and centered somewhere else, with 30 on top and 14 and 45 and 60 holding real shares behind it.

Then one token is drawn from the distribution. That draw is sampling, and it is the only place in the entire pass where anything is left to chance.

Temperature governs the draw. The number divides the logits before softmax runs, which sounds like a technicality and behaves like a dial on how adventurous the model is. Turn it down and the gaps between scores get exaggerated further, so the leading token takes even more of the weight and the long shots are squeezed out. Turn it up and the gaps flatten, so tokens that were barely in contention get a real chance of being drawn. At temperature 0 there is no draw at all: the highest-scoring token wins every time, and the same prompt returns the same answer forever.

That last line is worth having in your hands, because it is the most common mistake made with this dial:

Temperature governs the draw, not the ranking. It can make a model bolder or duller about picking a low-scoring token, and it cannot put a token into contention that the pass scored near zero. When the fact you needed was never in the distribution, no temperature setting will find it.

So a wrong answer that stays wrong at temperature 0 is not a randomness problem, and turning the dial down is not the fix. You have learned something instead: that answer sat at the top of the distribution on every pass, and the repair has to change what the model is drawing from, not how it draws.

That is the whole trip, and it is worth seeing laid out now that you have walked it:

and then it does it again

We now have one token. 45, drawn from a distribution built by a complete pass through every number in the model.

The answer is not finished, so the whole thing runs again.

The chosen token is appended to the text, and the model makes another full pass over all of it — your policy, the customer’s question, and the one word written so far — to produce the token after that. Then a third pass for the token after that. This is what autoregressive means, and the arithmetic is unforgiving: a hundred-token answer is a hundred complete passes, each one re-reading everything that came before it from the first character.

The text is the only thing that carries from one pass to the next. Every pass starts from the sequence of characters and rebuilds all thirteen thought states, then fourteen, then fifteen. Whatever plan or reasoning stood behind the last word existed as thought states, and those were thrown away the moment the token was drawn. A serving engine does keep one accounting shortcut, a cache of the per-position numbers attention reads, so the fourteenth pass can score the new position without recomputing the thirteen behind it. The cache holds exactly what the text would rebuild, and it is dropped when the request ends.

That is why a model’s own written output is the only working memory it has. Anything it needs to keep, it has to say. We come back to this hard in module 1.07, where writing things down turns out to lift a real ceiling rather than being a style of answering.

two kinds of number

Look back at what the pass touched, because it divides cleanly in two.

One set was fixed the day training ended and has not moved since. The tokenizer’s entries, the embedding table, every block’s stored patterns, the table that turns the final list into scores: on the order of a hundred billion numbers sitting in a file, a few hundred gigabytes that would fill a laptop’s whole drive, the same for you as for everyone else asking anything of that model today. Nothing in a pass writes to them. The model you run tonight computes exactly what it computed this morning.

The other set was built a fraction of a second ago out of the text in front of the model, carried up the stack, and thrown away when the answer ended. Your thirteen positions and their thought states are yours alone, and they last one request.

That text can arrive from more places than you might expect. You typed some of it. Someone may have pasted a document above it, a retrieval system may have gone looking and dropped three paragraphs in, or a tool may have run and returned its result a moment ago. Above all of it may sit a system prompt you never see. None of it arrives labeled. By the time the model reads it, it is one flat run of text, and the live numbers get built from all of it at once.

Two sets of numbers: the frozen numbers are stored and shared by everyone; the live numbers are yours alone and last one request. When someone says a model “knows” something, ask which set they mean. The answer changes what you should do about it.

Our three tokens now sort themselves. The s in “days” came out of the frozen set applying a rule to the live set. 45 came out of the live set, carried across by a rule in the frozen set. And 30 came out of the frozen set alone, filling a slot the live set had nothing to say about.

Remember what we established in module 1.01. A language model computes one thing, over and over: the next token. We named three ways it goes wrong. It confabulates: asked for something it does not hold, it produces a fluent answer shaped exactly like a true one. It has a grounding gap, having read every description of water ever written without ever being wet. And it carries systemic bias, reproducing the skews of the text it learned from.

On your first day, watching for three separate failures is exactly the right habit. Underneath, they are one mechanism seen from three angles, and you have just watched that mechanism produce 30 without hesitating. What we have not done is the arithmetic. Every comparison in this module, every “lands hard against” and “nothing matched,” is one operation repeated a few hundred billion times per word, and it is the same operation every time. Set that count against something you can picture: one word costs about as many multiply-and-add steps as there are stars in the Milky Way.

Before you go on, put the habit to work once. Take an answer a model gave you today, pick one token in it, and say which set supplied it: the frozen numbers everyone shares, or the live numbers built from your prompt. In the next module we do the arithmetic behind that call, and after that we build the three sources of the thought state one at a time.