The fixed depth of a single pass

Earlier we asked one question six ways, and the sixth run broke differently from the first five. We gave the model the return policy it had already read correctly, then asked for a refund instead of a lookup: three units at $2,450, delivered March 9, two returned unopened on April 22. Seven steps, each one needing the answer to the one before. Count the days to 44, check 44 against the 45-day window, check 44 against day 30 as well, multiply, take fifteen percent, subtract. It got the eligibility right, dropped the restocking fee, and answered $4,900 instead of $4,165, in the tone of something that had checked every line. Nothing was missing from the prompt, the document was right there, and the model had already proved it could read that document. One wrong answer does not tell us what caused it. An arithmetic slip, an instruction read as asking for a full refund, or a habit of refund answers that leave fees out would each produce the same sentence. The explanation this module follows is an architectural one: seven dependent results had to be worked out with no working written down, and a single pass has a fixed amount of room to work them out in. The answer is consistent with that limit, and the test that separates it from the others is to ask for the same refund with the steps written out and see whether the fee comes back.

So we go looking for where that room runs out, and then for the way around it that the model can take on its own.

one pass has a fixed depth

Follow what happens at the top of the stack in run two, where the return policy sat above the question and the model read 45 days straight out of the document, at 0.94.

That prompt held fewer than a hundred tokens. A working support prompt, with its instructions and the customer’s thread around the policy, runs nearer four hundred, so picture it at that size: four hundred positions, each one carrying its own residual stream of about eight thousand numbers up through eighty blocks. Every block adds its two contributions to every one of them. At the top of the stack, the machinery has roughly three and a quarter million numbers in hand — everything four hundred positions worked out about the question, the document, and the relationship between them.

Derivation Formal mechanical derivation

Where the three and a quarter million comes from: 400 positions × 8,192 numbers in each position’s residual stream ≈ 3.28 million, all of it live at the top of the stack. The read-out takes one of those 400 vectors.

Then it all narrows to one thing. Only the thought state at the very last position is read out. One vector, h. Eight thousand numbers. The other three hundred and ninety-nine positions’ conclusions are not consulted at this step at all; whatever they hold had to have been moved into the last position by attention on the way up — into h_ctx, the context term — or it plays no part in what gets said next.

That one vector is dotted against every candidate vector in the unembedding table, ℓ(A) = h · u_A, giving one raw score — one logit — for every chunk of text in the vocabulary, a hundred and twenty-eight thousand of them. Softmax turns those scores into probabilities adding to one. A single token is drawn from that distribution and appended to your prompt, and the next pass runs for that one new position, reading the keys and values already cached for every position before it.

That drawing overstates one thing. As token by token showed, the keys and values of earlier positions sit in the KV cache, so the vessel is not refilled from scratch on every token: each new token costs one pass for one new position, which runs through all eighty blocks and attends to the cached keys and values at each of them. The narrowing is real: every token’s decision does pass through one position’s residual stream. What the picture overstates is how much of the work is repeated to get there. Every limit this module states holds for one kind of model run one way: a dense decoder-only transformer generating one token per pass, the procedure token by token traced. Its sizes (eighty blocks, a residual stream about eight thousand numbers wide, a vocabulary of a hundred and twenty-eight thousand tokens, four hundred positions) are illustrative, and the work per token is constant in its depth only, because the attention part of each pass grows with the length of the sequence. The positions and shares in this module’s figures are shaped to show the mechanism and were measured from no run, and the probabilities this module carries over from module 3.07’s runs are that module’s illustrative numbers; that is said here once.

Three and a quarter million numbers, to eight thousand, to one token. Two things follow from that shape.

The work spent per token is constant. A trivial word and a word that resolves a hard sub-problem get the same eighty blocks and the same hundred billion multiplications. The model cannot choose to spend longer on the hard one. Run six is the case this module reads through that fact: a figure that needed seven dependent results, written with no working in front of it.

And if the conclusions will not all fit through the spout at once, the model has one way out of its own: write some of them down as tokens. Once written, they are poured back in as context, where every later pass can read them the way run two read the policy document. Writing matters even with the cache in place, because of where each kind of result can be read. A result that an earlier position worked out partway up the stack stays readable to later positions, but only through attention in the blocks at that height and above. A result written as a token enters every later position at the bottom of the stack, as part of the text, and all eighty blocks above can build on it. Working memory moves out of the residual stream and into the sequence itself.

That is the way out inside the model, and everything called reasoning in a product today is built on it. The other ways out sit outside the model: an application can hand a sub-problem to a tool, such as a calculator, a code runner or a search, and put the result back into the context as text.

what the model is doing when it “thinks”

You will hear a model’s thinking described in two ways, usually by people arguing with each other, and the machinery you have traced settles both questions rather than splitting the difference.

Is it only autocomplete? Prediction is the goal training aimed at, and what training built to hit that goal is a different question with a documented answer. What researchers have found by looking inside these models is a catalog of procedures: a copy-and-continue arrangement that picks up a pattern invented five seconds ago, a twenty-six head arrangement that resolves which name a sentence is about, a genuine sine-wave algorithm for arithmetic that wraps around like a clock face, a board representation in a model that was never shown a board, and rhyme words selected before the line that arrives at them. Those are procedures, running on whatever is in the context. Training discovered them because an algorithm predicts better than a lookup table does, and a description of the machinery has to account for them.

Counterpoint Strongest case against this

The deflationary reading is not silly, and it is worth being able to state. Every structure found inside these models was found because it lowered prediction error on text. Nothing in training asked for a world model, and finding machinery that behaves like one is not the same as finding one. What the evidence settles is that the machinery is there, not what to call it.

Then does it reason the way you do? Here the differences are structural rather than a matter of degree, and there are four of them worth holding separately.

The depth is fixed. Eighty blocks per token, always, whatever the token costs. You can spend an hour on one thought, walk away, come back. Without writing tokens, the model cannot. The formal version is sharp: a stack of fixed depth cannot solve certain step-by-step problems in one pass at any width, and writing intermediate tokens lifts that ceiling. Writing the steps out removes a real limit. The tokens on the page do work the stack has no room to do, which is why it helps on problems that no amount of instruction fixes. The move has a name worth keeping: chain of thought.

Nothing is kept. Your reasoning writes to long-term memory, and tomorrow you start from a changed position. The weights do not move at inference. Whatever a model works out lives in the residual streams over one prompt and ends with it. If you want tomorrow’s session to have today’s conclusion, write it to a file and put the file back in the prompt.

That is what an external memory system does: it keeps what matters about a user in a store outside the model and reads it back at the start of the next session. Two problems have to be solved to make it work, and they are the two problems anyone building memory faces. First, filtering: a raw transcript is mostly noise, so something has to decide which handful of facts from a conversation are durable enough to keep. Second, doctrine: once a stored fact comes back into the context, the agent needs written rules for how to cite it, when to trust it against something the user just said, and what to do when yesterday’s fact and today’s contradict each other.

The target was never truth. Human thinking was shaped by a world that punishes bad predictions. Pretraining was shaped by text, and post-training by which answers people rated well. Truth correlates with both and matches neither. That gap is why fluent confident wrongness comes out of the process naturally: training rewarded what sounded right more reliably than what was right, and module 1.01 showed why fluency alone certifies no claim.

The output is drawn from a distribution. In run four, where two retrieved documents disagreed, one saying 30 days and the current one saying 45, the model’s next-token distribution came out 0.52 for one against 0.41 for the other. That is a coin weighted slightly, and a single token is drawn from it. Ask the same question five times at an ordinary temperature and you get different answers with no change in inputs. Set the temperature to zero and the draw goes away while the margin stays: the answer is still the top of a distribution that had a real second place. Even then, as token by token showed, the same prompt can come back different across runs, because the serving arithmetic itself can vary with server load. Whatever variability human thinking has, it does not have this one: these components vary by design, at one known step, the draw from the distribution.

One more comparison belongs here. Can the model look inward the way you can? Start with the human side, where the research is decades old and unkind: people confidently report causes of their own choices that provably were not the causes, and a brain hemisphere disconnected from the one that acted will invent fluent explanations for actions it did not initiate. Human self-report is heavily reconstructed after the fact.

The honest difference is narrower and still matters. Your account of your reasoning has some causal connection to the reasoning, however patched. A model’s stated account can be produced by machinery largely disconnected from the machinery that produced the answer, and unlike the human case, we can measure it. We do that below, with an experiment you could run yourself in an afternoon.

	you	a standard model	a reasoning model
Steps per thought	as many as you like	eighty blocks, fixed	eighty per token, as many tokens as the budget and the window allow
Effort on hard problems	more	the same per token, more only if it writes its working	more
Working memory	a few things, fading	one residual stream	written down in the sequence
Kept afterwards	yes	nothing	nothing
Shaped by	acting in a world	text, then ratings	plus rewards on checked answers or on steps
Contact with reality	continuous	text, plus any tools given	text, plus any tools given
Same input twice	often a similar answer	a fresh draw; temperature 0 removes the draw	a fresh draw; temperature 0 removes the draw
Account of its own work	partial, reconstructed	written after the fact	does real work, still incomplete

What the word “reasoning” is naming here. A set of procedures found by training because they lowered prediction error, run at fixed depth, kept nowhere, and never tested against anything but text. Each of those four gaps has a fix you can build: depth comes from written tokens, persistence from a file you put back in the prompt, contact with truth from a tool or a checker, and stability from sampling the same question more than once.