The Narrows

In module 1.06 we asked one question six ways, and the sixth run broke differently from the first five. We gave the model the return policy it had already read correctly, then asked for a refund instead of a lookup: three units at $2,450, delivered March 9, two returned unopened on April 22. Seven steps, each one needing the answer to the one before. Count the days to 44, check 44 against the 45-day window, check 44 against day 30 as well, multiply, take fifteen percent, subtract. It got the eligibility right, dropped the restocking fee, and answered $4,900 instead of $4,165, in the tone of something that had checked every line. Nothing was missing from the prompt, the document was right there, and the model had already proved it could read that document. What ran out was room.

So we go looking for where the room runs out, and then for the one way around it.

the narrows

Follow what happens at the top of the stack, using the numbers from run two of the last module: the run where we put the return policy above the question and the model read 45 days straight out of the document, at 0.94.

That prompt held four hundred tokens, so four hundred positions, each one carrying its own residual stream of about eight thousand numbers up through eighty blocks. Every block added its two contributions to every one of them. At the top of the stack, the machinery has roughly three and a quarter million numbers in hand — everything four hundred positions worked out about the question, the document, and the relationship between them.

Then it all narrows to one thing. Only the thought state at the very last position is read out. One vector, h. Eight thousand numbers. The other three hundred and ninety-nine positions’ conclusions are not consulted at this step at all; whatever they hold had to have been moved into the last position by attention on the way up — into h_ctx, in the language of module 1.05 — or it plays no part in what gets said next.

That one vector is dotted against every candidate vector in the unembedding table, ℓ(A) = h · u_A, giving one raw score — one logit — for every chunk of text in the vocabulary, a hundred and twenty-eight thousand of them. Softmax turns those scores into probabilities adding to one. A single token is drawn from that distribution, appended to your prompt, and the whole thing runs again from the beginning for the token after it.

That drawing overstates one thing. Keys and values from earlier positions are cached, so the vessel is not genuinely refilled from scratch on every token. The narrowing is real: every token’s decision does pass through one position’s list. What the picture overstates is how much of the work is repeated to get there.

Three and a quarter million numbers, to eight thousand, to one token. Two things follow from that shape.

The work spent per token is constant. A trivial word and a word that resolves a hard sub-problem get the same eighty blocks and the same hundred billion multiplications. The model cannot choose to spend longer on the hard one. Run six is that fact stated as a failure.

And if the conclusions will not all fit through the spout at once, there is exactly one way out: write some of them down as tokens. Once written, they are poured back in as context, where every later pass can read them the way run two read the policy document. Working memory moves out of the running list and into the sequence itself.

That is the only way out the design has. Everything called reasoning in a product today is built on it.

against the human comparison

You will hear a model’s thinking described in two ways, usually by people arguing with each other, and the machinery you have traced settles both questions rather than splitting the difference.

Is it only autocomplete? Prediction is the goal training aimed at, and what training built to hit that goal is a different question with a documented answer. In module 1.04 we went through what researchers have found inside these models by looking: a copy-and-continue arrangement that picks up a pattern invented five seconds ago, a twenty-six head arrangement that resolves which name a sentence is about, a genuine sine-wave algorithm for arithmetic that wraps around like a clock face, a board representation in a model that was never shown a board, and rhyme words selected before the line that arrives at them. Those are procedures, running on whatever is in the context. Training discovered them because an algorithm predicts better than a lookup table does, and a description of the machinery has to account for them.

Then does it reason the way you do? Here the differences are structural rather than a matter of degree, and there are four of them worth holding separately.

The depth is fixed. Eighty blocks per token, always, whatever the token costs. You can spend an hour on one thought, walk away, come back. Without writing tokens, the model cannot. The formal version is sharp: a stack of fixed depth cannot solve certain step-by-step problems in one pass at any width, and writing intermediate tokens lifts that ceiling. Writing the steps out removes a real limit. The tokens on the page do work the stack has no room to do, which is why it helps on problems that no amount of instruction fixes. The move has a name worth keeping: chain of thought.

Nothing is kept. Your reasoning writes to long-term memory, and tomorrow you start from a changed position. The weights do not move at inference. Whatever a model works out lives in the running lists over one prompt and ends with it. If you want tomorrow’s session to have today’s conclusion, write it to a file and put the file back in the prompt.

You will build exactly that in module 2.05, where an agent keeps what matters about a user in a store outside the model and reads it back at the start of the next session. Two problems have to be solved to make it work, and they are the two problems anyone building memory faces. First, filtering: a raw transcript is mostly noise, so something has to decide which handful of facts from a conversation are durable enough to keep. Second, doctrine: once a stored fact comes back into the context, the agent needs written rules for how to cite it, when to trust it against something the user just said, and what to do when yesterday’s fact and today’s contradict each other.

The target was never truth. Human thinking was shaped by a world that punishes bad predictions. Pretraining was shaped by text, and post-training by which answers people rated well. Truth correlates with both and matches neither. That gap is why fluent confident wrongness comes out of the process naturally: nothing ever penalized being wrong in a way that sounded right.

The output is drawn from a distribution. In run four of the last module, two retrieved documents disagreed, one saying 30 days and the current one saying 45, and the model’s next-token distribution came out 0.52 for one against 0.41 for the other. That is a coin weighted slightly, and a single token is drawn from it. Ask the same question five times at an ordinary temperature and you get different answers with no change in inputs. Set the temperature to zero, as module 1.03 showed, and the draw goes away while the margin stays: the answer is still the top of a distribution that had a real second place. Whatever variability human thinking has, it does not have this one, and it means these components behave more like a stochastic process than like software.

One more comparison belongs here. Can the model look inward the way you can? Start with the human side, where the research is decades old and unkind: people confidently report causes of their own choices that provably were not the causes, and a brain hemisphere disconnected from the one that acted will invent fluent explanations for actions it did not initiate. Human self-report is heavily reconstructed after the fact.

The honest difference is narrower and still matters. Your account of your reasoning has some causal connection to the reasoning, however patched. A model’s stated account can be produced by machinery largely disconnected from the machinery that produced the answer, and unlike the human case, we can measure it. We do that below, with an experiment you could run yourself in an afternoon.

What the word “reasoning” is naming here. A set of procedures found by training because they lowered prediction error, run at fixed depth, kept nowhere, and never tested against anything but text. Each of those four gaps has a fix you can build: depth comes from written tokens, persistence from a file you put back in the prompt, contact with truth from a tool or a checker, and stability from sampling the same question more than once.

writing it down

Two things changed to produce reasoning models. One is a training change, and the other is mechanical.

The training change: run the model on a problem, let it produce a long stretch of tokens before answering, then score the final answer with a program rather than a person. Mathematics, code, anything a checker can verify. Reward the runs that ended correctly.

Read what that rewards, carefully. We will need it in a few pages, when we ask how much of a visible chain of thought you can actually believe. The score lands on the final answer and nowhere else. The tokens in the middle receive no direct supervision at all. One thing alone shapes them: whether the runs containing them ended correctly. Nothing scores their readability, and nothing checks them against what actually happened inside the pass that produced them. A trace that reaches the right answer by a route it does not mention scores exactly as well as one that narrates itself faithfully.

The mechanical change is the one that matters: those tokens go into the context. The model writes to its own prompt.

This is the one way out the amphora leaves open. One pass is depth-limited, but a value written at token forty is readable at token forty-one, and token forty-one gets a fresh full-depth pass to build on it. Step-by-step work stops being capped by the height of the stack and starts being limited only by how many tokens you will pay for.

The steps in that drawing are one token each. A real trace spends dozens of tokens per step, and the columns on the right are the same eighty blocks re-run rather than new ones.

the step ladder

Put the two ways of answering side by side, because the difference is easy to state now that you have the three sources in hand.

A standard model answers with a single associative jump. It reads the prompt, builds one thought state at the last position, dots it against the vocabulary, and draws. When the problem needs several deductions in a row, that one jump has to carry all of them, and what usually fills the slot is the loudest association the MLP holds for the pattern in front of it — the memory vote from module 1.05, arriving at full strength whether or not it is the right one. A thinking model routes through intermediate states before it answers. It writes each step down as tokens, and each later pass reads the earlier steps as context, so the context vote is built up rung by rung instead of being asked to hold the whole ladder at once.

Standard model — one hop:
[Problem] ─────────────────────────────────────────► [Answer]
              relies on the strongest immediate association

Thinking model — the step ladder:
[Problem] ──► [Step 1] ──► [Step 2] ──► [Step 3] ──► [Answer]
              builds context token by token; every step is a full pass

Here is a riddle old enough that you may know it, which is exactly why it makes the mechanism visible:

A man walks into a bar and asks the bartender for a glass of water. The bartender pulls out a gun and points it at him. The man says “Thank you” and walks out. What happened?

Watch the one-hop answer form. The context vote carries bar, gun, water, and thank you, and the memory vote is dominated by what co-occurs with bar and gun in ordinary text: hostility, a hold-up, a crime. The tally goes to the loudest association and the model leaps: “It was an attempted robbery.” The answer is wrong because the single jump skipped the constraint that ought to have governed it: nobody says thank you to a robber, and there was no second pass in which the constraint could be re-read against a candidate.

Now the step ladder. The model writes intermediate reasoning into its own context before answering. This trace is reconstructed, following the shape such traces take rather than quoting a run:

<think>
Step 1: Parse the core puzzle: why would someone say "thank you"
        after a gun is drawn on them?
Step 2: The gun caused sudden fear or surprise.
Step 3: What condition is treated by either a glass of water
        or a sudden shock?
Step 4: Hiccups. A glass of water cures hiccups; a sudden scare
        also cures hiccups.
Step 5: The scare solved the problem, so the water was no longer
        needed.
</think>
The bartender realized the man had the hiccups and cured them
by startling him.

Read step 1. The first thing the ladder does is write the constraint down — the thank you that the one-hop answer skipped — so that every later pass reads it as context and cannot lose it in the squeeze at the spout. Steps 2 and 3 narrow the search from bar and gun to water and shock, and step 4 is the memory vote firing on a pattern it had not been shown until the trace built it: cured by water or by a fright. That pattern was never in the prompt. The model wrote it, then read it.

Grade this before you carry it. The riddle is famous, so a large model may hold its answer as a stored fact and produce hiccups in one hop; take the example as the shape of a class rather than a benchmark. The class is: problems where the loudest immediate association is wrong, and the right answer needs a constraint held across several steps. Date-and-threshold arithmetic on a returns policy is squarely in it, and that is what we watch next.

Nobody specified the behaviors that came out of this training. They were selected because they raise the chance of a correct final answer. The model spends more tokens on harder problems and fewer on easy ones, which the architecture could not do before. It writes intermediate results down instead of holding them. It restates a result and checks it against a constraint. And it backtracks, which deserves a closer look, because a model cannot un-write a token. What it can do is write a correction, after which the corrected value sits in the prompt more strongly and more recently than the error. The wrong figure stays in the sequence; the correction outweighs it. That is the same vote we watched in run four, where the losing number never left the context and was only outnumbered. Here the model is casting that vote against its own arithmetic.

Watch it happen on the specimen. This trace is reconstructed rather than captured: the beats follow the shape real traces take on date-and-threshold arithmetic, including the error and the catch, but no model produced these exact words.

Three mechanisms are visible in that trace, and each one is something we built earlier.

The backtrack worked because the wrong figure had been written as a token. Attention could read it, and a later pass could write a better-supported number beside it. Had “50 days” stayed inside a running list, nothing would have been there to catch.

The re-read worked because the policy never left the prompt. Catching the delivery-versus-order distinction meant going back to the document after already computing something, and in one pass the constraint and the computation compete for the same eighty blocks. Across many passes the model can read, compute, and read again.

And the restocking fee survived because both comparisons against day forty-four were written out separately. Run six lost it in the squeeze at the spout. Here neither comparison had to share the opening with the other six results.

the context update at the end of the trace

There is one moment in a reasoning run worth slowing down for, because it is where the ladder pays out, and it is the moment budget limits break.

When the model writes the token that closes its scratchpad — </think> in the trace above — every step it wrote is still sitting in the context window. Nothing has been summarized or handed forward in a side channel. The pass that writes the first token of the visible answer runs attention across the whole sequence, the original prompt and the entire trace and the closing tag, and assembles a fresh context vector from all of it:

h_ctx = Attention( original prompt + thinking trace + "</think>" )

Now the query at the answer position is looking for the conclusion, and the keys it scores highest against are the positions where the trace wrote it. On the riddle, that is hiccups at one position, scare at another, cured at a third. Their values are blended into h_ctx, and the thought state at the answer position lands hard against the candidate vector for the word the trace arrived at:

ℓ("hiccups") = h · u_hiccups   →   p("hiccups") > 99%

That is the mechanism, and it has a failure mode you will meet in production the first time you cap reasoning tokens.

An unfinished scratchpad still gets read. When a reasoning budget runs out before the chain reaches a resolution, whether the framework closes the tag for you or the provider stops generation where it stands, attention at the answer position reads an incomplete scratchpad. The context vector it assembles points only toward the partial calculation, and the unembedding step has to make a low-confidence guess from unfinished premises. A truncated trace does not degrade gracefully into a shorter argument; it hands the answer back to the one-hop machinery, now with a half-built context in front of it.

So when you set a reasoning budget, set it for the shape of the task rather than for the average, and treat a trace that ends without a stated conclusion as a signal to re-run with room, not as an answer to ship.

what the trace does not tell you

A visible chain of thought looks like an explanation, and there is a measurement that says how far you can trust it as one.

The experiment is clean. Give a model a problem plus a hint at an answer, sometimes correct and sometimes deliberately wrong. Then check two things separately: whether the model’s answer moves toward the hint, and whether the visible reasoning mentions the hint at all.

That stops being a contradiction the moment you recall what the training rewarded. The tokens were shaped to reach correct answers. Nothing ever asked them to describe the path. Expecting fidelity is expecting a property nothing selected for.

What the trace closes and what it leaves open. A visible chain lets you catch arithmetic slips, wrong constants, and misread constraints, because those errors appear in the text and you can check them against the source. It does not tell you what else influenced the answer, and a fluent trace raises a reader’s trust without raising the answer’s reliability. Show it as working, never as warrant.

The habit: when you need to show a user why, generate something a machine or a person can check against the source. Quote the clause. Link the row. Print the calculation and re-run it. Surfacing the chain and labeling it an explanation puts weight on a beam that was never built to carry it.

where reasoning tokens are worth paying for

They pay off where the work is genuinely step-by-step and the result is checkable: mathematics, code, planning, constraint problems, anything with intermediate values worth holding. Run six is the shape.

They pay almost nothing on a single lookup. Run two does not improve with eight hundred tokens of deliberation, because there was never a depth problem there, only a routing problem. They pay nothing on tone and format.

And sometimes they cost you. On easy questions, a long chain can talk a model out of an answer it already had right, because more tokens means more chances to introduce an error and then build on it. On judgment and perceptual tasks, forcing an explanation can lower accuracy the same way describing a face makes people worse at recognizing it afterward: the reasons a person can articulate are not always the reasons that were working. And an early mistake is now sitting in the prompt as a strong, recent, readable signal for everything downstream. Backtracking catches some of those. Some of them get a confident structure built on top instead.

So sort the request before you spend on it. Sending all traffic through extended reasoning pays a large fixed cost in latency and tokens for a benefit most requests never collect, and a few are made worse by. That sorting step is a router, a shape you will build for yourself in module 2.03: an entry-level agent that reads each incoming request, classifies it, and hands it to the specialist that should answer it, answering nothing itself. The same shape works here, with the classes being how much thinking a request deserves rather than which specialist it belongs to.

what you can do with this today

Route by the shape of the task. Lookups do not need reasoning tokens. Step-by-step problems do. Classifying the request first is usually the largest cost saving available in a production system, and it tends to improve quality at the same time.

Show checkable justification, not the chain. Quote the clause, link the row, print and re-run the calculation. If you do surface a trace, label it as work in progress rather than as an explanation.

When a model drops a step, give it somewhere to put the step. Before you reach for a bigger model or a firmer instruction, ask whether the answer required more results held at once than one opening can carry. If it did, the fix is to let it write them down.