The weights hold more than facts

the weights hold facts, procedures, world models, and the prior

Aren’t the weights just compressed training data? Compression is the right instinct, and the storage is stranger than the phrase suggests: training compresses the data by finding the structure that generates it, and the structure is what gets stored. Four kinds of thing are worth telling apart.

Facts are there, and the recall path has been traced: middle blocks read an enriched representation out of the stored patterns at the subject’s position, and later attention carries it to where the prediction is forming. The machine that does the reading is the MLP. A fact is stored about as strongly as it was common and consistent in the training text. Fifteen trillion tokens is enough text that a person reading day and night would need tens of thousands of years to finish it; a fact that appeared five times in all of that is stored weakly or not at all. Interpretability work on one studied model, described in module 1.01, found internal features tied to whether an entity is familiar, and those features can fire without the knowledge behind them, so they do not give the model a reliable reading of which case it is in. Sit with that, because of what it costs: the model can answer a fact it holds firmly and one it never held with the same fluency, and the fluency does not tell you which case you are in. And a fact stored strongly is not the same as a fact stored alone: the training text that said Canberra also said Sydney, near the same words and far more often, and both associations sit under the same roof. Ask for one and the other is competing for the same slot.

Procedures are the part “compressed data” misses. Training discovers working algorithms, because an algorithm predicts better than a lookup table. The cleanest example is a two-head arrangement that finds an earlier occurrence of the current token, looks at what followed it last time, and predicts that. It is a general copy-and-continue routine, it runs on whatever is in the context, and it works on sequences that appeared nowhere in training. It is a large share of what looks like learning from your prompt. Researchers have also mapped a twenty-six head arrangement in a small model that resolves which name a sentence is about, and an arithmetic routine that lays the numbers around a clock face and adds them in sine waves. That one appears at the exact moment the model stops memorizing and starts generalizing.

World models exist and are narrower than the phrase suggests. A model trained only on Othello move lists, never shown a board, builds an internal picture of the board. A small classifier can read that picture out, and, decisively, you can edit it. Change the picture and the model’s next legal-move predictions change to match. Prediction pressure alone produced a model of the thing generating the text. Take the result in both directions: real internal models do form, and they form in well-structured domains and stay narrow.

The prior sits underneath all of it: the model’s defaults about what text looks like, which claims travel with which, what a policy document sounds like, what usually follows a question of this shape. Post-training reshapes it heavily toward helpful assistant behavior, and the substrate underneath is still a very strong set of expectations about plausible continuations. Hold onto it. It is the mechanism behind the most consequential failure these systems have, and learning to recognize it at work is most of the skill.

There is one more finding worth carrying, because it is the most recent and the most surprising. Trace what happens when a model writes a rhyming couplet and you find it activating candidate rhyme words before generating the line that arrives at them, then building the line to land on the chosen target. Suppress the planned word and the whole line comes out different. Some amount of looking ahead happens inside a single pass.

Two limits on all of this. These procedures have mostly been located in small models on narrow tasks, and nobody has a full account of what a frontier model computes. And the tidy story of low blocks doing tokens, middle blocks doing meaning, and high blocks doing prediction has real evidence behind it and real exceptions; the work is distributed and redundant and does not respect that division neatly.

why the run failed: neither set held the answer

The run you watched failed for a reason we can now state precisely: the weights held nothing about the C-40, and the activations held nothing but the question. Neither set contained the answer, so there was no answer to be had.

That points at the repair, and it is the one you would reach for at work. If the model does not hold the fact, put the fact in front of it. Paste the returns policy above the question and ask again:

Returns policy, rev. 2026-02: Aperture Instruments accepts returns
of the C-40 within 45 days of delivery. Units returned after day 30
incur a 15% restocking fee. Opened lens kits are non-returnable.

How many days do I have to return the C-40?

This works. But “now the model can see the document” explains nothing, and the machinery it glosses over is the part worth understanding.

Look at what actually arrived. The document is more tokens in the same flat sequence, chopped by the same lookup table, turned into vectors in the same way — several dozen positions now instead of thirteen. Every position arrives with the same standing: the policy tokens read as ordinary text, exactly as the question does, and any link between the two is left for the machinery to work out. And the number the answer needs, 45, is sitting in one position of its own, a few dozen positions away from the position where the answer will be written.

So two separate things have to happen inside the stack, and each block holds one machine for each of them. One machine moves information between positions, which is how 45 travels from where it sits to where it is needed, and what it writes is the context term of the thought state. The other works within a single position, comparing what is already there against the frozen patterns and adding what was filed under the matches, and what it writes is the memory term. Each machine does its piece of our question, and of the France question, and the two contributions are added to the token’s own baseline to make the one vector that chooses the word.

what you can do with this today

The two sets are a diagnostic you can run in the middle of a working day, and it costs nothing.

When an answer is wrong, ask where the right answer was supposed to come from. If it was supposed to come from the weights, you are relying on a fact stored in proportion to how often it appeared in the training text, with no signal telling you whether it is there. If it was supposed to come from the prompt, then the question is whether the text was present and whether anything pointed the machinery at it. The first case is fixed by putting the fact in the context. The second is fixed by rewording, reordering, or removing whatever else in the prompt was competing for attention. Those two problems look identical from the outside and have opposite fixes.

In our next lesson we build the vector that both fixes act on, one source at a time.