Two kinds of number

Every comparison in the last module was the same piece of arithmetic. When the position composing an answer went looking for a number of days and landed on the policy line, that was one operation. When a stored pattern matched the thought state closely enough to contribute, that was the same operation again. A model runs it a few hundred billion times per word, and it is nearly all the arithmetic there is: what surrounds it is a thin layer of scaling and re-centering.

In module 1.03 we followed a single token through a complete pass: text cut into chunks, each chunk turned into a vector of about eight thousand numbers, those vectors carried up through forty to a hundred and twenty blocks that each read what arrived and added to it, and one token drawn from a distribution at the top. Two sets of numbers were in play the whole way. The frozen set is the model itself, on the order of a hundred billion numbers fixed the day training ended. The live set is built from whatever text sits in front of the model, and it lasts one request.

So we go down one more level. What that operation is, why a list of numbers can carry a meaning at all, how to read the number it returns, and what the frozen set holds besides facts. In the module after this one we use all of it to build the thought state from its three sources.

We keep the same specimen: Aperture Instruments, its C-40 camera, and a customer who wants to know how long they have to send one back.

the one operation

A model has one job at every step: read the text so far and produce a score for every token it could say next. A machine cannot weigh words against each other, so the text becomes numbers before anything else happens, and every judgment about that text then has to become a calculation on those numbers. Which earlier word does this one depend on. Is this fragment part of a name or part of a date. What kind of answer does a question shaped like this usually get. All of it, arithmetic.

So each token becomes a long list of numbers, and the list works like a set of coordinates: tokens that play similar roles in text come out pointing in similar directions. Meaning is stored as a position. Once meaning is a position, asking whether this piece of text has anything to do with that one stops being a question about language. It becomes a question about two lists of numbers, and getting the answer takes one cheap calculation.

A list of numbers used this way is a vector, and before we run any arithmetic on two of them, be clear about what you are looking at.

A real one holds about eight thousand numbers. Ours will hold three, because three fits on a page. The second simplification matters more. Each dimension of a real vector tracks some abstract feature discovered during training. A positive value marks the presence of a trait, and a negative value marks its absence or its opposite. The individual slots still stand for nothing you could name. The model has far more properties to represent than slots to keep them in, so it packs several into each slot and any one of them is doing a few jobs at once. We come back to that in the next module, under the name it goes by. For now we pretend the opposite, and give each of our three slots one clean, nameable property, because that is what makes the arithmetic legible.

Each slot measures a scale with two opposite ends and a zero in the middle:

Now you can read a sign. A positive number means the text sits toward that scale’s positive end, a negative number means it sits toward the negative end, and a number near zero means the scale has nothing to say about this text. The size of the number says how strongly.

That answers the first question anyone asks when they see one of these lists: why does a single list carry both positive and negative numbers? Because a piece of text lands at the far end of some scales, the opposite end of others, and off in the middle of the rest, and one list has to record all three verdicts at once. A negative entry is not a deficiency or a missing value. It is a measurement pointing the other way.

Here are four short pieces of text, measured on our three scales:

Take a and b first. Both are mildly formal, both are about what the customer is allowed to do rather than what they must do, and both concern a span of time. Two texts, three scales, and the same verdict on all three.

Here is the calculation that detects it.

Take two lists of numbers of the same length. Multiply the first against the first, the second against the second, and so on down the line. Add up the results.

Look at what each multiplication did, because this is where the signs stop being decoration and start doing work.

Slot 2 is the interesting one. Both lists carry a negative number there, −1.2 and −1.0, and multiplying two negatives gives +1.20, the largest contribution of the three. The two texts agree emphatically that this is about a permission rather than an obligation, and agreement at the negative end of a scale counts exactly as much as agreement at the positive end. Slots 1 and 3 do the same thing at the other end: positive times positive, both small and positive out.

So a pair of numbers on the same scale settles into one of three cases. Same sign, either sign, and the product is positive: the two texts sit at the same end, and the score goes up. Opposite signs, and the product is negative: they sit at opposite ends, and the score goes down. Either number near zero, and the product is near zero whatever the other one says: that scale abstains, because one of the two texts had no opinion to register on it.

Add the three together and you have a tally of agreements against disagreements, with each scale voting in proportion to how strongly both texts committed on it. That is what 1.83 means.

Now run the same arithmetic on a pair that disagrees. Keep list a, and compare it against c, “don’t lose the lens cap”: casual where a is formal, an obligation where a grants a permission, about a physical object where a is about a span of time. Opposite verdict, all three scales.

Same machine, opposite verdict, and note that the machine did not detect absence. It detected opposition, and reported it with a sign.

Which leaves the third case, the one people usually get wrong. Compare a against d, “the licensee shall submit quarterly reports”: 0.3 × 1.0 = 0.30, then −1.2 × 0.5 = −0.60, then 0.8 × 0.4 = 0.32. The total is 0.02, near enough to nothing. Both texts are formal and both concern time, so slots 1 and 3 agree; slot 2 disagrees hard enough to cancel both. A score near zero does not mean the two texts are opposites, and it does not always mean they have nothing in common. It means the agreements and the disagreements came out even. The other route to zero is simpler: one list sits near zero on the scales the other one commits to, so there is nothing to multiply against.

the compass

Three scores, one operation, and they lay out along a line. In physics the dot product measures how much two forces pull in the same direction. In a language model it measures how many traits two vectors share, and the three cases you just worked are the three readings the compass gives:

Keep that table. Every time this part says “matches,” “lines up,” “lands hard against,” or “nothing to do with,” it is naming one of those three rows.

That line is worth having the vocabulary for, because it is how the field talks.

Two vectors pointing the same way are aligned, or parallel. Two pointing opposite ways are antiparallel, and what they share is a negative correlation: a strong relationship running backwards, where each one predicts the opposite of the other. Two vectors at right angles to each other are orthogonal, which is the word for genuinely unrelated. Orthogonal is the middle of the scale, and it is where the overwhelming majority of pairs sit in a real model, because in eight thousand dimensions there is a great deal of room for two directions to have nothing to do with one another.

The arithmetic behind the picture is one line, and it explains why the scores landed where they did:

a · b = |a| × |b| × cos θ. The score is the length of the first vector, times the length of the second, times the cosine of the angle between them.

Read the three factors separately, because they answer different questions. The two lengths measure how emphatic each vector is: how far from zero it sits, which you get by squaring every slot, adding those squares up, and taking the square root. The cosine measures direction alone, and it runs from +1 when the two point the same way, through 0 at right angles, to −1 when they point opposite ways. Multiply direction by both magnitudes and you get the raw score the machinery actually computes.

Divide the two lengths back out and you are left with the cosine by itself, which is called cosine similarity. That is the number under each marker in the figure above: our agreeing pair comes to 0.98, about twelve degrees apart; the opposed pair to −0.98, about a hundred and sixty-nine degrees apart; the canceling pair to 0.01, which is a right angle to within a degree. When someone says two pieces of text are “similar” in a vector space, this is nearly always the number they mean, because dividing the lengths out stops a merely emphatic vector from scoring high against everything it half-agrees with.

Both measures are worth keeping straight. Cosine similarity is what you reach for when comparing meanings, in a retrieval system or a clustering job. The raw score, lengths included, is what the network itself runs, and that is deliberate: a position that has built up a strong commitment gets a loud vote, and one that is unsure gets a quiet one. Hold onto “loud” and “quiet.” In the next module the whole story of a wrong answer turns on one source arriving at a higher magnitude than another.

One honest note on all of this before we scale it up. Real starting vectors come from single tokens, not from phrases like “within 45 days of delivery,” and a phrase acquires a meaning of its own only after several blocks of attention have mixed its tokens together. Treat our four lists as thought states partway up the stack rather than as rows straight out of the lookup table. The arithmetic is exactly right; what is idealized is how tidy the things being compared are.

That is the question the network asks, a few hundred billion times per word: does this match that? The operation has a name, the dot product, and you now hold the vocabulary that travels with it: vectors, their magnitude and their direction, orthogonal for the unrelated pair, cosine similarity for the comparison with magnitude divided out. Keep all of it. Every paper, every library, and every engineer you work with uses those words for exactly the arithmetic you just watched.

Now stack thirty-two thousand stored vectors into a grid and run the comparison against every one of them at once. One vector goes in; thirty-two thousand match scores come out, one per row, each saying how strongly the input resembled that stored pattern. The grid is a matrix, and this is what one layer of a neural network does. Nothing more exotic is coming.

the frozen set

The stored patterns were set by training and never move again. All of them together, on the order of a hundred billion individual numbers, are the weights.

They sit in a stack of forty to a hundred and twenty identical-shaped blocks. Each block holds two machines, which we open in the next module. Below the stack is a table that turns each chunk of text into its starting vector, and above it is a table that turns a vector back into a score for every chunk of text the model knows.

The stack is read-only while you talk to it. A pass consults those hundred billion numbers and leaves every one of them exactly as it found them.

In module 1.02 you put that property to work before you could see why it held. That was the module where you downloaded an open-weight model, pointed a local runner at it, and answered questions with no API key and no data leaving your disk. The argument for doing it rested on one claim: the weights are a file, and the file does not change. Now you can see the mechanism behind the claim. Answering a question is a pass of arithmetic over stored numbers, and arithmetic reads its inputs. That is why the model you run tonight computes exactly what it computed this morning, and will still compute it in ten years, whatever any vendor ships or retires in the meantime.

Check it yourself; it takes two minutes and it settles the idea better than another paragraph will. Open a conversation with any model and tell it something it has no way to know — the name of your dog, a number you have picked. Ask for it back and it answers correctly, which feels like learning. It is not. Your words are still sitting in the conversation, and the whole conversation is fed back through the stack from the beginning on every single turn, so the model is reading your dog’s name, not remembering it. Now open a fresh conversation and ask again. Nothing. The file on disk is byte-for-byte what it was before you started typing.

That experiment puts the two sets side by side. The first conversation answered from the live set, which held your dog’s name; the fresh one had only the frozen set, which never did. Anything a product appears to remember about you across sessions is a system outside the model that saved your text and pasted it back into a later prompt.

the live set

Now let’s return to the question we are following. A customer has typed this into a support box:

How many days do I have to return the C-40?

That sentence cannot touch the weights as it stands, because the weights are numbers and the sentence is text. Both steps that turn one into the other came up in module 1.03, and they are worth having in front of you again. The text gets chopped into tokens by a fixed lookup table, somewhere between fifty thousand and two hundred thousand entries assembled once from the training text; then each token is looked up in a second stored table and comes back as a vector of about eight thousand numbers.

Our question comes apart into thirteen tokens:

["How"]["␣many"]["␣days"]["␣do"]["␣I"]["␣have"]
["␣to"]["␣return"]["␣the"]["␣C"]["-"]["40"]["?"]

Read the end of that line again, because this is the module where it starts to cost something. “C-40” is three separate tokens, C, then -, then 40, and no entry in the table spells the product, because Aperture Instruments and its camera did not exist when the table was built. Whatever the model ends up saying about the C-40, it assembles on the spot out of three fragments that mean nothing on their own.

Each of those thirteen slots in the sequence is a position, and each position carries its own vector up through the stack in its own lane. That grid — thirteen positions wide, eight thousand numbers deep — is the live set entire. The model’s whole view of the customer’s question is that grid; the sentence and its letters reach the stack only as these vectors. The grid did not exist a second ago, and the moment the answer finishes it is discarded.

Both sets are now on the table, so it is worth stopping to see what a real model does with this exact question. The frozen set holds nothing about the C-40. The live set holds the question and nothing else — no policy document, no product page, no company. Here is one run, asked cold, quoted word for word.

Look at what the absence of a fact did and did not produce. There is no error. There is no blank. The model reached for the closest thing it had, which is a well-supported general picture of how return windows work, and every number it gave is true of some real retailer. What it could not do is tell the difference between a question it holds an answer to and one it does not, which is why it asked you to supply the missing half rather than announcing that the missing half exists.

Credit what the run gets right, because it is the product of real work: this model hedged instead of inventing a C-40 policy outright, and asking for the brand is the correct move. That behavior comes from post-training rather than from the stack, it is uneven across models and across phrasings, and it is not a guarantee. Give the same absence a prompt that already assumes the answer exists, or a format with no room to ask a question back, and the gap gets filled silently at the same fluency. We spend the first of module 1.06’s six runs watching exactly that happen.

Two sets of numbers: weights are stored and shared by everyone; the live numbers are yours alone and last one request. When someone says a model “knows” something, ask which set they mean. The answer changes what you should do about it.

Put the France question from module 1.03 beside the C-40 question and the rule does its first piece of work. “What is the capital of France?” is answered out of the frozen set: the pairing was in the training text millions of times, and no document is needed. “How many days do I have to return the C-40?” cannot be, because the frozen set never met the product; the answer has to arrive through the live set or it does not arrive at all. Same machinery, same arithmetic, and the honest answer to “does the model know this?” is different for the two questions because the word “know” is pointing at different sets.

what the weights actually hold

Aren’t the weights just compressed training data? Compression is the right instinct, and the storage is stranger than the phrase suggests: training compresses the data by finding the structure that generates it, and the structure is what gets stored. Four kinds of thing are worth telling apart.

Facts are there, and the recall path has been traced: middle blocks read an enriched representation out of the stored patterns at the subject’s position, and later attention carries it to where the prediction is forming. The machine that does the reading is the MLP, and we open it in the next module. A fact is stored about as strongly as it was common and consistent in the training text. Fifteen trillion tokens is enough text that a person reading day and night would need tens of thousands of years to finish it; a fact that appeared five times in all of that is stored weakly or not at all, and nothing tells the model which case it is in. Sit with that last clause, because we spend the first run of module 1.06 watching what it costs: the model has no way to distinguish a fact it holds firmly from one it never held, so it answers both with the same fluency. And a fact stored strongly is not the same as a fact stored alone: the training text that said Canberra also said Sydney, near the same words and far more often, and both associations sit under the same roof. The next module shows what happens when they are asked for at once.

Procedures are the part “compressed data” misses. Training discovers working algorithms, because an algorithm predicts better than a lookup table. The cleanest example is a two-head arrangement that finds an earlier occurrence of the current token, looks at what followed it last time, and predicts that. It is a general copy-and-continue routine, it runs on whatever is in the context, and it works on sequences that appeared nowhere in training. It is a large share of what looks like learning from your prompt. Researchers have also mapped a twenty-six head arrangement in a small model that resolves which name a sentence is about, and an arithmetic routine that lays the numbers around a clock face and adds them in sine waves. That one appears at the exact moment the model stops memorizing and starts generalizing.

World models exist and are narrower than the phrase suggests. A model trained only on Othello move lists, never shown a board, builds an internal picture of the board. A small classifier can read that picture out, and, decisively, you can edit it. Change the picture and the model’s next legal-move predictions change to match. Prediction pressure alone produced a model of the thing generating the text. Take the result in both directions: real internal models do form, and they form in well-structured domains and stay narrow.

The prior sits underneath all of it: the model’s defaults about what text looks like, which claims travel with which, what a policy document sounds like, what usually follows a question of this shape. Post-training reshapes it heavily toward helpful assistant behavior, and the substrate underneath is still a very strong set of expectations about plausible continuations. Hold onto it. It is the mechanism behind the most consequential failure these systems have, and module 1.06 is about watching it work.

There is one more finding worth carrying, because it is the most recent and the most surprising. Trace what happens when a model writes a rhyming couplet and you find it activating candidate rhyme words before generating the line that arrives at them, then building the line to land on the chosen target. Suppress the planned word and the whole line comes out different. Some amount of looking ahead happens inside a single pass.

Two honest limits on all of this. These procedures have mostly been located in small models on narrow tasks, and nobody has a full account of what a frontier model computes. And the tidy story of low blocks doing tokens, middle blocks doing meaning, and high blocks doing prediction has real evidence behind it and real exceptions; the work is distributed and redundant and does not respect that division neatly.

where the two sets meet

The run you watched failed for a reason we can now state precisely: the frozen set held nothing about the C-40, and the live set held nothing but the question. Neither set contained the answer, so there was no answer to be had.

That points at the repair, and it is the one you would reach for at work. If the model does not hold the fact, put the fact in front of it. Paste the returns policy above the question and ask again:

Returns policy, rev. 2026-02: Aperture Instruments accepts returns
of the C-40 within 45 days of delivery. Units returned after day 30
incur a 15% restocking fee. Opened lens kits are non-returnable.

How many days do I have to return the C-40?

This works, and we watch it work in module 1.06. But “now the model can see the document” explains nothing, and the machinery it glosses over is what the next module is about.

Look at what actually arrived. The document is more tokens in the same flat sequence, chopped by the same lookup table, turned into vectors in the same way — a few hundred positions now instead of thirteen. Every position arrives with the same standing: the policy tokens read as ordinary text, exactly as the question does, and any link between the two is left for the machinery to work out. And the number the answer needs, 45, is sitting in one position of its own, dozens of slots away from the position where the answer will be written.

So two separate things have to happen inside the stack, and each block holds one machine for each of them. One machine moves information between positions, which is how 45 travels from where it sits to where it is needed, and what it writes is the context term of the thought state. The other works within a single position, comparing what is already there against the frozen patterns and adding what was filed under the matches, and what it writes is the memory term. In the next module we watch each of them do its piece of our question, and of the France question, and we add the two together with the token’s own baseline to get the one vector that chooses the word.

what you can do with this today

The two sets are a diagnostic you can run in the middle of a working day, and it costs nothing.

When an answer is wrong, ask where the right answer was supposed to come from. If it was supposed to come from the weights, you are relying on a fact stored in proportion to how often it appeared in the training text, with no signal telling you whether it is there. If it was supposed to come from the prompt, then the question is whether the text was present and whether anything pointed the machinery at it. The first case is fixed by putting the fact in the context. The second is fixed by rewording, reordering, or removing whatever else in the prompt was competing for attention. Those two problems look identical from the outside and have opposite fixes.

In the next module we build the vector that both fixes act on, one source at a time.