Every score is a dot product

Every comparison in the last module was the same piece of arithmetic. Take two lists of three numbers, multiply each pair, and add the three products together. When the position composing an answer went looking for a number of days and landed on the policy line, that was this operation. When a stored pattern matched the residual stream closely enough to contribute, that was the same operation again. A model with a hundred billion weights runs it more than ten million times per token, on lists thousands of numbers long, which comes to about a hundred billion multiplications, and it is nearly all the arithmetic there is: what surrounds it is a thin layer of scaling and re-centering.

Earlier we followed a single token through a complete pass: the tokenizer cut the text into chunks, an embedding table turned each chunk into a vector of several thousand numbers, those vectors travelled up through forty to a hundred and twenty blocks that each read what arrived and added to it, and one token was drawn from a distribution at the top. Two sets of numbers were in play the whole way: the weights, fixed the day training ended, and the activations, the vectors built from your text for this run and discarded when the answer ends.

This module idealizes three things to make the arithmetic visible, and this paragraph is the one place it says so. First, a real vector holds thousands of numbers; ours hold three. Second, a real slot stands for nothing you could name and is crowded with several properties at once, because the model has far more properties to represent than slots to keep them in (the next module names that packing: superposition); each of our three slots is given one clean, nameable property so that you can read the arithmetic off the page. Because our slots carry names, the sign of a product reads directly as agreement or opposition on a named scale. A learned slot carries no name, so in a real model the sign of a score, a score near zero, and the length of a vector are geometry in that model’s representation, and what they say about meaning has to be established separately. Words used in the same contexts, such as hot and cold, tend to receive similar vectors in learned embeddings, so a pair of antonyms can sit close together, and the length of a learned vector is not by itself a measure of how certain the model is or of how often anything appeared in training. Third, the four short texts we compare are treated as vectors partway up the stack, after several blocks of attention have mixed their tokens together, rather than as rows straight out of the lookup table, because a phrase acquires a meaning of its own only once that mixing has happened. The arithmetic is exactly right; what is idealized is how tidy the things being compared are.

So we go down one more level: what that operation is, why a list of numbers can carry a meaning at all, how to read the number it returns, and what the weights hold besides facts. In our next lesson we use all of it to build the thought state from its three sources.

We keep the same specimen: Aperture Instruments, its C-40 camera, and a customer who wants to know how long they have to send one back.

every score comes from the dot product

A model has one job at every step: read the text so far and produce a score for every token it could say next. A machine cannot weigh words against each other, so the text becomes numbers before anything else happens, and every judgment about that text then has to become a calculation on those numbers. Which earlier word does this one depend on. Is this fragment part of a name or part of a date. What kind of answer does a question shaped like this usually get. All of it, arithmetic.

So each token becomes a long list of numbers, and the list works like a set of coordinates: tokens that play similar roles in text come out pointing in similar directions. Meaning is stored as a position. Once meaning is a position, asking whether this piece of text has anything to do with that one stops being a question about language. It becomes a question about two lists of numbers, and getting the answer takes one cheap calculation.

A list of numbers used this way is a vector, and before we run any arithmetic on two of them, be clear about what you are looking at.

A real one holds about eight thousand numbers. Ours will hold three, because three fits on a page, and each of the three is given one clean, nameable property. In a real vector a positive value marks the presence of some trait the model discovered during training and a negative value marks its absence or its opposite, and no slot has a name you could read off; ours are labelled so that the arithmetic is legible.

Each slot measures a scale with two opposite ends and a zero in the middle:

Slot	Its negative end	Its positive end
1	casual speech	formal policy language
2	grants the reader a permission	imposes an obligation
3	names a physical object	names a span of time

Now you can read a sign. A positive number means the text sits toward that scale’s positive end, a negative number means it sits toward the negative end, and a number near zero means the scale has nothing to say about this text. The size of the number says how strongly.

That answers the first question anyone asks when they see one of these lists: why does a single list carry both positive and negative numbers? Because a piece of text lands at the far end of some scales, the opposite end of others, and off in the middle of the rest, and one list has to record all three verdicts at once. A negative entry is not a deficiency or a missing value. It is a measurement pointing the other way.

Here are four short pieces of text, measured on our three scales:

The text	slot 1	slot 2	slot 3
a · “within 45 days of delivery”	+0.3	−1.2	+0.8
b · “how many days do I have to return”	+0.5	−1.0	+0.6
c · “don’t lose the lens cap”	−0.4	+1.1	−0.5
d · “the licensee shall submit quarterly reports”	+1.0	+0.5	+0.4

Take a and b first. Both are mildly formal, both are about what the customer is allowed to do rather than what they must do, and both concern a span of time. Two texts, three scales, and the same verdict on all three.

Here is the calculation that detects it.

Take two lists of numbers of the same length. Multiply the first against the first, the second against the second, and so on down the line. Add up the results.

Look at what each multiplication did, because this is where the signs stop being decoration and start doing work.

Slot 2 is the interesting one. Both lists carry a negative number there, −1.2 and −1.0, and multiplying two negatives gives +1.20, the largest contribution of the three. The two texts agree emphatically that this is about a permission rather than an obligation, and agreement at the negative end of a scale counts exactly as much as agreement at the positive end. Slots 1 and 3 do the same thing at the other end: positive times positive, both small and positive out.

So a pair of numbers on the same scale settles into one of three cases. Same sign, either sign, and the product is positive: the two texts sit at the same end, and the score goes up. Opposite signs, and the product is negative: they sit at opposite ends, and the score goes down. Either number near zero, and the product is near zero whatever the other one says: that scale abstains, because one of the two texts had no opinion to register on it.

Add the three together and you have a tally of agreements against disagreements, with each scale voting in proportion to how strongly both texts committed on it. That is what 1.83 means.

Now run the same arithmetic on a pair that disagrees. Keep list a, and compare it against c, “don’t lose the lens cap”: casual where a is formal, an obligation where a grants a permission, about a physical object where a is about a span of time. Opposite verdict, all three scales.

Same machine, opposite verdict, and note that the machine did not detect absence. It detected opposition, and reported it with a sign.

Which leaves the third case, the one people usually get wrong. Compare a against d, “the licensee shall submit quarterly reports”: 0.3 × 1.0 = 0.30, then −1.2 × 0.5 = −0.60, then 0.8 × 0.4 = 0.32. The total is 0.02, near enough to nothing. Both texts are formal and both concern time, so slots 1 and 3 agree; slot 2 disagrees hard enough to cancel both. A score near zero does not mean the two texts are opposites, and it does not always mean they have nothing in common. It means the agreements and the disagreements came out even. The other route to zero is simpler: one list sits near zero on the scales the other one commits to, so there is nothing to multiply against.

three readings of the dot product

Three scores, one operation, and they lay out along a line. In physics the dot product measures how much two forces pull in the same direction. On our labelled vectors it measures how far two texts agree on the scales both commit to, and in a language model it measures how far two vectors point the same way in the model’s own representation. Read the result as a compass, and the three cases you just worked are its three readings:

The score	Physically	On our named scales
large positive (a · b = 1.83)	constructive — the two reinforce each other	agreement: the vectors sit at the same end of the scales both commit to, such as permission, time span, formal
near zero (a · d = 0.02)	orthogonal — independent	no net agreement: the agreements and disagreements cancel, or one vector sits near zero on the scales the other commits to; a and d share two of three traits and still land here
large negative (a · c = −1.84)	destructive — the two compete	opposition: a positive value in one meets a negative value on the same scale in the other, and the score is pulled down; casual against formal, obligation against permission

Keep that table. Every time this part says “matches,” “lines up,” “opposes,” or “cancels out,” it is naming one of those three rows.

That line is worth having the vocabulary for, because it is how the field talks.

Two vectors pointing the same way are aligned, or parallel. Two pointing opposite ways are antiparallel, and what they share is a negative correlation across their slots: wherever one sits toward an end of a scale, the other sits toward the opposite end. Two vectors at right angles to each other are orthogonal: their dot product is zero, because the agreements and disagreements across their slots cancel exactly or one sits at zero wherever the other commits, and the pair can still share traits, as a and d do. Orthogonal is the middle of the scale, and it is where the overwhelming majority of pairs sit in a real model, because in eight thousand dimensions there is a great deal of room for two directions to sit at right angles to one another.

The arithmetic behind the picture is one line, and it explains why the scores landed where they did:

a · b = |a| × |b| × cos θ. The score is the length of the first vector, times the length of the second, times the cosine of the angle between them.

Read the three factors separately, because they answer different questions. The two lengths measure how emphatic each vector is: how far from zero it sits, which you get by squaring every slot, adding those squares up, and taking the square root. The cosine measures direction alone, and it runs from +1 when the two point the same way, through 0 at right angles, to −1 when they point opposite ways. Multiply direction by both magnitudes and you get the raw score the machinery actually computes.

Divide the two lengths back out and you are left with the cosine by itself, which is called cosine similarity. That is the number under each marker in the figure above: our agreeing pair comes to 0.98, about twelve degrees apart; the opposed pair to −0.98, about a hundred and sixty-nine degrees apart; the canceling pair to 0.01, which is a right angle to within a degree. When someone says two pieces of text are “similar” in a vector space, this is nearly always the number they mean, because dividing the lengths out stops a merely emphatic vector from scoring high against everything it half-agrees with.

You can watch the two measures part ways by changing one vector’s length and nothing else. Double every slot of a, so that “within 45 days of delivery” sits at the same end of every scale as before, twice as far out: 2a = (0.6, −2.4, 1.6). Its length doubles too:

$$|2\mathbf{a}| = \sqrt{0.36 + 5.76 + 2.56} = \sqrt{8.68} \approx 2.946$$

Now take the raw dot product against b, “how many days do I have to return”:

$$2\mathbf{a} \cdot \mathbf{b} = 0.30 + 2.40 + 0.96 = 3.66$$

The raw score doubles, from 1.83 to 3.66, and so does every score 2a takes part in: −1.84 against c becomes −3.68, and 0.02 against d becomes 0.04. Cosine similarity divides the new length back out, and the factor of two cancels:

$$\cos(2\mathbf{a}, \mathbf{b}) = \frac{3.66}{2.946 \times 1.269} = \frac{3.66}{3.738} \approx 0.98$$

Normalizing goes one step further. Divide a by its length of 1.473 and you get the unit vector â = (0.204, −0.815, 0.543), with length 1.000. Divide 2a by its length of 2.946 and you get the same â, so the two have identical cosine similarity with every other vector. Doubling left a at the same end of every scale and changed the size of every raw score it takes part in. The raw score mixes direction with length, and cosine similarity keeps the direction alone.

Both measures are worth keeping straight. Cosine similarity is what you reach for when comparing meanings, in a retrieval system or a clustering job. The raw score, lengths included, is what the network itself runs, so a longer vector pushes every score it takes part in further from zero, in whichever direction the angle between the two sends it, exactly as the doubled a did. What that length stands for in a learned model is a separate question the arithmetic does not answer. The arithmetic itself matters in the next module, where three vectors are added into one, and a wrong answer often turns on one of them arriving at a higher magnitude than another.

That is the question the network asks, more than ten million times per token: does this match that? The operation has a name, the dot product, and you now hold the vocabulary that travels with it: vectors, their magnitude and their direction, orthogonal for the pair at right angles, cosine similarity for the comparison with magnitude divided out. Keep all of it. Every paper, every library, and every engineer you work with uses those words for exactly the arithmetic you just watched.

Watch External visual explanation

The first third of Grant Sanderson’s Transformers, the tech behind LLMs (27 min, 3Blue1Brown) shows the same arithmetic from the other side: a direction in the embedding space that stands for one trait, and the dot product as the test of how far a word’s vector leans along it. His worked traits are gender and plurality; ours are register, permission and time. The text version has the same figures.

Now stack thirty-two thousand stored vectors into a grid and run the comparison against every one of them at once. One vector goes in; thirty-two thousand match scores come out, one per row, each saying how strongly the input resembled that stored pattern. The grid is a matrix, and this is what one layer of a neural network does. Nothing more exotic is coming.

the weights: fixed at training

The stored patterns were set by training and never move again. All of them together, on the order of a hundred billion individual numbers, are the weights, fixed the day training ended. Nothing you do while talking to the model moves one of them.

They sit in a stack of forty to a hundred and twenty identically shaped blocks. Each block holds two machines, attention and the MLP. Below the stack is an embedding table that turns each token into its starting vector, and above it is an unembedding table that turns a vector back into a score for every token the model knows.

The stack is read-only while you talk to it. A forward pass consults those hundred billion numbers and leaves every one of them exactly as it found them.

You put that property to work the first time you downloaded an open-weight model, pointed a local runner at it, and answered questions with no API key and no data leaving your disk. The argument for doing it rested on one claim: the weights are a file, and the file does not change. Now you can see the mechanism behind the claim. Answering a question is a pass of arithmetic over stored numbers, and arithmetic reads its inputs. That is why the weights you run tonight are byte-for-byte the weights you ran this morning, and will still be in ten years, whatever any vendor ships or retires in the meantime. The answers computed from them can still differ from run to run, because of the sampling draw and the serving arithmetic the last module described; what stays fixed is the file.

Check it yourself; it takes two minutes and it settles the idea better than another paragraph will. Open a conversation with any model and tell it something it has no way to know: the name of your dog, or a number you have picked. Ask for it back and it answers correctly, which feels like learning. The mechanism says what happened instead. Your words are still sitting in the conversation, every turn sends the whole conversation to the model again as its prompt, and attention reads your dog’s name off the page. A serving engine can skip recomputing the earlier turns from its KV cache, as the last module showed, but the cache holds only what that text determines, so the name reaches the model because the text is there. Now open a fresh conversation with persistent memory out of the picture: switch off any memory feature in your chat settings, use a temporary chat, or send the question through a direct API call, where the request carries only the text you send. Ask again. Nothing. The file on disk is byte-for-byte what it was before you started typing.

That experiment puts the two sets side by side. The first conversation answered from the activations — the vectors built from your text for this run and discarded when the answer ends — and those held your dog’s name. The fresh one had only the weights, which never did. Anything a product appears to remember about you across sessions is a system outside the model that saved your text and pasted it back into a later prompt.

the activations: built fresh for each run

Now return to the question we are following. A customer has typed this inquiry into a support box:

How many days do I have to return the C-40?

That sentence cannot touch the weights as it stands, because the weights are numbers and the sentence is text. Two stored tables turn one into the other. First, the tokenizer cuts the text into tokens against a lookup table of fifty thousand to two hundred thousand entries, assembled once from the training text before training began. Second, an embedding table turns each token into a starting vector of about eight thousand numbers.

Our question decomposes into thirteen tokens:

["How"]["␣many"]["␣days"]["␣do"]["␣I"]["␣have"]
["␣to"]["␣return"]["␣the"]["␣C"]["-"]["40"]["?"]

Read the end of that line again, because this is the module where it starts to cost something. “C-40” is three separate tokens, C, then -, then 40, and no entry in the table spells the product, because Aperture Instruments and its camera did not exist when the table was built. Whatever the model ends up saying about the C-40, it assembles on the spot out of three fragments that mean nothing on their own.

Each of those thirteen slots in the sequence is a position, and each position carries its own vector up through the stack in its own lane. That grid, thirteen positions wide and eight thousand numbers deep, is the activations entire. The model’s whole view of the customer’s question is that grid; the sentence and its letters reach the stack only as these vectors. The grid did not exist a second ago, and the moment the answer finishes it is discarded.

Both sets are now on the table, so it is worth stopping to see what a real model does with this exact question. The weights hold nothing about the C-40. The activations hold the question and nothing else: no policy document, no product page, no company. One real run of this question, asked cold, shows what a model does with that gap.

Look at what the absence of a fact did and did not produce. There is no error. There is no blank. The model reached for the closest thing it had, which is a well-supported general picture of how return windows work, and every number it gave is true of some real retailer. It also said, in its first sentence, that it needed to know what brand and product the C-40 is, and then it filled the space with general figures anyway. One output cannot show what happened inside the model to produce that mix; module 1.01 described what interpretability research found about familiarity and declining to answer in one studied model.

Credit what the run gets right, because it is the product of real work: this model hedged instead of inventing a C-40 policy outright, and asking for the brand is the correct move. That behavior comes from post-training rather than from the stack, it is uneven across models and across phrasings, and it is not a guarantee. Give the same absence a prompt that already assumes the answer exists, or a format with no room to ask a question back, and the gap gets filled silently at the same fluency.

Two sets of numbers: the weights are stored and shared by everyone; the activations are yours alone and last one request. When someone says a model “knows” something, ask which set they mean. The answer changes what you should do about it.

Put the France question beside the C-40 question and the rule does its first piece of work. “What is the capital of France?” is answered out of the weights: the pairing was in the training text millions of times, and no document is needed. “How many days do I have to return the C-40?” cannot be, because the weights never met the product; the answer has to arrive through the activations, built from a policy you put in the context, or it does not arrive at all. Same machinery, same arithmetic, and the honest answer to “does the model know this?” is different for the two questions because the word “know” is pointing at different sets.