One vector, three sources

At the top of every pass, one position holds one vector, and that vector decides the word. Everything the model has read, remembered, and worked out has to be in it, or it plays no part in what gets said. In module 1.03 we called it the thought state and wrote it as h. In module 1.04 we learned the arithmetic that reads it, the dot product, and the two sets of numbers it is built from: the frozen weights, shared by everyone, and the live vectors built from your text and discarded when the answer ends.

Now we build h. It is a sum of three sources, and our whole job here is to watch each one arrive:

  h_initial   the baseline — the current token's own embedding
+ h_ctx       the attention context — pulled in from earlier positions
+ h_prior     the parameter memory — injected by the MLP from the frozen weights
─────────────────────────────────────────────────────────────────────────────
= h           the thought state at this position

We keep both specimens from module 1.03. "What is the capital of France?" is the question whose answer lives in the weights, so the memory term does the work. "How many days do I have to return the C-40?" is the question whose answer lives nowhere in the weights, so the context term has to do it, and when the context is missing the memory term fills the slot with something else. One equation, two questions, and by the end you will be able to say, for any wrong answer, which of the three terms was supposed to supply the right one.

the additive highway

Start with how anything gets into h at all, because the design decision here determines what a model can do with something it worked out early.

Follow one position. In the policy sentence “Aperture accepts returns of the C-40 within 45 days of delivery,” take the position holding the final word, “delivery.” It arrives at the bottom of the stack holding its starting vector, straight from the lookup table — the generic word, carrying nothing yet about this company or this camera. That starting vector is h_initial.

Block 1 reads that vector. It does not replace it. It computes a vector of its own, the same length, and adds it in slot by slot. What leaves block 1 is the sum of the two. Block 2 receives that sum, reads it, computes its own contribution, and adds that in. Block 3 does the same to the result. Forty to a hundred and twenty times, all the way up.

Watch it happen on the three toy slots from module 1.04 — formal register, obligation against permission, time against object:

Three things in that table are worth naming.

The position arrived generic and left committed. Its final row is list a from module 1.04 — “within 45 days of delivery,” the vector that scored 1.83 against the customer’s question. The word “delivery” did not carry that meaning when it came out of the lookup table. It acquired it on the way up, from blocks that read the words around it.

The third block’s first number is negative. That block pulled slot 1 back down, undoing part of what the blocks below had written there. So a contribution pointing the opposite way cancels an earlier one. That is module 1.04’s antiparallel case, two vectors pointing opposite ways with a negative product between them, doing an ordinary day’s work. Nothing gets erased, but a note can be crossed out by a later note.

And every block reads the accumulated total, never the previous block’s contribution on its own. Block 3 sees +0.4, −1.1, +0.7 — the starting numbers plus everything written since. This is the part that matters: information written low in the stack is still legible high in the stack. A block that works out in position nine that “40” belongs to a product name has put that finding somewhere every later block can read it, seventy blocks later, while the answer is being composed.

The running sum is the residual stream, and the name comes from the arithmetic you just watched: each machine computes a small correction, a residual, and adds it to what arrived, so what flows up the stack is the running total of every correction written so far. Think of it as an additive highway, one lane per token position. Nothing in a lane is ever removed; the machines along it can only merge more in. Early blocks merge in the simple things: this token is a number, this one is part of a proper name. Middle blocks merge in meaning and relationships. Late blocks merge in something close to a prediction, and everything already merged stays readable to every block above.

Now look at what each block adds, because that is where the three sources come from. Every block holds two machines, and each writes one kind of contribution onto the highway. The first machine reads other positions and adds what it found there: that is h_ctx, the attention context, and it accumulates block by block. The second machine reads only this position, matches it against stored patterns, and adds what was filed under the matches: that is h_prior, the parameter memory, and it accumulates too. Sum the starting vector with everything both machines wrote across every block, and you have h.

That is the equation at the top of the module, and it is a bookkeeping identity rather than a theory. Every number in h came from the lookup table, from an attention machine, or from an MLP, because those are the only three things that ever write to the highway. Group the contributions by which of the three wrote them and the equation falls out. We take the three groups in order.

the baseline: what the token itself brings

The smallest of the three sources is the one the position starts with, and it is worth a moment because it does more than most people expect.

h_initial is the vector retrieved from the input embedding table when the token enters the network. It is the same for that token wherever it appears and whatever surrounds it, before any block has written anything: the token’s own baseline statistics, learned in training, of what kind of thing it is and what tends to follow it. For a question mark, h_initial carries the baseline traits of a punctuation boundary at the end of a question — the statistical expectation that a response follows. For a period, the expectation is a continuation. For a #, a heading or a comment or a block of code.

Here is why that matters in practice. The last token of your prompt is the position that will write the first token of the answer, and its baseline is where that position starts before attention and memory add anything. Swap the final punctuation and you shift the starting coordinate.

Baseline shift. Replacing a prompt’s final punctuation mark shifts h_initial at the position that writes the answer. A question mark sets a baseline expectation of an answer; a period or a # sets an expectation of a continuation or a syntax block. If the surrounding prompt is clear, the attention context readily overpowers a stray character. If the prompt is ambiguous, that baseline shift can tip the model toward a different first token.

You can test the rule on the C-40 question in a minute. Ask “How many days do I have to return the C-40?” and the first token is almost always the start of an answer. Ask “How many days do I have to return the C-40” with nothing at the end and the model will sometimes complete your sentence instead — “…if I bought it on sale?” — because the last position’s baseline is a bare word mid-sentence, and the context did not say the question was over. It is a small effect and a real one, and it explains a class of oddities that otherwise look random: the prompt that ends in a colon and gets a list, the one that ends in an open bracket and gets code.

Two consequences follow, and both come back later. The baseline is small compared with the other two terms in any well-formed prompt, which is why prompts are forgiving of typos. And it is never zero: even before the stack has done anything, the position writing the answer already leans somewhere.

the attention filter: building the context

Now the term that reads your text.

Consider "What is the capital of France?" Seven tokens, seven positions, and the answer will be written at the position after the question mark. Something at that position has to gather what the sentence is about, and it cannot do it by averaging: combine all seven earlier vectors into a flat mean and you get a vector that is a little bit “what,” a little bit “the,” a little bit “France,” and useless. The mechanism that builds h_ctx has to be selective. It works as a relevance filter, and it has four steps.

Prompt tokens:   [What]   [is]   [the]   [capital]   [of]   [France]   [?]
                                                                        ▲
                                                                  current slot

Step one, the query. In each block, the vector at the current slot is multiplied by a frozen grid of weights and comes out as a short vector, a few dozen to a couple of hundred numbers long, called the query. Read in words, this position’s query says something like “I need a geographic entity associated with the nation identified in this sentence.” Hold that translation lightly. A query is a vector, and it means something only when compared against another one; the comparison is the dot product you learned in module 1.04.

Step two, the keys. Every position in the sequence multiplies its own vector by a second frozen grid and comes out as a key, the same short length, advertising what that position holds:

• "the" → grammar token, a determiner
• "capital" → administrative center, a geographic attribute
• "France" → sovereign nation, a European country

Step three, the relevance check. The model takes the dot product of the query against every key. One number comes back per position, saying how strongly what this slot is after lines up with what that position holds:

• Q · K("the") ≈ 0.05 — negligible
• Q · K("capital") ≈ 0.85 — high
• Q · K("France") ≈ 0.95 — high

Those raw scores are then put through softmax, the same step that turns logits into probabilities at the top of the stack. It converts the pile of scores into fractions that add to one:

• "France" → 52%
• "capital" → 43%
• everything else combined → 5%

Step four, the value sum. Each position multiplies its vector by a third frozen grid and comes out as a value: the content it will hand over if asked. The model scales each value by the fraction its key earned and adds them all together:

h_ctx = (0.52 × V("France")) + (0.43 × V("capital")) + (0.05 × V(others))

That vector is added into the residual stream at the current slot. It carries the combined concept of France and capital city, concentrated at the one position that has to write the answer.

The three short vectors are called the query, the key, and the value, and the names come from database lookup: you match a query against keys, and what you get back is the value filed under the key that matched. The difference here is that you never get exactly one value. You always get a blend of all of them, weighted by how well each key scored, and a confident match means one value dominates the mixture. The machine that does all this is an attention head, and every block runs dozens of them at once, each with its own three grids of weights, and so each with its own notion of what to match on.

Two constraints shape every set of scores. A position can only be read from if it came earlier in the sequence: the model is predicting what comes next, so a position is never allowed to look at text further along than itself. And the shares always add to one, so attention redistributes what is already there rather than inventing anything. If nothing in the sequence matches, the query still gets its shares — spread thinly across positions that hold nothing useful.

the same filter on our question

That is the mechanism. Now run it on the question where it has to carry the whole answer.

The prompt holds the policy sentence “Aperture accepts returns of the C-40 within 45 days of delivery,” and the model is composing its answer at the last position in the sequence. The number it needs is eleven positions back. Nothing has carried it forward on its own: the residual stream at the last position holds only what the blocks below wrote there, and 45 was written into a different lane entirely.

So the composing position forms a query that amounts to a number of days, attached to this product. It runs the dot product against the key at every earlier position. Keys on grammar words score near zero. The key on the document’s product name scores high, and so does the key on the number sitting beside it. Softmax hands most of the shares to that region of the document, and the value carried by the position holding “45” is blended into the residual stream where the answer is being written. The number 45 has crossed from one lane to another. h_ctx at the composing position now carries it.

Now notice which part is stored and which part is live, because this is module 1.04’s two sets of numbers showing up inside a single machine. The three grids that turn a vector into a query, a key, and a value are frozen weights, identical for every user and every request. The scores they produce are computed fresh from your text, every single time. The weights hold a rule for what to match on; your context decides what that rule lands on.

That is why a frozen network can answer about a product invented this morning. A head that learned “read from the number nearest this product name” will do that for the C-40 exactly as readily as for anything it saw in training, because the rule never mentions the C-40 at all. And it is why deleting the policy changes everything: the same query, run against a sequence that holds no number, earns its shares from positions that carry nothing about days, and h_ctx arrives at the composing position without one. The slot is still there. What fills it comes from the third source.

parameter memory: injecting the prior

Once attention has added h_ctx to the stream, the stream passes into the second machine in the block. This one stays inside a single position. It takes the vector there and works on that alone, and its job is to add what the model knows to what the context has assembled.

Start with the vector at one position, all eight thousand numbers. Sitting in the block is a grid of about thirty-two thousand stored patterns, each one a vector of the same length, fixed since training ended. Run the comparison down the grid, eight thousand numbers against eight thousand numbers, thirty-two thousand times over, and you get thirty-two thousand scores. That one sweep is about a quarter of a billion multiplications, for one position in one block. Each says how strongly this position’s current state resembles that particular stored pattern.

Then comes a step that sounds crude and is doing real work: every score that came out negative is set to zero. Recall what a negative score meant on the spectrum in module 1.04. It means the running vector points the opposite way from that stored pattern, and a row the state points away from has nothing to add, so it stays quiet. Only rows that genuinely matched survive the cut, and in practice a small fraction of the thirty-two thousand do.

Now the write. Paired with every stored pattern is a second stored vector, its contribution: what this row adds to the stream when it matches. Take each surviving score, multiply its paired contribution by it, and add all of those into the vector at this position. A row that matched strongly writes a lot, a row that matched weakly writes a little, and a row that was zeroed out writes nothing at all. That sum is h_prior.

Watch it on the France question. By the time the stream reaches the middle blocks, h_ctx at the last position carries France together with capital. Rows in the grid whose stored pattern looks like that combination fire, and what is filed under them is a contribution pointing toward Paris. The frozen weights are acting as an associative memory bank: recognize a pattern, hand back whatever was stored with it.

The machine is a multi-layer perceptron, almost always shortened to MLP and sometimes called the feed-forward layer. Each stored pattern with its paired contribution is a neuron, and “the neuron fires” means its match score came out high. It never looks at any other position. Looking elsewhere is attention’s job. Roughly two-thirds of all the model’s numbers sit in these paired vectors, and this is where most remembered facts appear to live. That is why this machine, and not attention, is what fails when a model does not know something, and what wins when the model knows something you did not ask for.

Two machines, two jobs. That split is tidier than the truth, because plenty of real behavior smears across both, and it is still close enough to reason with.

fact collision

Here is the case that shows why the three-way split is worth keeping in your head rather than just admiring.

Ask a model "What is the capital of Australia?". Attention does its work exactly as before: h_ctx at the last position carries Australia together with capital, and that combination is what the MLP is asked to recognize.

Now think about what the training text held. Every mention of Australia’s capital said Canberra. But text mentioning Australia at all, in travel, sport, business, and weather, said Sydney far more often than it said Canberra, and it said it near the same words. So two things sit filed in the frozen weights under a pattern shaped like Australia: a Canberra association attached to the capital part of the pattern, and a much heavier Sydney association attached to the Australia part. The MLP fires both. h_prior comes back as a blend, and the Sydney direction can arrive at higher magnitude than the Canberra one, because magnitude tracks how often and how consistently a pairing appeared.

Now the two terms are in tension. h_ctx says the capital, which favors Canberra. h_prior says the city that goes with Australia, which favors Sydney. If the memory term’s pull is stronger than the context term’s, the model writes the high-frequency distractor instead of the fact it was asked for. It has produced a part standing in for the whole, the famous city for the country. That failure is a fact collision: the memory term holds more than one association for the pattern in front of it, and the loudest one wins rather than the one the question named.

Grade this honestly before you carry it. A large current model asked that exact question says Canberra, because the capital pairing is well rehearsed and post-training sharpened it. The collision shows itself on rarer facts, on smaller models, and on any question where the asked-for association is quieter than a neighbor that shares its pattern: the second-largest city, the previous CEO, the drug’s older name. The mechanism is the same everywhere. Nothing in the MLP checks that the loudest association is the one the question asked for, because nothing in the MLP knows what a question is. It matches patterns and hands back what was filed.

You have seen the softer version of the same failure already. In module 1.03 the C-40 question, asked without its policy, came back with 30 days: h_ctx carried the shape of the question and none of the answer, and h_prior supplied the number most return policies say. That is fact collision with one side of the collision empty. Here both sides are full, and the wrong one is louder.

One caution before we go on. It is tempting to imagine a neuron for Canberra and a neuron for the C-40, each holding its concept. Models represent far more concepts than they have numbers to spare, so they pack them into overlapping directions instead of separate ones, and a single neuron will fire for DNA sequences, legal citations, and one flavor of Python syntax. That packing is called superposition. It is why you cannot find a model’s belief about something and edit it cleanly, and why techniques that try tend to damage neighboring facts.

one score, three votes

We have the three sources. Now watch them decide the word, because this is where the equation stops being a description and starts being a tool.

At the top of the stack, the finished vector h at the last position is scored against every token the model could say next. Each candidate token A has a stored vector of its own in the unembedding table, its candidate vector u_A, and its raw score, its logit, is the dot product of the two:

ℓ(A) = h · u_A

You know what that number means from module 1.04: how many traits the thought state shares with the identity of that word, agreements minus disagreements, weighted by how strongly both committed. Now use the fact that h is a sum. The dot product distributes over addition, so the score for any candidate is the sum of three separate scores:

ℓ(A) = (h_initial · u_A) + (h_ctx · u_A) + (h_prior · u_A)

Read that as three votes. The baseline casts one, the context casts one, the memory casts one, and the logit is the tally. Every candidate in the vocabulary gets its own tally, softmax turns the tallies into shares, and the draw is made.

That is exactly the picture we need for the Australia question. Here it is with illustrative numbers — shaped to show the mechanism, measured from no particular model, with the shares taken over these four candidates alone:

Look at where each candidate’s strength comes from. Canberra wins the context vote, because capital is what the question said. Sydney wins the memory vote, because Australia is what the training text said most often. The baseline votes for nothing in particular, which is what a well-formed question’s baseline usually does. And the tally goes to Sydney, by 0.6 of a logit, which softmax stretches into a lead of nearly two to one.

Now the same picture for the France question, where the two big terms agree. The context vote for Paris is large, the memory vote for Paris is large, they add rather than fight, and the logit clears the runners-up by three points, +5.0 for Paris against +2.0 for Marseille. That is the table you saw at the end of module 1.03, and it is a lead of twenty to one after softmax. Agreement between the context and the memory is what a confident right answer looks like from the inside. Disagreement between them is what a fact collision looks like. And a context vote that carries no fact at all, as in the C-40 question with its policy deleted, is what a confabulation looks like: the memory vote decides alone, and the pass runs to the end regardless.

Three votes, one score. Every logit is the sum of three dot products: what the token itself brought, what attention carried in from the context, and what the MLP injected from memory. A right answer is usually the context and the memory agreeing. A confabulation is the memory voting alone. A fact collision is the memory outvoting the context. The pass runs to the end in all three cases, and nothing inside it records which one happened.

Two honest limits on the picture. In a real model the three groups are not separable after the fact. The MLP in block 40 reads a stream that already contains what attention wrote in block 39, so “the memory vote” is memory reacting to context, and the equation groups contributions by who wrote them, never by who caused them. And the numbers in the table were shaped to make the mechanism visible. What is exact is the structure: three writers, one sum, one score per candidate, and a softmax that turns small leads into wide margins.

what you can do with this today

The equation is a diagnostic you can run in the middle of a working day, and it costs nothing. When an answer is wrong, ask which term was supposed to supply the right one.

If it was supposed to come from h_ctx, the question is whether the fact was in the text at all and whether anything pointed the machinery at it. Put the fact in, or move it closer, or remove whatever else in the prompt was competing for the same shares. This is what fixes the C-40 question.

If it was supposed to come from h_prior, you are relying on a fact stored in proportion to how often it appeared in training, with no signal telling you whether it is there or whether a louder neighbor shares its pattern. Rephrasing sometimes helps here, because a different phrasing fires different rows. Putting the fact into the context is the move that works, because it turns a memory vote into a context vote. It is not a guarantee: a well-worn association can still outvote a fact sitting in a long window, and where the fact sits in that window matters too. This is what fixes the Australia question, and it is the reason the next module spends six runs on what the text around a question does.

And if the answer starts wrong from the very first token in a way that looks arbitrary, check the end of your prompt. h_initial is the smallest term, and it is the one you can shift with a single character.

Here is the whole pass on one card, in the vocabulary you now own:

1. INPUT
   Tokens: ["What", "is", "the", "capital", "of", "France", "?"]
   The current slot (?) starts from h_initial, its own embedding.

2. ATTENTION — communication between positions
   The query from "?" is dotted against every key.
   Shares: France 52%, capital 43%, others 5%.
   h_ctx is blended from the values and added to the stream.

3. MLP — memory retrieval at one position
   Stored patterns recognize "France" + "capital".
   h_prior, pointing toward "Paris", is computed and added.

4. RESIDUAL STATE
   h = h_initial + h_ctx + h_prior

5. UNEMBEDDING AND SOFTMAX
   Logit per candidate:  ℓ(A) = h · u_A  — three votes in one sum
   Probability:          p(A) = e^ℓ(A) / Σ e^ℓ(t)
   Small logit gaps become decisive shares.

6. ONE TOKEN OUT
   Drawn, appended, and the whole pass runs again for the next one.

In the next module we stop building and start predicting. Same weights, same question, six different arrangements of text around it, six different answers, and each one is a question of which of the three votes wins.