How attention builds the context term

At the top of an inference pass, a single vector at the last position decides the next token. Everything the model has read, remembered, and worked out has to be in that vector, or it plays no part in what gets said. Earlier we called it the thought state and wrote it as h, we learned the arithmetic that reads it, the dot product, and we separated the two sets of numbers in play: the weights, fixed the day training ended, and the activations, the vectors built from your text for this run and discarded when the answer ends.

That constraint is the design problem this module answers. If everything the answer needs has to be sitting in one vector at one position before a token can be chosen, where does each part of that vector come from? What carries a fact forward from a few dozen positions back, what pulls a fact out of the weights, and how do the two meet in the vector each position carries up the stack?

The answer has a shape, and it is a sum of three sources:

  h_initial   the baseline — the current token's own embedding
+ h_ctx       the context term — pulled in from earlier positions
+ h_prior     the memory term — injected by the MLP from the weights
─────────────────────────────────────────────────────────────────────────────
= h           the thought state, at the last position

We keep both specimen questions. In "What is the capital of France?", the answer lives in the weights, so the memory term does the work. In "How many days do I have to return the C-40?", the answer lives nowhere in the weights, so the context term has to carry it, and when the policy is missing the memory term fills the slot with something else. One equation, two questions, and by the end you will be able to say, for any fact an answer needs, which of the three terms was supposed to supply it.

One statement about this module’s numbers, pictures and simplifications, made once. The attention scores and shares are shaped to show the mechanism and were measured from no run; a real head spreads its scores far wider than the figures do. The running-sum table and the three-vote table use the three toy slots from the last module and invented numbers. The figures draw one head and one clean path where a real model spreads the work over many heads in many blocks, most of them reading weakly from many places at once. And the three sources are a bookkeeping identity, grouped by which machine wrote each contribution. That grouping tracks which machine wrote each number into the residual stream, and it cannot sort context-derived knowledge from training-derived knowledge: attention computes its queries, keys and values with learned weights, and the MLP in block 40 reads a stream that already holds what attention wrote in block 39, so the groups cannot be separated cleanly after the fact. Addition also lets a later contribution cancel part of an earlier one, so the sum does not keep every contribution separately readable. The three-vote table therefore shows how a logit split this way could produce a wrong answer, in a toy whose terms are labelled by construction; what caused a real model’s wrong answer is a question the identity cannot settle.

Our worked examples are raw completions, where the last token of your prompt is the last position. In a chat product, the chat template wraps your message in control tokens and appends the start of an assistant message after it (<|im_start|>assistant in one common format), so the position that writes the first token of the reply holds a template token. We also set aside five standard details of the architecture. The original transformer adds a position encoding to each input embedding, so the vector entering block 1 differs by position, and many current models apply position to the queries and keys instead; the toy ignores position. The standard computation divides each query-key dot product by √d_k, the square root of the key length, before softmax; read the scores below as already divided. A position may read from itself as well as from earlier positions. The heads’ outputs pass through one more learned grid, the output projection W^O, before they are added to the residual stream. And most models normalize h before the unembedding, and again inside every block: the normalization subtracts the mean of h’s entries, divides by one number computed from all of h (RMS norm skips the subtraction), and multiplies slot by slot by learned gains. The three dot products still sum to the logit after that, each divided by the same shared number, so each term’s share of the score depends on the other two through that divisor. What is exact is the structure: three writers, one sum, one score per candidate.

every block adds to a running sum

Start with a problem the stack has to solve. In the customer’s question, block 2 works out that “40” is part of a product name rather than a quantity. The block composing the answer is seventy blocks higher, and it needs that finding intact. If each block replaced the vector it received with a new one, a finding made near the bottom of the stack would be gone long before the top. What design lets a finding made in block 2 still be read in block 80?

The design is that a block adds to the vector it receives and never replaces it.

Follow one token position to observe the arithmetic. In the policy sentence “Aperture accepts returns of the C-40 within 45 days of delivery,” consider the position holding the final word, “delivery.” It arrives at the bottom of the stack holding its starting vector, straight from the lookup table: the generic word, carrying nothing yet about this company or this camera. That starting vector is the baseline, h_initial.

Block 1 reads that vector and computes a vector of its own, the same length, and adds it in slot by slot. What leaves block 1 is the sum of the two. Block 2 receives that sum, reads it, computes its own contribution, and adds that in. Block 3 does the same to the result. Forty to a hundred and twenty times, all the way up.

Watch it happen on our three toy slots: formal register, obligation against permission, and time against object:

After	What that block added	The running sum
the lookup table (h_initial)	—	+0.1, −0.2, +0.1
block 1	+0.1, −0.3, +0.2	+0.2, −0.5, +0.3
block 2	+0.2, −0.6, +0.4	+0.4, −1.1, +0.7
block 3	−0.1, −0.1, +0.1	+0.3, −1.2, +0.8

Three things in that table are worth naming.

First, the position arrived generic and left committed. Its final row corresponds to vector a: “within 45 days of delivery,” which scored 1.83 against the customer’s query. The token “delivery” did not carry that specific commercial policy meaning when it emerged from the lookup table. It acquired it on the way up, from blocks that read the words around it.

Second, the first value added by block 3 is negative. That block pulled slot 1 back down, countering part of what the lower blocks contributed. So a contribution pointing the opposite way cancels part of an earlier one. That is the antiparallel case, two vectors pointing opposite ways with a negative product between them, doing an ordinary day’s work. Block 3 overwrote nothing, and yet slot 1 of the sum no longer shows everything the lower blocks wrote there: the sum keeps the total, and the separate contributions cannot be read back out of it.

Third, every block reads the accumulated total, never the previous block’s contribution on its own. Block 3 receives +0.4, −1.1, +0.7: the baseline numbers combined with every layer’s contribution up to that point. This is the part that answers the opening question: information written low in the stack stays in the sum that blocks high in the stack read, unless a later block writes against it. When block 2 works out that “40” belongs to a product name, it has put that finding somewhere every later block can read it, seventy blocks later, while the answer is being composed.

This accumulating vector is the residual stream. The name comes from the arithmetic you just watched: each machine computes a small correction, a residual, and adds it to what arrived, so what flows up the stack is the total of every correction written so far. Think of it as an additive highway, one lane per token position. No machine along it overwrites the lane; each one computes a vector and adds it in. Early blocks merge in the simple things: this token is a number, this one is part of a proper name. Middle blocks merge in meaning and relationships. Late blocks merge in something close to a prediction. Everything merged stays in the sum that every block above reads, though a later addition pointing the opposite way can cancel part of an earlier one, as block 3 did to slot 1.

Now look at what each block adds, because that is where the three sources come from. Every block holds two machines, and each writes one kind of contribution into the residual stream. The first machine, attention, reads the earlier positions and this one and adds what it found there: that is the context term, h_ctx, and it accumulates block by block. The second machine, the MLP, reads only this position, matches it against patterns stored in the weights, and adds what was filed under the matches: that is the memory term, h_prior, and it accumulates too. Sum the baseline, h_initial, with everything both machines wrote across every block, and at the last position, at the top of the stack, you have the thought state, h.

That is the equation at the top of the module, and it is a bookkeeping identity rather than a theory. Every number in h came from the embedding table, from an attention machine, or from an MLP, because those are the only three things that ever write into the residual stream. Group the contributions by which of the three wrote them and the equation falls out. We take the three groups in order.

the baseline: what the token itself brings

The smallest of the three sources is the one the position starts with, and it is worth a moment because it does more than most people expect.

h_initial is the vector retrieved from the input embedding table when the token enters the network. It is the same for that token wherever it appears and whatever surrounds it, before any block has written anything: the token’s own baseline statistics, learned in training, of what kind of thing it is and what tends to follow it. For a question mark, h_initial carries the baseline traits of a punctuation boundary at the end of a question — the statistical expectation that a response follows. For a period, the expectation is a continuation. For a #, a heading or a comment or a block of code.

Here is why that matters in practice. The last token of your prompt is the position that will write the first token of the answer, and its baseline is where that position starts before attention and memory add anything. Swap the final punctuation and you shift the starting coordinate.

Baseline shift. Replacing a prompt’s final punctuation mark shifts h_initial at the position that writes the answer. A question mark sets a baseline expectation of an answer; a period or a # sets an expectation of a continuation or a syntax block. If the surrounding prompt is clear, the context term readily overpowers a stray character. If the prompt is ambiguous, that baseline shift can tip the model toward a different first token.

You can test the rule on the C-40 question in a minute. Ask “How many days do I have to return the C-40?” and the first token is almost always the start of an answer. Ask “How many days do I have to return the C-40” with nothing at the end and the model will sometimes complete your sentence instead — “…if I bought it on sale?” — because the last position’s baseline is a bare word mid-sentence, and the context did not say the question was over. It is a small effect and a real one, and it explains a class of oddities that otherwise look random: the prompt that ends in a colon and gets a list, the one that ends in an open bracket and gets code.

Two consequences follow, and both come back later. The baseline is small compared with the other two terms in any well-formed prompt, which is why prompts are forgiving of typos. And it is never zero: even before the stack has done anything, the position writing the answer already leans somewhere.

the attention filter builds the context term

Now the term that reads your text.

Consider "What is the capital of France?" Seven tokens, seven positions, and the answer will be written at the position after the question mark. Something at that position has to gather what the sentence is about, and it cannot do it by averaging: combine all seven earlier vectors into a flat mean and you get a vector that is a little bit “what,” a little bit “the,” a little bit “France,” and useless. The mechanism that builds h_ctx has to be selective. It works as a relevance filter, and it has four steps.

Prompt tokens:   [What]   [is]   [the]   [capital]   [of]   [France]   [?]
                                                                        ▲
                                                                  current slot

Step one, the query. In each block, the vector at the current slot is multiplied by a stored grid of weights and comes out as a short vector, a few dozen to a couple of hundred numbers long, called the query. Read in words, this position’s query says something like “I need a geographic entity associated with the nation identified in this sentence.” Hold that translation lightly. A query is a vector, and it means something only when compared against another one; the comparison is the dot product.

Step two, the keys. Every position in the sequence multiplies its own vector by a second stored grid and comes out as a key, the same short length, advertising what that position holds:

• "the" → grammar token, a determiner
• "capital" → administrative center, a geographic attribute
• "France" → sovereign nation, a European country

Step three, the relevance check. The model takes the dot product of the query against every key. One number comes back per position, saying how strongly what this slot is after lines up with what that position holds. In this toy pass, “What”, “is”, “of” and the ”?” itself, which this slot may also read, each score about the same as “the”:

• Q · K("the") ≈ 0.05 — negligible
• Q · K("capital") ≈ 3.81 — high
• Q · K("France") ≈ 4.00 — high

Those raw scores are then put through softmax, the same step that turns logits into probabilities at the top of the stack. It converts the pile of scores into fractions that add to one:

• "France" → 52%
• "capital" → 43%
• everything else combined → 5%

Step four, the value sum. Each position multiplies its vector by a third stored grid and comes out as a value: the content it will hand over if asked. The model scales each value by the fraction its key earned and adds them all together:

h_ctx = (0.52 × V("France")) + (0.43 × V("capital")) + (0.05 × V(others))

That vector is added into the residual stream at the current slot. It carries the combined concept of France and capital city, concentrated at the one position that has to write the answer.

The three short vectors are called the query, the key, and the value, and the names come from database lookup: you match a query against keys, and what you get back is the value filed under the key that matched. The difference here is that you never get exactly one value. You always get a blend of all of them, weighted by how well each key scored, and a confident match means one value dominates the mixture. The machine that does all this is an attention head, and every block runs dozens of them at once, each with its own three grids of weights, and so each with its own notion of what to match on.

Two constraints shape every set of scores. A position can read from itself and from positions that came earlier in the sequence: the model is predicting what comes next, so a position is never allowed to look at text further along than itself. And the shares always add to one, so attention redistributes what is already there rather than inventing anything. If nothing in the sequence matches, the query still gets its shares — spread thinly across positions that hold nothing useful.

Watch External visual explanation

Attention in transformers, step-by-step (26 min), chapter 6 of Grant Sanderson’s Deep Learning series, animates the four steps above with the matrices drawn out: a query and key space of 128 numbers, the mask that blanks later positions before softmax runs, and 96 heads per block in GPT-3 where we say dozens. The text version walks the same example.

applying the attention filter to the question

That is the mechanism. Now run it on the question where it has to carry the whole answer.

The prompt holds the policy sentence “Aperture accepts returns of the C-40 within 45 days of delivery,” and the model is composing its answer at the last position in the sequence. The number it needs sits more than a dozen positions back, behind every token of the question. Nothing has carried it forward on its own: the residual stream at the last position holds only what the blocks below wrote there, and 45 was written into a different lane entirely.

So the composing position forms a query that amounts to a number of days, attached to this product. It runs the dot product against the key at every earlier position. Keys on grammar words score near zero. The key on the document’s product name scores high, and so does the key on the number sitting beside it. Softmax hands most of the shares to that region of the document, and the value carried by the position holding “45” is blended into the residual stream where the answer is being written. The number 45 has crossed from one lane to another. h_ctx at the composing position now carries it.

Now notice which part is stored and which part is live, because this is the two sets of numbers showing up inside a single machine. The three grids that turn a vector into a query, a key, and a value are weights, identical for every user and every request. The scores they produce are computed fresh from your text, every single time. The weights hold a rule for what to match on; your context decides what that rule lands on.

That is why a network fixed at training can answer about a product invented this morning. A head that learned “read from the number nearest this product name” will do that for the C-40 exactly as readily as for anything it saw in training, because the rule never mentions the C-40 at all. And it is why deleting the policy changes everything: the same query, run against a sequence that holds no number, earns its shares from positions that carry nothing about days, and h_ctx arrives at the composing position without one. The slot is still there. What fills it comes from the third source.