Choosing a layer for a fact

choosing a layer by cost, trust, and reversibility

	What changes	Takes effect	Cost	Reversible?	Facts reliable?
Pretraining	all weights	months	tokens × parameters, in hardware hours	no	fuzzy
Fine-tuning	some weights	hours to days	examples, base model size, adapter or full weights	yes, via versioned adapters	weakly held, hard to check or change
Prompting	input only	instantly	tokens per call	yes, trivially	not applicable
Grounding	input only	instantly	index + tokens per call	yes, edit the source	yes, if a sound source comes back and supports the answer
Tools	input and the world	per call	latency + the service’s price per call	depends on the action	yes, as far as the API is
Memory	input only	next turn	storage + tokens	yes, delete the entry	yes, if the store is kept clean

The last column grades every layer at its best, so read it with suspicion. Grounding gives you reliable facts only when three conditions hold: the documents in the store are right, retrieval returns the passage that answers the question, and the answer the model writes is supported by that passage. Run four broke the first two at once: a stale document sat in the store beside the current policy, retrieval returned both, one saying 30 days and the current one saying 45, and 3.07’s illustrative distribution split 0.52 against 0.41. Run five showed another hole, where text arriving from a retrieved page carried an instruction rather than a fact. Read that column as “yes, once you have checked what came back and what the answer claims from it” rather than as something the layer hands you free.

Read down the last two columns and the shape of a working architecture falls out of them. Reversibility lives in the context, and so does the ability to check a fact against the line it came from. The weights hold behavior, which is the thing they are good at and the thing a prompt struggles to make stick.

Behavior in the weights, facts in retrieval. This single division prevents the most expensive mistake in the field. Teams fine-tune on their company documents expecting a model that knows their business, and get a model that sounds like their documents while inventing facts — run one wearing a corporate voice.

Two more corrections worth carrying.

A real system usually runs several at once. Every system built on a pretrained model uses that layer and a prompt. The others are added when a problem calls for them, and a working system often carries several: perhaps a fine-tune for format, retrieval over documents, tools for live data, and memory for continuity. So the question on any working day is which layer this particular problem belongs in.

Work upward, not downward. Start with prompting. Add grounding when it is a facts problem. Add tools when it is a live-data or action problem. Reach for fine-tuning only once you have proved that prompting cannot hold the behavior. Reach for pretraining approximately never. Most teams that jump to fine-tuning first are solving a prompt problem expensively.

diagnose downward · stop at the first yes

is the shape, register, or length wrong?

prompting

worked examples and an explicit format

free, instant, and the first thing to try every single time

is a fact missing, stale, or uncitable?

grounding

retrieve the source text into the prompt

and demand a citation, so the claim can be checked against a line

does it need live data, or an action?

tools

a function the model requests and your code runs

the first layer that can change something outside the conversation

must it survive into the next session?

memory

write it down, retrieve it next time

grounding pointed at a store the system writes to itself

has prompting repeatedly failed to hold a behavior?

fine-tuning

behavior only — format, register, task shape, what to decline

not for facts you must change or check; they go stale with nothing to tell you

last

pretraining

a new language or modality, months of compute, and almost certainly not your problem

four architectural moves we argue against

Each of these comes up constantly, and each contradicts something in our preceding lessons.

Asking the model for a confidence score. The estimate goes through the same machinery, from the same prompt, subject to the same fallback to the prior. The token “0.95” is produced the way every other token is produced. It is weakly informative at best, and it is worst exactly where you need it most: on plausible-sounding unknowns. Use retrieval coverage, agreement across independent samples, or an outside check.

Adding more examples until it works. Returns diminish fast, and the order of the examples moves results more than most people expect, because each example conditions every token after it and its position in the window changes how strongly. Examples specify shape far more reliably than content, so if your problem is factual they will not fix it, and they will narrow what the answer is able to say. In run three we put three short worked examples above the question. None of them happened to mention a restocking fee, so the answer came back in their shape and dropped the fee, which had been sitting in the policy the whole time. Examples do not only show a model how to answer; they show it what an answer contains.

Fine-tuning so the model “knows our domain”. Sometimes right, often the wrong tool, for the reasons in the table above. A fine-tune can carry a fact, and changing that fact takes another training run, while checking it means questioning the model because no source sits behind the answer. Facts you will need to change or cite belong in the prompt, through retrieval. Behavior that should hold on every call belongs in the weights.

Showing the reasoning so users trust it. This trades on an intuition the measurements do not support, and it moves in the wrong direction: a fluent trace raises trust without raising reliability. If you show it, show it as work in progress.

the limits of this interpretive map

Following a single token through the stack takes nothing but arithmetic: a fixed table cuts text into chunks, each position carries a vector, every block reads that vector and adds to it, and the last position’s finished thought state is dotted against every candidate vector in the unembedding table into one logit per token. None of that is in dispute. So is the bookkeeping that groups the thought state into three sources by which machine wrote them; what is idealized is any claim to separate the three cleanly after the fact, since each machine reads what the others wrote below it. The six runs are 3.07’s thought experiments: the direction of each effect is documented in the literature cited below, and the numbers were chosen to show it, so none of them is a measurement. The rest deserves a clearer warning.

The procedures described here have mostly been located in small models on narrow tasks. Nobody has a full account of what a frontier model computes, and the circuit stories here are illustrative fragments of a system whose whole we cannot yet draw.

The block-by-block account in run one is cleaner than the truth. Low blocks doing tokens, middle blocks doing meaning, high blocks doing prediction has real evidence behind it and real exceptions; the work is distributed and redundant and does not respect that division neatly.

Why worked examples work as well as they do is still open. The copy-and-continue procedure and the task-configuring numbers explain a great deal and not everything, and there are competing accounts nobody has settled between.

What reasoning training actually changes is contested. Whether it teaches new capability or sharpens a model’s ability to reach capability it already had is an active argument, and the answer probably differs by domain.

Whether the unfaithful trace can be fixed is genuinely unresolved, and the obvious fix is dangerous: training a model to produce faithful-looking chains optimizes for chains that look faithful, which is worse than a chain you know to be incomplete.

And the prior-and-evidence framing that carried the six runs is a description of behavior, not something derived from the architecture. It matches the direction of the documented effects the runs were built on. Something better will replace it.

reference: terms, sources, and the six runs

the words this part introduced

Every term below was taught at the point its machinery appeared. This is a lookup table, not the lesson; the course-wide glossary carries these plus everything from Parts 1 and 2.

Term	What it names
tokenizer	The fixed lookup table that cuts text into chunks. Built once before training, never changed since.
position	One slot in the sequence of tokens, and the residual stream that lives there.
residual stream	The running vector at each position that every block reads and adds into. An additive highway: things are merged in, never removed.
thought state (h)	The residual stream at the position that writes the next token, at the top of the stack. The one vector the vocabulary is compared against.
the three sources (h_initial, h_ctx, h_prior)	The thought state grouped by who wrote it: the token’s own embedding (the baseline), what attention pulled in from earlier positions (the context term), and what the MLP injected from the weights (the memory term). h is their sum.
baseline shift	Changing the last token of a prompt moves h_initial at the answer position; small, real, and decisive only when the context is ambiguous.
fact collision	In the three-source reading, the memory term holding more than one association for the pattern in front of it, and the loudest winning over the one the question asked for: Sydney for Australia’s capital, the part standing in for the whole.
unembedding table	The stored table above the stack holding one candidate vector u_A per vocabulary token.
logits	The raw score per candidate, ℓ(A) = h · u_A: three dot products in one sum, before softmax makes them probabilities.
softmax	Turns any set of numbers into fractions adding to one. Used for the final distribution, and again for attention scores.
sampling	Drawing one token from that distribution. The only step in a pass left to chance.
temperature	The dial on the draw. Low exaggerates the gaps, high flattens them, and neither changes the ranking.
autoregressive	One token per pass, appended and run again. A hundred-token answer is a hundred passes.
dot product	Multiply two lists element by element, add the results. One number out, large when they agree, negative when they oppose.
vector	A list of numbers used that way.
cosine similarity	The same comparison with both lengths divided out, leaving direction alone: +1 aligned, 0 orthogonal, −1 antiparallel.
matrix	A grid of stored lists, compared against an input all at once.
weights	Every stored number in the model. Fixed when training ended.
activations	The vectors built from your text for one run and discarded when the answer ends: the residual stream at every position, and the thought state at the last.
attention head	The machine that reads from other positions. Every block runs dozens at once.
query, key, value	The three short vectors a head derives from a residual stream: queries and keys decide where to read, values carry what arrives.
causal mask	The rule that a position may only read from itself and positions earlier than itself.
MLP	The machine that works in place: match against stored patterns, drop the misses, add the mixture back. Two-thirds of all parameters.
neuron	One stored pattern with its paired contribution. “Fires” means it matched.
superposition	Packing more concepts than dimensions by letting them overlap.
the prior	The model’s defaults about what text looks like. What fills a slot when nothing matches.
in-context learning	Picking up a pattern from the prompt without any weight change. Nothing is kept.
chain of thought	Writing intermediate work as tokens before the answer.
scratchpad (thinking trace)	The written chain, kept in the context; the pass after it closes builds h_ctx from the whole of it, which is why a truncated one fails.
KV cache	Keys and values already computed, saved so they are not redone each step. Speed, not memory.

sources for the findings

Copy-and-continue heads and their link to learning from examples: Olsson and colleagues, 2022. Stored patterns in the MLP as a keyed memory: Geva and colleagues, 2021. Where facts are read out: Meng and colleagues, 2022. The twenty-six head name-resolution arrangement: Wang and colleagues, 2022. The Fourier algorithm for modular arithmetic: Nanda and colleagues, 2023. Board state in an Othello model, with the editing result: Li and colleagues, 2023, and Nanda’s follow-up. Superposition: Elhage and colleagues, 2022. Rhyme planning and the tracing method: Lindsey and colleagues, 2025. Task-configuring numbers extracted from a prompt: Hendel and colleagues, and Todd and colleagues, both 2023. Examples working with randomized labels: Min and colleagues, 2022. The depth result for written reasoning: Merrill and Sabharwal, 2024, and Li and colleagues at ICLR 2024. Unfaithful traces: Turpin and colleagues, 2023, and Anthropic’s 2025 replication on reasoning models. Confabulated self-report in people: Nisbett and Wilson, 1977, and Gazzaniga’s split-brain work. Describing a face making recognition worse: Schooler and Engstler-Schooler, 1990.

This reading stops before mid-2026. The architecture will age well. The interpretability findings will age fastest, so treat the specific procedures as a snapshot and check the dates.

the videos this part pairs with

Grant Sanderson’s Deep Learning series at 3Blue1Brown animates the same machinery with the numbers of one real model, GPT-3, and the first three modules of this part point to the chapter that matches a section wherever the two overlap. In order, they are what a neural network is (19 min), gradient descent (21 min), backpropagation (13 min) and its calculus (10 min) for how the weights were set; transformers (27 min) for the pass from embedding to softmax; attention (26 min) for the context term; how LLMs store facts (23 min) for the memory term; and large language models explained briefly (8 min) for the whole loop in one sitting. Every lesson also exists as text. Where his numbers and ours differ, his are GPT-3’s from 2020 (12,288 numbers per vector, 96 blocks, a 2,048-token window) and ours are rounded for a current model; the mechanism is the same in both.

build the six runs yourself

The fastest way to own this material is to reproduce it. Invent a product, write a three-line policy for it, and run the six prompts and the fact-collision check that follows them against any model you already have access to, with temperature at 0.7 so you can watch run four wobble, or at the model’s default where the provider recommends keeping it (Google asks for 1.0 on its Gemini 3 models). Nothing here needs a local model or a key you do not already have.

1. Bare question

   The bare question, no document. Count how many details it invents.

2. Question with policy

   The question with the policy above it.

3. Question with examples

   The question with three terse examples above it. Note what the format makes the answer unable to say.

4. Question with conflicting documents

   The question with the policy and a contradicting older document. Run it twenty times and count the answers.

5. Question with hidden instruction

   The question with an instruction hidden in one of the documents.

6. Refund calculation with reasoning

   The refund calculation, first with reasoning turned off, then on. If your model lets you cap reasoning tokens, cap it hard once and read what the incomplete scratchpad produces.

7. Capital questions without documents

   Two questions with no document at all: the capital of France, and the capital of a country whose largest city is not its capital. Ask the second on the smallest model you can reach and count how often the largest city answers.

Run four is the one worth doing properly. Twenty samples takes two minutes and turns a claim in a lesson into a number you measured, and that number is the argument you will need the next time somebody proposes to fix a retrieval conflict with a line in the system prompt.

a return policy belongs in retrieval, and its tests say why

Take the policy the six runs were built on: Aperture Instruments’ returns policy for the C-40, rev. 2026-02, with 45 days from delivery, a 15% restocking fee after day 30, and opened lens kits non-returnable. Before choosing a layer for it, you write down three things about the fact: how often it changes, what evidence an answer about it must carry, and which cases would show that the choice works.

The update frequency settles most of it. Picture a return window that changed last quarter, with the current policy on a wiki page nobody has pasted into anything. A fact that changed within a quarter will change again. Each change held in the weights means another training run and another round of questioning the model to see whether the new number took; held in a document store, it means editing one file, and the next answer reads the new text. The policy belongs in retrieval, with the policy document as its source.

The evidence requirement sets what your application checks. Every answer about a return cites the clause it rests on and the revision it came from, and your code confirms that the cited clause appears in the retrieved text before the answer ships. An answer whose claim has no matching clause does not ship: the customer is told the policy does not cover the question, and the case goes to a person. Write that outcome down before you build, because the check will sometimes fail and sometimes have nothing to run against.

The evaluation cases come from the runs. A question with only the current policy retrieved should come back with 45 days and the restocking fee. The same question with the stale 30-day document retrieved beside it, as in run four, should cite rev. 2026-02 or flag the conflict. A retrieved page carrying an instruction, as in run five, should leave the answer unchanged. A question about a product the policy does not cover should produce the declared fallback. The refund from run six, two of three C-40s returned on day 44, should come to $4,165 with the fee taken off. Write the five before the pipeline exists, run them after every change to the index or the prompt, and record what your model does with each. If the replies then keep drifting from the house format after prompting has been tried, that behavior is the candidate for a fine-tune, and the policy stays in the store.

Then do the same for one fact your own system depends on. Name how often it changes, the evidence an answer about it must carry, and four or five cases that would catch it going wrong, and let those three choose its layer.

This is the last room of the stoa. Everything you built in Parts 1 and 2 was standing on the mechanism you have now read down to the vector. When an agent gives you a wrong answer, ask where the right one should have come from: the prompt, a retrieved document, a tool, or the weights. That is a design question you can act on, because it names the layer to fix. Which internal contribution produced the error is a separate question, and one output cannot answer it. The syllabus holds the whole course; wander back through it whenever a specimen needs a second look.