The Architecture of Prediction

Traditional computation rests on binary logic: discrete states of true or false that provide exact, repeatable precision for ledgers and physical simulations. Human language operates differently. Language evolved to communicate functional approximations across an ambiguous world, trading mathematical precision for flexible nuance. For decades, software required humans to translate fluid intent into rigid syntax, where a single misplaced character halts execution. A large language model bridges that divide by making language itself directly computable, reading and generating fluid prose through numerical calculation.

A traditional computational function is deterministic: identical inputs yield identical outputs on every run. A language model computes probabilities. Given an input prompt, it scores its entire vocabulary to determine what text is most likely to follow, generating its answer one token at a time.

Token: the unit a model reads and writes. The pieces are fixed in advance, so a common word is usually one token and a long or unusual word is several: “the” is one token, and “unbelievable” arrives as three, un · belie · vable. Every limit you will meet, the context window included, is counted in tokens.

A large language model does one thing: it predicts the next token. Everything an agent appears to be, a code reviewer, a clinical assistant, a research strategist, is built on top of that single calculation, run over and over. We come back to this rule in every module of the course.

That drawing of new text, token by token, is what the “generative” in generative AI names. The model composes sentences that appear nowhere in its training data, and the result is a conversational layer between human reasoning and the digital systems underneath.

To predict each token, the model relies on the weights, fixed the day training ended across trillions of words of human writing. A dataset of that scale forces the network to absorb structural patterns in how the world is described. Yet that same training rewards the likely continuation, and a continuation built on a pattern the system observed a billion times can read exactly as fluently as one built on a pattern it barely encountered at all.

When you examine those two facts together, the system’s operational boundaries become clear: next-token predictions are statistically coherent, but nothing in a fluent continuation certifies where prediction shades into invention. That reality explains both the utility of modern language models and their most severe errors. Once you understand this computational mechanism, failures stop being surprises and resolve into a defined set of failure modes you can design around: three ways the guess goes wrong, and two loops that catch it from outside.

One statement about this module’s pictures and numbers, made once. The split of “unbelievable” into un · belie · vable reflects one model family’s cut, and other families cut the same word differently. The context-window figure draws the window as a row of slots, whereas real systems fill and trim it in their own ways. The failure figure draws three separate branches, whereas one live answer can carry two at once. The two-loops figure draws a straight line, whereas the review agent’s rejection branch runs the generator again and human operators sample the approved branch. On numbers, the 50,000-to-200,000-row vocabulary, the 128,000-token window, and the 13,002-weight recognizer are real figures; a hundred tokens to seventy-five English words, three hundred passes for a two-hundred-word answer, and 96,000 words or a 350-page book for the window are rules of thumb derived from that ratio.

generation is a loop of scoring and drawing tokens

Go to your trusted AI chat tool. Type a question and press send. What happens under the hood to answer your question?

What reaches the model is plain text. The system cuts that text into tokens using a fixed lookup table called the tokenizer: sometimes a whole word, a fragment of a word, or a single mark of punctuation.

Those tokens feed into the network, a stack of stored numbers called weights, which were established during training and remain identical for every user. Here the central design problem appears: the network outputs one score per candidate token for the immediate next position, and nothing else. It cannot produce a sentence or an entire paragraph in a single step. An answer requires many tokens of unpredictable length. To produce an extended response, the system must execute repeatedly.

The architecture solves this problem by running an iterative loop. Consider a miniature vocabulary of five words. When the prompt arrives, the network reads it once, in a first pass that evaluates all prompt positions together. The network assigns a score to each of the five candidate words based on the text so far. The system converts those five scores into probabilities across the distribution. Next, the runtime selects one word through sampling: drawing a token based on those probabilities, favoring likely continuations while permitting variation. The system appends that chosen word to the text and runs another pass for the one new position, reusing the work it stored for the earlier tokens instead of calculating it again from scratch, and repeats the cycle to predict the word that follows.

A production model runs this exact mechanism at scale. Instead of five words, the vocabulary spans 50,000 to 200,000 candidate tokens. Generating a two-hundred-word answer requires roughly three hundred of these passes. Each pass reads the entire accumulated sequence, drawing on the stored work for earlier positions, to score every candidate in the vocabulary and draw the next token. That stored work holds nothing the text does not already determine, since running the same text again would produce it again; it makes each pass faster, and it lasts only as long as the request.

The vocabulary holds a dedicated end-of-sequence token. The loop executes repeatedly until the model samples that token, or until an external length limit halts generation.

Two operational realities govern the system when generation stops. First, the model evaluates only what is present in the text. In the plain generation this module describes, there is no side channel and no private scratchpad; the entire sequence of text the network can evaluate at one time, known as the context window, has a fixed maximum size. When a reasoning model works through intermediate steps before its final answer, it writes those steps into that same window as additional tokens, often hidden from the user by the chat product; module 3.08 examines them. Second, the model writes nothing down into its parameters. The weights that produced your answer remain unchanged after inference concludes; the network does not record user facts or modify itself during execution. In a multi-turn conversation, the application appends previous exchanges directly into the input text, allowing the next pass to read prior turns as context. That textual persistence enables follow-up questions to work without altering the model’s weights.

output is governed by weights, context, and tokens

the weights store patterns without certainty

In the generation loop, a score for every candidate token comes out of the network. When the context ends with “the capital of France is”, the model scores “Paris” near the top of the vocabulary. What inside the network makes that decision, and who set the values that produce it?

The machine responsible is a neural network, a mathematical function that transforms inputs into outputs through layers of calculations. At its smallest, imagine a network with a few inputs, a single output, and a multiplier on each connection that sets how strongly that signal carries forward. That multiplier is a weight. If the network produces the wrong output, you adjust those few multipliers until the calculation improves. A frontier language model is that same structure scaled up: dozens of layers and tens or hundreds of billions of weights. Together they are the entire contents of the model: its grasp of syntax, its reasoning habits, its turns of phrase, and its statistical memory. There is no separate database, no hidden rule engine, and no lookup table of facts inside the file. The weights are all there is.

No engineer chooses these numbers by hand. They come from training, a mechanical process of trial, error, and adjustment:

1. Feed the network real text with the next token hidden.
2. Let it guess which token should follow, using the weights it currently holds.
3. Measure the error between that guess and the text.
4. Nudge every weight a microscopic amount in the direction that would have made the right guess more likely.
5. Repeat across trillions of examples.

The structure that results is simply whatever reduced the error across those training examples. Nobody wrote the structure that results and nobody can read it back, and a fact rehearsed a thousand times and one seen twice come out of the same arithmetic at the same speed.

The training material spans far more than conversational prose: frontier models train across source code, digitized books, transcripts, images, and structured tables. Even so, the network stores no lookup table and no database of facts, and beyond the starting algorithm and the finished result, nobody can say what any given connection came to hold.

We can trace what that nudging does to a fact during training. A fact that appeared thousands of times in the training text was nudged into the weights thousands of times, leaving a heavily reinforced pattern. A fact that appeared twice was nudged twice, leaving a weak impression or none at all.

Nothing in the procedure ever recorded which case a given fact is in. The guess was scored, the numbers moved, and no ledger of “well rehearsed” against “barely seen” was kept anywhere. At runtime a well-rehearsed fact and a barely-seen one come out of the same arithmetic, at the same speed, and nothing in the continuation they produce certifies which case it came from. That is the last line of the derivation, and all three of this module’s failure classes come from it.

the context window is a fixed token budget

Two sets of numbers were in play just now, and the split between them is the second thing to hold. The weights are one set, and we have just seen that they were fixed when training ended. Everything else in the pass, your question, your documents, the answer so far, is the other set, assembled from your text for the request, extended by one token on each pass and, as the press-send loop showed, thrown away when the conversation ends.

Weights vs. context: the weights are stored, shared across all users, and fixed at inference time. Everything else, your prompt, your retrieved data, and the tokens generated so far, is ephemeral text held in active memory during the session and discarded when it ends. When evaluating what a model “knows,” always check whether that knowledge is assumed to be in the weights or provided directly in the active context.

Here is a two-minute experiment that makes the split concrete. Tell a model something it cannot possibly have been trained on, such as the name of your dog. It will use the name happily for the rest of the conversation. Now open a fresh conversation and ask what your dog is called, with persistent memory out of the picture: switch off any memory feature in your chat settings, use a temporary or incognito chat, or send the question through a direct API call, where the request carries only the text you send. The weights learned nothing; the name existed only as tokens in the earlier conversation’s context, and that context is not part of the new request, so the model has nothing to answer from. If a chat tool with memory switched on does produce the name in a new conversation, an application feature saved your earlier text and placed it in the new prompt. Every memory feature you will ever meet is engineering built on top of that fact: a filing system around the model, never a change to the model itself.

Everything the model can consider on a given pass has to fit inside one budget, measured in tokens, called the context window. Your system instructions, every retrieved document, every previous message in the conversation, and the answer being generated all share that one budget.

Modern windows are large. A 128,000-token window holds roughly 96,000 words at a hundred tokens to seventy-five words, which is a 350-page book. That sounds like enough room to stop worrying, and in agent systems it very often is not: a long-running agent accumulates tool outputs, retrieved documents, and its own previous turns, and those grow much faster than a conversation does.

What happens when you exceed this budget depends on the environment. At the raw API level, exceeding the limit throws an error, which means that if you are building your own tool, you are the one who has to handle a context that has grown too large. Chat interfaces and agent frameworks handle the boundary for you, dynamically, and the moves they make are ones you will build yourself later in this course: dropping the oldest turns out of the window, summarizing a long stretch of conversation into a shorter one, moving what matters into structured files the agent can retrieve when it needs them, and fetching only the documents a turn needs rather than carrying all of them. The rolling-history case is the one that catches people.

Context overflow: when conversation history and reference documents exceed the context window, systems configured with rolling history silently drop the oldest tokens out of scope. The model produces no warning; it simply continues generating answers without access to the dropped instructions.

Picturing the failure helps wrap your mind around the importance of maintaining the context window. Suppose you set up an agent with careful instructions at the top of the conversation. Forty turns later, those instructions have scrolled past the edge of the window. The agent keeps answering, fluently and confidently, with no access to the rules it was given, and nothing anywhere reports a problem. The behavior change looks like the model “forgetting” or “drifting,” which is why people reach for prompt rewrites when the actual fix is a window budget.

tokens are the model’s fixed vocabulary units

Before any of that arithmetic can run, your text has to be converted into tokens, and the unit matters because a token belongs to a fixed vocabulary: a lookup table built once, before training, and never changed afterwards, holding somewhere between fifty thousand and two hundred thousand entries. Each entry has a number, and that number, its row in the table, is what the network actually receives; every weight downstream is arithmetic on values that began as one of those rows. The table is built by finding the chunks that appear most often across a huge sample of text, so common short words get a row of their own while longer or rarer words are assembled from several pieces, as the figure above showed.

You pay per token. Pricing, rate limits, and speed are all measured in tokens rather than words. As a rough conversion for English prose, a hundred tokens is about seventy-five words, so a thousand-word document costs you roughly thirteen hundred tokens. Keep that ratio in your head; it is the one you reach for whenever you size a prompt, and it is the ratio the context window above was measured in.

And the model cannot see inside a token. A token is the indivisible unit of the model, its atom: it has no direct access to the letters, because letters are below the resolution of what it reads. The well-known demonstration is asking a model how many times the letter “r” appears in “strawberry” and watching it answer confidently and wrongly. Many current models now get this right, because post-training taught them to spell the word out letter by letter first, which converts the problem into one they can see. Grade that honestly: the underlying limitation has not changed at all; it has been routed around. Any task that depends on counting characters, reversing a string, or reasoning about spelling is working against the grain of how the model reads, and the practical move is to give the model the letters as tokens, spelled out, or to hand the counting to code.