Foundational failures of LLMs

Type “The capital of France is” into a chat tool and it answers “Paris”, from what training stored in its weights. Ask the same tool for a court ruling on an obscure property dispute and it can answer just as quickly with a case name, a volume number, a judge and a quoted passage, for a case no court ever decided. That is what happened to a New York lawyer in Mata v. Avianca, whose record appears later in this lesson. Both answers came out of the same loop: score every token, draw one, append it, repeat. Nothing in that loop checks a claim against the world, so a true answer and an invented one arrive in the same fluent sentences. This lesson sorts the ways that goes wrong into three classes of failure, and then places the checks that catch them, all of which sit outside the model.

probabilistic generation produces three classes of failure

A system that predicts statistically probable text produces predictable failure modes.

All three failures detailed below originate from a single computational property: the model generates statistically likely continuations, and nothing in that generation certifies factual correctness. Recognizing that these errors share this exact root allows you to approach wrong answers as predictable architectural challenges rather than unrelated bugs.

Asked for a court precedent on an obscure property dispute, a model returns a citation with a volume number, a judge’s name, a jurisdiction and a quoted passage, for a case that never existed. A model tells a camper to boil water over an open flame in a thin plastic bag, without noting that the bag melts. And a hiring agent trained on ten years of a company’s resumes assigns lower fit scores to qualified candidates whose work histories carry markers associated with underrepresented groups.

confabulation is sincere but fabricated generation

When an input requests information missing from its weights, a language model can abstain and say that it does not know. It can also continue the sequence by generating text that matches the expected structural shape of an answer.

The industry calls this behavior hallucination, but it is more precisely described as confabulation. In psychology, confabulation describes filling memory gaps with plausible fabrications while remaining entirely confident in the result.

In a language model, the mechanism is statistical: the network completes the prompt using high-probability linguistic patterns. It constructs convincing citations, dates, or explanations out of familiar training patterns, delivering fabricated details in the exact same authoritative tone as verified facts.

Ask a model for a court precedent on an obscure property dispute and you may get a citation complete with volume number, judge’s name, jurisdiction, and a quoted passage, for a case that never existed. Every surface feature of authority is correct because the model has read tens of thousands of real citations and knows exactly what one looks like. The only thing missing is the case.

Why is that dangerous rather than merely annoying? Because the output text carries no built-in certificate of truth. A fabricated citation and a real one are produced by the same process, at the same speed, with the same fluency and the same confidence. Inside the model, researchers have found something more specific. In an interpretability study of Claude 3.5 Haiku, Anthropic’s “Tracing the thoughts of a large language model” identified a circuit that is on by default and leads the model to state that it has insufficient information, and a competing “known entities” feature that inhibits it; a hallucination occurred when that familiarity feature activated without the knowledge behind it and suppressed the default. That finding shows a model can carry an internal signal related to what it knows. It does not show that a model’s expressed confidence is calibrated to factual correctness, and it was established for one model with particular methods. Nothing on the surface of the output certifies whether a claim is true, which means every check has to come from outside the model. That gap between fluency and truth is the point people misjudge most often.

Fluency is not accuracy: fluent phrasing and a high token probability show that a continuation fits strong patterns in the training text, and neither one certifies that the claim is factually true. High fluency often masks complete invention.

the grounding gap comes from symbols learned without contact with the world

The second failure is about contact with the world. A model works entirely inside a closed system of symbols, learned from descriptions. It has read every account of water ever written, but it has never been wet.

So it will cheerfully tell a camper to boil water over an open flame in a thin plastic bag, without noting that the bag melts. It will write code calling a database function that was retired two years ago, because that function appears throughout the text it learned from and nothing told it the world moved on. Both errors are the same shape: a statement that is perfectly consistent with the descriptions the model absorbed, and inconsistent with present physical or system reality.

This one has a straightforward fix, which is the whole reason the rest of the course exists. Give the model contact with reality by putting reality in the text.

systemic bias reflects statistical skews in the training data

The third failure is inherited. The network mirrors the statistical skews of what it read, and historical prejudices and representation imbalances in the source documents carry straight into the weights.

For example, take an automated hiring agent trained on ten years of a company’s own resume data. It learns what that company’s past decisions looked like. If those decisions systematically favored certain backgrounds, the agent reproduces the pattern, assigning lower fit scores to qualified candidates whose work histories carry markers associated with underrepresented groups. The model is doing what it always does, which is to continue the pattern it was shown. That is exactly the problem.

Bias differs from the first two failures in one important way: prompting and grounding alone cannot reliably eliminate it. You can instruct a model to ignore demographic indicators, and the underlying statistical skews remain in the weights. It has to be addressed at the system design level, measuring outcomes in aggregate, auditing across groups, and constraining what you allow the agent to decide at all. Where the first two failures show in a single answer, this one shows in aggregates, which is why the check for it is a measurement across many outputs rather than a read of one.

your turn: mark an answer against its source and sort each mark by its fix

An assistant answers residents from this announcement:

“The Riverside repair café meets on the first Saturday of each month, 10 a.m. to 1 p.m., in the library’s community room. Volunteers fix small appliances, bicycles and clothing. Repairs are free; donations are welcome.”

A resident asks, “Can I get my bicycle fixed at the repair café?” The assistant answers:

“Yes. The repair café meets on the first Saturday of each month, 10 a.m. to 1 p.m., in the library’s community room, and volunteers fix bicycles there. Bicycle repairs are led by the café’s mechanic, Sam Okafor, who has run the table since 2019. Bus 12 stops outside the library, and its racks carry bicycles.”

The first sentence stays unmarked: the day, the hours, the room and the bicycle repairs are all in the announcement. The mechanic’s name and the year are not, and they have the shape of a detail an answer like this usually carries, so they are a confabulation; the fix is the source in the context and a demand for a citation. The bus route is not in the announcement either, and it describes the present state of the world, which the model knows only from descriptions it read. That is a grounding gap, caught by putting the current timetable in the context or giving the assistant a tool that looks it up.

grounding repairs the gap by putting facts in the context

Grounding is the practice of supplying verified reference material, database records, source files, policy documents, or live system state, directly into the context window before the model generates a response.

Grounding works because the model’s predictions are conditioned on everything in the active sequence. Ask about a company’s return policy without providing the text and the model generates the plausible shape of a policy, synthesized from thousands of generic examples in its training data. Paste the actual policy into the prompt and that specific text dominates the context window, steering the next-token probabilities toward the exact terms, dates, and numbers in the document.

Two engineering advantages follow. Facts live outside the model, so you update information by editing a document or querying a database rather than retraining a network. And grounded systems let you require citations, which converts an unverifiable claim into a statement tied to a source line you can inspect.

To see the failure mode grounding prevents, look at the public legal record. In 2023 a New York attorney used an ungrounded ChatGPT session to find supporting precedent for a federal brief. The model returned convincing citations with court names, volume numbers, and quoted rulings. Six of the cases did not exist. When the attorney asked the same session whether one of them, Varghese v. China Southern Airlines, was a real case, the model apologized for “the confusion earlier”, said it had double-checked, and replied that the case could be found on Westlaw and LexisNexis. The clerk of the Eleventh Circuit later found no such case, and the court fined the two attorneys and their firm five thousand dollars. A request to confirm is most often followed by a confirmation in the text the model learned from, and the model held no record of where its first citation came from.

This is why factual checks must sit outside the model, and grounding is the most direct way to enforce them. Placing verified reference documents in the prompt anchors generated text to the specific source in front of the model rather than to generic statistical patterns in its weights. Requiring exact citations gives you an audit trail you can verify immediately.

Understand what the technique actually does. It does not alter the underlying weights or make the model inherently truthful. It structures the context window so that extracting facts from the source, or stating plainly that the source lacks the answer, becomes the highest-probability continuation. Accuracy is an engineering constraint you build from the outside, never an internal property of the model. Writing prompt rules that steer the model toward this behavior (cite the source or say the source does not answer) is a craft in its own right. The final safeguard remains yours: before you rely on generated output, open the citation and check the primary source.

checking output requires a loop outside the model

Because these errors come from the mechanics of the machine rather than from a mistake anyone made, checking output is part of the architecture rather than a courtesy. Someone or something outside the model has to look, because the generated text does not certify itself, and the model’s internal signals are no calibrated guarantee of truth. Two loops do the looking, one built around your judgment and one built around volume.

a human checks the input, and the output at low volume

For individual decisions and low-volume systems, that someone is you, working at two distinct stages.

At the input stage, you control what enters the window: stripping credentials and sensitive data that should never be sent, and deliberately placing the grounding material the answer will need. Most output problems are input problems, and this is the cheapest place to solve them.

At the verification stage, you check what came out: auditing claims against sources, following citations to confirm they exist and say what the model said they say, and testing logic rather than accepting fluency as evidence of it.

The discipline sounds obvious and is genuinely hard to sustain, for a reason worth naming so you do not mistake it for carelessness. Fluent, confident text is persuasive, and reviewing it is boring. After forty correct outputs, the forty-first gets less scrutiny, precisely when a wrong one is most costly. Building the check into the system, a step that must be completed, a field that must be filled, beats relying on attention that will naturally fade.

an independent agent inspects at production volume

When a system runs thousands of operations a minute, no human reviews every turn. People can only inspect samples. So you place a second model in the path and give it one job: inspection.

A review agent sits between the primary generator and the user. It takes the draft, evaluates it against strict rules, a required output shape, or a safety policy, and then either passes it, halts it, or sends it back to the generator with specific correction instructions.

Why a second model, rather than asking the first one to check itself? Asking a generator to evaluate its own output simply prompts another continuation. Asking it to check its own work in the same pass tends to get you a fluent explanation of why the draft is fine, because that is the statistically likely continuation of “review your work.” The property that makes the review agent work is separation: its own instructions, its own narrow task, and no stake in the draft being good. A separate pass with a separate instruction produces an actual second opinion. Give that reviewer a scoring rubric and its verdict becomes a number. The number is still the reviewer’s judgment; what a program can then do deterministically is compare it against a threshold you set, which is what lets code rather than a person decide whether the draft passes.

The review agent closes one channel and leaves another open, and you should carry both. It catches drafts that break a rule you wrote down. It shares the weights’ blind spots with the generator, so a fact both models were trained to believe passes review, and a bias in the weights is not caught by a second copy of the same weights. That is why the human stays in the loop as an auditor of samples even when the agent does the volume.

what you can do with this today

You now understand the working picture of generative AI: text becomes tokens, the network guesses one token at a time from a fixed set of weights, everything it can use has to be inside the context window, and nothing from the conversation reaches the weights. Three failure classes follow because a likely continuation carries no certificate of its own truth, and every check on them sits outside the model, in one of the two loops.

By design, this module carries information that will immediately improve your results in generative AI tools, like chat interfaces and coding tools. By understanding the fundamentals of how large language models work and taking up your role as the human supervisor, you can steer these tools effectively and responsibly.

You can now diagnose wrong answers by asking three questions in order:

Was the material that determines the right answer in the window? Before rewriting a prompt, check whether it was actually present, and whether it was still inside the window when the answer was written. Most of the time it was not, and no amount of rephrasing will conjure it. If it was missing, that is a grounding problem or a context overflow, and the fix is to put the fact in and keep it in.

Did the answer sound sure? Then treat that apparent confidence as uncertified. Fluency and hedging are both generated by the same process as everything else. A model that sounds certain and a model that sounds cautious are telling you about the shape of the text, and neither tone settles the reliability of the claim. Follow the citation; run the code; check the figure against the source.

Where was the check, and could the draft have talked it out of the verdict? Put your check where it cannot be: a citation you follow, a schema that validates, a test that runs, a second agent with its own instruction, anything that does not depend on the same pass that produced the draft. And if the wrong answer was a pattern across many outputs rather than a single slip, that is bias, and the check is a measurement across groups rather than a read of one transcript.

Here is the whole pass on one card, in the vocabulary you now own:

1. TEXT IN
   Your message, plus everything already in the conversation.
   All of it has to fit the context window; under a rolling history,
   older tokens drop out of view without warning.

2. TOKENS
   A fixed lookup table cuts the text into chunks. About 100 tokens
   per 75 English words. The model reads these, never letters.

3. THE PASS
   The weights, fixed and shared, read the whole sequence and score
   every candidate token. Fluency and a high token probability do
   not certify a claim as fact.

4. ONE TOKEN OUT
   Scores become probabilities; one token is drawn; it is appended,
   and the pass runs again. A 200-word answer is ~300 passes.

5. THE THREE FAILURES
   Confabulation: a gap filled with the shape of an answer.
   Grounding gap: consistent with descriptions, wrong about the world.
   Systemic bias: the skew of the training text, continued.

6. THE TWO LOOPS
   You, at the input stage and the verification stage.
   A review agent, with its own instruction and no stake in the draft.
   Both sit outside the model, because fluent output does not
   certify itself.

Next we begin taking steps toward commanding your own army of agents. We will do this by downloading a language model onto your own device and instructing it to take on a personality; the module after that gives it a purpose. Everything the model is lives in a fixed file of numbers, and in the next module you place that file on your own machine and run it: no account, no meter, and no data leaving your desk. The syllabus points the way.