Diagnosing prompt failures by variable

When an agent produces an output that is incorrect, incomplete, or malformed, the root cause almost always resides in the instruction you provided. That is natural, because the instruction is the primary portion of the context under your direct control. The underlying model is rarely the defect; genuine exceptions (tasks falling below a model’s capability floor, as introduced in our foundation on model capabilities) make themselves known by failing in the exact same manner regardless of how you phrase the instruction.

Test an instruction before you trust it, because a prompt written in natural language fails in ways you cannot predict by reading it. The model samples its reply token by token, it can read your wording in a way you did not intend, and a rule that holds on one input can give way on the next. The Tutor’s first draft in our lesson on agent design read as a complete teaching procedure and stated the key idea on its very first draw. A set of runs is the only evidence of how a prompt behaves: you test it, read what comes back, and refine it from what the runs show. This module turns that practice into a method.

Revising an instruction is inexpensive, yet that flexibility presents a subtle trap. Advising someone to simply “change the prompt” encourages random rewrites until an output arbitrarily passes, leaving the author with no clear understanding of which adjustment resolved the issue.

We approach instruction refinement as a diagnostic science, relying on one consistent loop throughout. You run the agent, isolate precisely where the output broke down, and modify exactly one of five variables: clarity, precision, context, instruction, or format. Then you evaluate the result again. Once you master this manual discipline, you can delegate the task to a second agent: a dedicated critic that quantifies output quality into a numeric score your workflow can gate upon automatically. By the end of this lesson, you will be able to pinpoint a failing output’s core variable in a single sentence, and you will construct a two-agent system that revises a draft until it clears the quality bar of 85 or reaches a three-round cap, and that returns the critic’s verdict with its score in either case.

One statement about this module’s numbers and pictures, made once. The DebugPanel branches and the flawed and fixed pairs in the five-variable section are real model outputs, measured from runs. The Tutor trace is a reconstructed session written in the Tutor’s register, and the critic-loop trace with its scores of 62 and 91 is built to match the critic loop’s configuration, measured from no run. The one recorded run is the 2026-07-11 trace, which scored 98. The two flow figures draw one failure per output and omit the orchestrator model that compares the score against 85. The five-column figure draws the variables as separate pillars where a real fault, such as a vague verb, often lands in two columns at once; the loop takes them one at a time, loudest first.

The Riverside Community Fair helper, a fictional agent for a one-day fair, answers volunteers’ questions from the fair’s schedule, which is pasted into its instruction. The schedule includes these lines:

Bake sale: Saturday 9 a.m. to 1 p.m., stall 14.
Face painting: Saturday 10 a.m. to 3 p.m., stall 6.
Volunteer check-in: the green tent by the main gate, from 8 a.m.

A volunteer asks: “What time does the bake sale open on Saturday?” The helper replies: “The bake sale is one of the fair’s favourite stalls, with homemade cakes and pies from local bakers, and it is a lovely place to start your morning.” What do you do first?

the debugging loop isolates one variable at a time

Let us establish the foundational method before analyzing its components.

When you run an agent against a realistic input and receive an unsatisfactory result, your first move must be deliberate. State precisely what is defective about the output, in a single sentence, before modifying a single word of your prompt. That diagnostic sentence determines which variable requires attention. Saying “the model answered a completely different question than the one asked” points to an entirely different problem than saying “the model answered correctly, but formatted the response in a way our parser cannot read.”

Next, change exactly one variable. Why enforce such strict isolation? If you alter three elements of a prompt simultaneously and the output improves, you gain no transferable knowledge. You cannot determine which adjustment solved the failure, you cannot apply the insight to your next agent, and you cannot safely prune the two superfluous edits that contributed nothing.

After making that singular change, run the prompt again on the failing case alongside the challenging edge cases that caused concern initially. This process leads directly to our governing rule:

One variable at a time. Say in one sentence what is wrong with the output; that sentence names the variable. Change that one variable and nothing else. Run again on the input that failed. A change that did not help goes back before the next one goes in.

Running the model on your own hardware removes provider token fees from this iterative loop. Evaluating a prompt against local model weights costs local compute time and electricity, with no per-call API charge, allowing you to test modifications across thirty test cases just as easily as testing the single prompt that originally failed.

the loop · one variable per pass

run

the agent, on a real input

and on the awkward ones you already worry about

name it

say in one sentence what is wrong with the output

this sentence is what picks the variable; do not skip it

adjust

change clarity, precision, context, instruction, or format

fixed?

yes

keep the failing input as a test case

a bug you fixed and did not record will come back

no

put the change back and pick a different variable

stacked half-fixes are how prompts become unmaintainable

For the bake-sale reply, the diagnostic sentence is: “It talked about the bake sale and never gave the opening time the schedule states.” The sentence works because it describes the output, not the prompt, and it names no fix yet.

five prompt variables produce characteristic failures

Every failure mode exhibits a characteristic symptom, and learning to recognize that symptom guides your diagnosis. The short examples below show each variable in isolation, and the API documentation case later in this section shows all five operating on a realistic prompt.

clarity: ambiguity forces the model to guess the task

When a prompt contains ambiguous language, undefined domain jargon, or sentences open to multiple interpretations, the model must select an interpretation on its own. It naturally selects the most statistically probable path, which frequently differs from your intended goal.

• Flawed: “Process the customer message regarding their recent issue.”
• Fixed: “Extract the customer’s account number and primary billing complaint from the incoming message.”

The directive “process” might imply summarizing, classifying, drafting a reply, or escalating to human support. The corrected version explicitly defines the operation and names the two specific fields to extract.

precision: loose constraints produce unusable output shapes

In this scenario, the model understands the core objective, but the resulting output remains unusable because your operational constraints were phrased too loosely.

• Flawed: “Provide a brief summary of the incident.”
• Fixed: “Summarize the incident report in exactly three bullet points. Keep the total length under 100 words.”

The word “brief” carries a subjective definition determined by training distributions rather than your requirements. Constraining the output to three bullet points and under 100 words establishes concrete boundaries.

context: missing data induces confabulation

Here, the model produces an articulate response to an inquiry for which it holds no grounding data. This is confabulation entering production: when asked for specific domain facts it does not have, a model will not reliably halt or decline. Instead, it generates tokens that mirror the surface syntax of a plausible answer with complete fluency and zero hesitation.

• Flawed: “Determine if this customer is eligible for a refund.”
• Fixed: “Determine if this customer is eligible for a refund by comparing their purchase date against the 30-day policy rules provided below: [Refund Policy Document].”

The first remedy is placing the necessary reference material directly into the context window, with an instruction saying what to do when a fact is missing and a check of the output against that material. Run one inquiry across several conditions and the split is consistent: worked examples define the structure and scope of a reply, but the factual grounding arrives only when the source material is explicitly supplied in the window.

instruction: observational phrasing elicits commentary

Phrasing directives in a passive or observational tone encourages the model to generate descriptive commentary. Using direct action verbs compels the model to execute the operational task.

• Flawed: “Look over this technical support transcript.”
• Fixed: “Categorize this support transcript into exactly one category: Billing, Software Bug, or Feature Request.”

Verbs such as categorize, extract, calculate, rank, and rewrite assign unambiguous tasks. Verbs such as look over, consider, and examine invite open-ended contemplation.

format: a missing schema breaks downstream parsing

The semantic content generated by the agent is correct, but downstream software components cannot ingest or parse the output structure.

• Flawed: “Return the results as JSON.”
• Fixed: “Return a JSON object containing two key-value pairs: message (string) and confidence (integer between 0 and 100).”

A generic request for “JSON” leaves key naming conventions, nesting hierarchies, and data types entirely unconstrained, causing them to drift between invocations. Note the explicit contract in the corrected example: two designated keys, with one defined strictly as an integer between 0 and 100. The automated critic later in this lesson returns these two keys and a third, an issues list the drafter reads, and the stability of all three relies entirely on pinning down this format variable.

Take the bake-sale reply, which described the stall and never gave its opening time. The helper’s task line read “Look over the volunteer’s question and the schedule.” That puts the failure in the instruction column, because “look over” invites commentary where an action verb would set a task. It also touches the clarity column, because “look over” does not say what to produce. The commentary is the louder symptom, so the one change goes to the task line: “Answer the volunteer’s question with the time, place or stall number the schedule gives.”

adjusting the wrong variable hides the failure

If you tighten the format constraints on a prompt that actually suffers from missing context, the model will return a beautifully formatted, schema-compliant, confidently false response. In doing so, you have made the failure significantly harder for human reviewers to spot and far more likely to pass unchecked into downstream systems.

This outcome is a direct consequence of how language models operate. Formatting instructions govern only the structural shape of subsequent tokens. If the factual foundation was never loaded into the context window, no structural directive can summon it into existence. Authentic system traces show this directly. Given the prompt “Write documentation for our API. Keep it under 120 words.”, the model generated an articulate overview of completely fictional endpoints. Refining clarity by requesting clear, simple prose merely caused it to invent a different fictional API. Refining precision by demanding exactly three sections titled Auth, Endpoints, and Errors produced three impeccably organized sections of fabricated endpoints (the cleanest confabulation in the set). Only the context adjustment—supplying the actual API specification inside the prompt—corrected the factual contents. Test a single query across many conditions and the same behavior holds: output facts shift only when reference material is placed into the context window.

Diagnose the symptom thoroughly before writing an edit. That is the essence of isolating one variable at a time.

finding failures with adversarial personas

The testing loop so far only fixes failures you happened to trip over during development. The inputs a customer will send in production are precisely the ones you did not think of; what would have to exist to find those inputs before the customer does? You need a dedicated evaluation prompt engineered specifically to probe and challenge your agent’s instructions, directing the model to uncover failure modes with no obligation to be agreeable.

<system_identity>
You are a skeptical senior QA analyst who trusts nothing that has not failed
at least once in front of you. Your goal is to find flaws, false premises,
and edge-case failures in the target system prompt.
</system_identity>

<execution_framework>
Review the target prompt and enumerate ten realistic user inputs designed to
trigger confabulation, policy bypasses, missing-context traps, or format
errors. Rank them by probability and potential harm.
</execution_framework>

This prompt is an adversarial persona. Read that as: an identity that says what this agent is solely for, and a workflow block that says what to produce and in what order. Both come from the block anatomy, and the second is the workflow block, optional in general and present here because this job has steps. Nothing exotic. The value is entirely in pointing it at your own work.

Whenever the adversarial persona uncovers a vulnerability, treat that input as an asset. Retain the failing case immediately. This practice forms one of our most cost-effective development habits:

Every fixed failure becomes a test. When an adversarial run finds a hole, save the input that found it next to the corrected prompt. A bug you fixed and did not record is a bug that returns the next time somebody edits the instruction.