Testing & refining an agentic workflow

When an agent returns something wrong, incomplete, or badly shaped, the failure almost always lives in the instruction you gave it, because the instruction is the part of the context window you control. The model itself is rarely the problem, and the exceptions, tasks that sit below a model’s capability floor from module 1.02, announce themselves by failing the same way no matter how the instruction is written.

That is good news, because instructions are cheap to change. It is also a trap. “Change the instruction” is an invitation to rewrite prompts at random until something works and to have no idea afterwards which change mattered.

So we do it as a diagnosis instead, and we work one frame the whole way through. Run the agent and isolate exactly where it went wrong. Change one of five variables: clarity, precision, context, instruction, or format. Run again. Then, once you can do that by hand, hand the judgment to a second agent, a critic, that scores the output as a number a loop can gate on, so the diagnosis happens on every request without you. By the end you will be able to name a failing output’s variable in one sentence, and you will have run a two-agent loop that ships only what clears a threshold.

the debugging loop

Here is the method before the parts.

You run the agent on a real input and it produces something unacceptable. What is the first move? Say precisely what is wrong with the output, in one sentence, before you touch the prompt, because that sentence chooses the variable. “It answered a different question than I asked” points somewhere completely different from “it answered the right question in a format my code cannot parse.”

Then you change one thing. Only one. Why so strict? Because if you adjust three parts of a prompt at once and the output improves, you have learned nothing transferable: you cannot say which change did it, so you cannot apply the lesson to the next agent, and you cannot safely remove the two changes that did nothing.

Then you run it again on the same input, and on the awkward inputs that made you nervous the first time. The last line of that reasoning is the rule the rest of the module leans on:

One variable at a time. Say in one sentence what is wrong with the output; that sentence names the variable. Change that one variable and nothing else. Run again on the input that failed. A change that did not help goes back before the next one goes in.

Module 1.02 makes this loop nearly free. A run on local weights costs processor time and nothing else, so you can afford to run a change against thirty inputs rather than the one that happened to fail.

the five variables

Each variable has a symptom, and the symptom is what you read first. The examples below are short so the variable is visible; the exercise after them shows all five moved on one real prompt.

clarity — the model did the wrong task

Ambiguous words, unexplained jargon, and sentences with several possible readings all leave the model to pick an interpretation. It picks the most statistically likely one, which is not always yours.

• Flawed: “Process the customer message regarding their recent issue.”
• Fixed: “Extract the customer’s account number and primary billing complaint from the incoming message.”

“Process” could mean summarize, categorize, reply, or escalate. The fix names the operation and names the two things to pull out.

precision — the right task, the wrong shape or size

The task landed, and the result is unusable because you described the constraints loosely.

• Flawed: “Provide a brief summary of the incident.”
• Fixed: “Summarize the incident report in exactly three bullet points. Keep the total length under 100 words.”

“Brief” is a word whose meaning is set by the training data, not by you. Three bullets and a hundred words are yours.

context — a confident invention

The model produced a fluent answer to a question it had no information about. This is module 1.01’s confabulation arriving in your product: a model asked for something it does not hold will not stop and will not say so, and it produces the text that best matches the shape of a correct answer, at full fluency and with no warning.

• Flawed: “Determine if this customer is eligible for a refund.”
• Fixed: “Determine if this customer is eligible for a refund by comparing their purchase date against the 30-day policy rules provided below: [Refund Policy Document].”

The repair is to put the deciding material in the window. Nothing else fixes this one. In module 1.06 you ran one question six ways and watched the answer change only when the deciding text was placed in the prompt; worked examples set the shape and scope of the answer, and the fact itself arrives only if you put it in the window.

instruction — it described the work instead of doing it

Passive or observational phrasing produces commentary. Action verbs produce actions.

• Flawed: “Look over this technical support transcript.”
• Fixed: “Categorize this support transcript into exactly one category: Billing, Software Bug, or Feature Request.”

Categorize, extract, calculate, rank, rewrite are jobs. Look over, consider, examine are invitations to muse.

format — right answer, unparseable

The content is correct and the downstream code cannot read it.

• Flawed: “Return the results as JSON.”
• Fixed: “Return a JSON object containing two key-value pairs: status (string) and confidence_score (integer between 0 and 100).”

“As JSON” leaves the key names, nesting, and types to chance, and they will vary between runs. Keep the fixed version in mind: two named keys, one of them an integer from 0 to 100. That is the exact shape the critic returns at the end of this module, and the reason it can be gated is that its format variable was pinned.

why adjusting the wrong variable makes things worse

Tighten the format on a prompt whose real problem is missing context and you get a neatly structured, schema-valid, confidently wrong answer. You have made the failure harder to spot and easier for downstream code to accept.

Why does that follow rather than merely happen? Because of what you know about the machine: format instructions constrain the shape of the continuation, and the missing fact was never in the window to begin with, so nothing about the shape can conjure it. Watch it on the prompt in the exercise below, whose outputs are real runs, compressed. “Write documentation for our API. Keep it under 120 words.” produced a fluent page of invented endpoints. Move the clarity variable, asking for clear, well-organized, simple language, and it invented a different API. Move the precision variable, demanding exactly three sections titled Auth, Endpoints, and Errors, and it produced three tidy sections of invented endpoints, the neatest confabulation of the set. Only the context move, pasting the real specification into the prompt, changed what the documentation said. Module 1.06 ran the same experiment on one question six ways and reached the same finding: the answer moved only when the deciding content was placed in the context.

Diagnose first. Then edit. That is what one variable at a time means in practice, and the panel below is where you drill it.

Test the diagnosis on real outputs below. Each option shows you the genuinely revised prompt and the genuinely revised result, including the adjustments that leave the failure exactly where it was:

Now watch one variable move on the Tutor from module 1.02. The session is illustrative, composed in the Tutor’s register to show the machinery of its YAML, and what it shows is the instruction variable moving on its own while the grounding doctrine and the laws hold still:

finding the failures before your users do

Fixing what you happened to notice leaves everything you did not think to try. So put someone hostile on it deliberately.

An adversarial persona is a separate prompt whose whole job is to attack your agent’s instruction. You are asking a model to be a difficult reviewer, and nothing about the request asks it to be helpful.

<system_identity>
You are a skeptical senior QA analyst who trusts nothing that has not failed
at least once in front of you. Your goal is to find flaws, false premises,
and edge-case failures in the target system prompt.
</system_identity>

<execution_framework>
Review the target prompt and enumerate ten realistic user inputs designed to
trigger confabulation, policy bypasses, missing-context traps, or format
errors. Rank them by probability and potential harm.
</execution_framework>

Read that as: an identity that says what this agent is solely for, and a workflow block that says what to produce and in what order. Both are module 2.01’s block anatomy, and the second is the workflow section that module called optional, present here because the job has steps. Nothing exotic. The value is entirely in pointing it at your own work.

What do you do with what it finds? Each hole it opens is an input that broke your agent, and the loop above already told you: keep the failing input. That habit is worth its own rule, because it is the cheapest one in the module and the most often skipped.

Every fixed failure becomes a test. When an adversarial run finds a hole, save the input that found it next to the corrected prompt. A bug you fixed and did not record is a bug that returns the next time somebody edits the instruction.

Take the pre-built tester with you: qa-persona-prompt.md.

automated review loops

Hand debugging builds the intuition. It does not scale to a system answering thousands of requests an hour, where nobody is reading the outputs at all.

So you put the reviewer inside the system. Module 1.01 introduced this as the agent in the loop: a second model placed between the generator and the user with one job, inspection, and no stake in the draft being good. Here we build one. Melchizedek’s critic.yaml splits the work in two: a DrafterAgent produces the response, and a separate CriticAgent scores it before anything reaches the user.

The critic returns a structured object rather than a paragraph of opinion:

- name: "CriticAgent"
  generateContentConfig:
    responseMimeType: "application/json"
  outputSchema:
    type: "OBJECT"
    properties:
      message:    { type: "STRING" }
      confidence: { type: "INTEGER" }   # below 85 triggers a revision loop
    required: ["message", "confidence"]

Read that as: the critic is only allowed to answer with two fields, a string and an integer, and the orchestrator’s instruction branches on the integer. In the full file the critic’s own instruction gives the integer felt scale: 90 to 100 means excellent and factually verified, 70 to 89 means good with minor gaps, and below 70 means significant issues. The gate at 85 sits inside the middle band, so “good but could be clearer” is not good enough to ship. Where did the paragraph of opinion go? Into the format variable. Pin the shape to two named keys and the judgment becomes a value code can compare. That reasoning ends in a rule:

A gate reads a number, not a paragraph: an automated quality gate reads a machine-parsed score, never conversational text. A number crosses a threshold deterministically; a paragraph saying “this looks pretty good” does not.

This is module 2.01’s standard applied to the review step: every rule must be checkable against a transcript, or the model treats it as optional. “Is this response good enough” cannot be checked. “Is confidence at least 85” is checked by an if statement.

why the critic must be a separate agent

The obvious cheaper design is to ask the drafting model to review its own work in the same conversation. Why doesn’t that work? The reason is mechanical rather than psychological.

Everything the drafter wrote is sitting in its context window: the reasoning, the assumptions, the phrasing. Those tokens act as strong statistical conditioning on every token that follows, so asking “is this correct?” in that same window makes agreement the likely continuation; the whole preceding text is an argument for the answer it just gave. You reliably get a fluent explanation of why the draft is fine, because the conditioning that produced the error is still in the window, still shaping the next token. Module 1.01 said the same thing about the review agent, and the mechanism modules of Part 1 showed you the arithmetic behind it: a model checking its own work in the same pass is running the same weights over the same context that produced the error.

A separate call gives you three things instead:

An independent context window. The critic sees the draft and the quality bar, and none of the drafter’s reasoning conditions its tokens. It judges the output rather than the intention behind it.

A specialized instruction. The critic’s prompt is about evaluation and error detection only. Nothing in it asks the critic to be helpful about the request.

Deterministic routing. A number below the threshold sends the draft back with the critic’s specific concerns attached, up to three rounds.

One hard constraint holds the structure up, found through testing and now written into the file’s own comments: in this framework, an agent that promises a structured output cannot also hold the power to transfer work to other agents; give one agent both and the run stalls and never returns. So the schema lives on a leaf agent with its transfer capability stripped, and the orchestrator is a plain sequencer. In module 2.04 you will apply this as a rule to every schema you place, and read the whole schema surface key by key; for now, take it as the constraint it is.

Grade the gate before you trust it. Two limits stay open by design. The critic and the drafter in critic.yaml run on the same model, so a blind spot in the weights, a fact the model has wrong, is a blind spot in both windows, and a separate call cannot catch it; the independent window catches errors of context, not errors of memory. And the threshold is only as honest as the integer behind it: 85 is a score the critic assigns to itself, and nothing in the loop checks the critic’s calibration. What the design closes is the same-window agreement failure, and that is the one it was built to close.

Watch the loop run. The trace below is illustrative, built to the machinery in critic.yaml, and its numbers are chosen to show the gate working: a first draft scored 62, sent back with the critic’s concerns quoted verbatim, and a revision scored 91, returned to the user word for word:

The workflow download carries one unedited live run beside the YAML, from 2026-07-11: asked in two sentences why the Library of Alexandria declined, the loop passed in a single round at confidence 98. Provoked three times, with a normal question, a false premise, and an impossible request, the drafter survived all three in one round; the false premise was corrected and the impossibility was proven. Read that as the honest scale of the instrument: with strong models the gate mostly stays green, and you keep it anyway, because the day it goes red is the day it earns its keep.

run the critic loop

1. Take the workflow definition: critic-workflow.md.
2. Take the adversarial tester: qa-persona-prompt.md.
3. Run npm run syndicate:critic -- "Your test query". This one uses cloud models, so it wants the free Gemini key; the five-variable drills stay free on your local Tutor.

what you can do with this today

Every mechanism in this module is a question you can ask on a working day, and each answer has a fix attached.

What, in one sentence, is wrong with this output? Say it before you touch the prompt. The sentence picks the variable. Editing before you can write that sentence is guessing, and the discipline starts here.

Which one variable does that sentence name? Change that one, then re-run on the same input. Revert what did not help, so the prompt stays a document you can reason about.

Did I keep the input that broke it? Every fixed failure becomes a test. The input that broke it becomes a permanent test, the cheapest regression suite in software, and almost nobody builds it.

Who is grading this, and were they in the window that wrote it? Never let the maker grade the making. Whether the reviewer is you at the verification stage or a critic agent in the loop, the check has to come from outside the pass that produced the work, and an automated one reads a number, never a paragraph.

Here is the whole discipline on one card, in the vocabulary you now own:

BY HAND — one variable at a time
1. RUN      the agent on a real input, and the awkward ones
2. NAME     one sentence: what is wrong with the output
3. PICK     the variable that sentence names
              clarity     it did the wrong task
              precision   right task, wrong shape or size
              context     a confident invention
              instruction it described instead of did
              format      correct but unparseable
4. CHANGE   that one variable, nothing else
5. RE-RUN   same input; revert what did not help
6. KEEP     the failing input as a test

IN THE LOOP — critic.yaml
1. DRAFTER  produces the response, its own window
2. CRITIC   a separate call, a leaf with the schema:
              { message: STRING, confidence: INTEGER 0–100 }
3. GATE     confidence >= 85 ships; below 85 loops with the concerns
4. CAP      three rounds, then the latest verdict ships, score attached

Splitting work between a specialist that produces and a specialist that judges is the smallest multi-agent system there is. In the next module you will scale it up: choose among four shapes for how work flows between agents, read an orchestrator’s instruction as a contract for handing work off, and give each agent a voice you specified as countable laws, all on the local open weights you already run. Open the syllabus to continue.