Grading agents with eval suites

Consider this output from a delegation router, when asked to write a Python Fibonacci function with a short docstring:

The coding task has been completed by CodeExpert. Let me know if you need any further assistance!

Notice what happened here: there is no code, and there is no docstring. Yet from the perspective of runtime telemetry, no error was raised. The orchestrator routed the request to the correct specialist, the specialist returned an answer, and the relay back to the user executed cleanly. Every operational health check remained green, yet the user received a perfectly structured sentence containing none of the requested work.

Our foundation on model mechanics explains why this mode of failure is so pervasive in agent architectures. A language model predicts the most probable token continuation. Following a delegation step, a polite handoff message is a statistically probable continuation. The underlying runtime maintains no internal representation of whether the requested artifact was actually delivered. This failure is silent by design; consequently, the only mechanism capable of detecting it is a dedicated system that evaluates and grades the output.

You built an initial version of this mechanism in our lesson on prompt testing and refinement, where you manually debugged an execution trace, isolated the failing variable, and implemented a critic agent that scored drafts so an execution loop could gate on quality. Now, we expand that concept in two directions. First, we will run the critic across an entire dataset across controlled configurations, using rigorous statistics so that claims of improvement represent defensible differences. Second, we will apply these identical graders to production traffic using an append-only ledger that records every turn your agents take. By the end of this lesson, you will be able to evaluate any modification to an agent, quantify whether it improved performance, determine the statistical confidence of the change, and make an automated deploy decision.

One statement about this module’s numbers, tools and pictures, made once. Every number in this module is measured from a run of a delegation router or a fact-checker suite (the Fibonacci and derivative outputs, the rubric scores, the $0.00068 smoke test, the confidence-interval tables, the judge-comparison table, the calibration record, the gate report, and the 4,786-character production turn); none is shaped. Those runs used the router’s earlier instruction. The router’s current instruction carries a guardrail written after the Fibonacci failure, which forbids a reply that only reports the hand-off, and the case stays in the suite as the test that the guardrail holds. The grader half of this lesson is a case study drawn from a private evaluation harness: you can read its suite files, commands and reports here, and you cannot install it. The ledger half describes a telemetry schema that records every production turn. Two figures idealize: the harness figure draws the judge kinds side by side where a suite stacks whichever it needs and runs them over the same records in turn; the ledger figure draws twelve columns of a row that carries about forty, and its three questions are a sample of what the columns support.

a suite is a directory of cases, variants, and judges

An eval suite organizes three foundational elements within a single directory: a dataset, the variants you wish to compare, and the judges tasked with grading performance. In the harness, each element is declared as a distinct configuration block inside suite.toml, defining the operational parameters of the test harness.

The dataset is stored as a JSONL file containing one test case per line. Each case specifies an input (or a structured list of turns for multi-turn conversations), an optional expected output value, and diagnostic tags:

{"id": "code-01", "input": "Write a Python function that returns the nth Fibonacci number, with a short docstring.", "expected": {"delegate": "CodeExpert"}, "tags": ["code"]}
{"id": "chat-01", "input": "Hi there! How's it going today?", "expected": {"delegate": null}, "tags": ["chat"]}

The variants represent the experimental configurations under your direct control. A variant designates a base team definition alongside specific overrides: an alternative model, a thinking level, a sampling temperature, a system prompt addition, or a direct patch to an individual agent. These overrides are applied entirely in memory to the parsed definition, preserving the integrity of on-disk configuration files while modifying nested subagent references. When diagnosing a specific cause, you modify a single parameter at a time, because a difference between the two variants can then be traced to that parameter. When evaluating an operational migration, such as moving to a smaller model with thinking turned off to reduce cost, variants often combine several parameter changes. The router suite’s two variants both point at the same router definition: lite-nothink sets the model to gemini-3.5-flash-lite with thinking off, and flash-low sets gemini-3.8-flash with thinking at low. Because lite-nothink changes both the model and the thinking level, its report says whether the lighter configuration does worse as a whole and cannot say which of the two lines produced the difference.

The judges evaluate each execution result to produce an explicit verdict: a boolean pass/fail status, a normalized continuous score between zero and one, structured output fields, and an explanatory rationale. Eval suites typically combine multiple judges, and the resulting report presents each evaluation criterion in a dedicated column.

The total execution workload corresponds to the Cartesian product: variants × cases × trials. A trial represents an independent run of the same test case under identical variant settings. We require multiple trials because language model outputs are inherently stochastic; a single prompt run twice does not guarantee an identical response. For instance, executing two trials across ten cases and two variants yields forty total runs. When evaluating small, fast models, the cost of forty runs is negligible. A twelve-run smoke test of our fact-checker suite cost $0.00068, meaning a full evaluation pass of thirty labeled claims across two variants and two trials (one hundred and twenty runs) costs approximately seven tenths of a cent ($0.007). Calculate your costs once, and you can comfortably integrate evaluations into every prompt iteration rather than saving them for major releases.

One architectural decision in the harness matters most. The harness manages suite orchestration, dataset ingestion, statistical scoring, and reporting, but it delegates all agent inference. Every agent-team execution and judge invocation routes through a Node bridge that compiles the active agent graph with the same compiler the A2A server uses. This ensures that your evaluation suite executes against the exact same compiled graph that the A2A server deploys to production users. If an evaluation harness compiles its own independent representation of an agent, it merely measures that copy; this harness measures the exact system you ship.

one directory · suite.toml

dataset

cases.jsonl

input · expected · tags, one JSON object per line

variants

a team definition plus overrides

model · thinking · per-agent patches

the bridge

the same compiler the A2A server runs

the server’s compiler, so an eval measures the graph you serve

script

facts a program knows

right specialist · valid JSON · latency · length

rubric · golden

a judge model reads the answer

weighted criteria, or a check against a reference

pairwise

two answers, both orders

a winner must survive the swap or it is a tie

worse than the pinned baseline?

exit 0 · the deploy proceeds

exit 1 · the deploy script refuses to publish

four ways to grade an answer

Return to the Fibonacci request that opened this module. The dispatcher routed it to CodeExpert, the router’s relay of the specialist’s answer did not fall back, and the process returned without error. The delegation suite’s deterministic check reads the execution trace and confirms exactly those three facts: the delegate was the one the case expected, no relay fallback fired, and no error was raised. Everything the check inspected is true. Yet the response contains a polite handoff sentence and no code.

The checks this suite ran all read the execution trace, and the trace satisfied every invariant they asserted; none of them examined the generated answer. A program that reads the answer would have caught this failure as well, because a parser finds no Python function definition in it. Whether a delivered function returns the right Fibonacci numbers is a question for tests that execute it, and whether its docstring is short and clear is a question for an evaluator that reads the text of the answer against the request.

That distinction divides evaluation into four kinds of question: a fact a program can extract, a quality only a reader can weigh, a fact checked against a reference answer, and which of two answers is better. The harness answers these through four judge configurations.

To verify a fact a program can extract from the execution trace, configure a programmatic judge (kind = "programmatic"). The programmatic judge reads structured values directly from the trace: a JSON payload, a regular expression match, the routing path chosen by a dispatcher, or the specific subagent invoked. It compares these extracted attributes against expected values defined in the test case or asserts global invariants over the execution. Common programmatic assertions include verifying valid JSON syntax, confirming the absence of unhandled errors, preventing unwanted emoji characters, enforcing maximum character budgets (for example, staying under four thousand characters), ensuring proper specialist delegation, and asserting that latency stays under fifteen seconds. When evaluating binary classification tasks, the harness automatically generates accuracy, precision, recall, F1 scores, a full confusion matrix, and internal consistency metrics tracking whether the agent repeated identical decisions across trials. If an assertion requires custom logic beyond standard configuration rules, you can place a Python module alongside the suite to export a custom grading function. For instance, our fact-checker suite uses a programmatic judge to assess model calibration: confident yet incorrect predictions are penalized heavily, causing the evaluation to fail any run where model confidence is at least 0.9 while its verdict is wrong. Whenever an assertion can be expressed as deterministic code, use a programmatic judge. It executes with negligible computational cost and eliminates subjective variance.

To evaluate a quality only a reader can weigh, configure an llm judge (kind = "llm"). For generated code, the question separates into three tiers. Syntax belongs in a programmatic judge: a parser or compiler checks deterministically whether the code parses. Behaviour under specified tests also belongs in a programmatic judge: a custom grading script executes the code against test cases the suite specifies and confirms that it returns the expected outputs. Open-ended qualities such as semantic alignment with user intent, clarity and conversational conciseness are what the llm judge grades. You define an explicit rubric containing named criteria, scoring descriptions, numerical scales, and relative weights. The harness compiles this rubric into a structured-output JSON schema. The judge is a standard agent constrained by this schema, routed through the shared model registry to run against Gemini, Claude, or local open-weight models. Because structured outputs are strictly enforced at the provider API level, all rubric dimensions return as structured JSON properties rather than free-form text that requires post-hoc parsing.

To evaluate a fact checked against an authoritative reference answer, configure a golden judge. A golden judge extends an llm judge by supplying a reference answer key. Each test case includes a human-authored reference answer that the judge inspects alongside the agent’s output. By default, the rubric grades factual correctness as a discrete enum (correct, partially_correct, or incorrect), evaluates completeness, and identifies explicit contradictions and omissions. Variations in phrasing are permitted; factual discrepancies are penalized. Reference answers can be simple text strings or structured metadata objects containing explicit instructions for the grader. For example, in our concierge suite, a test case for “what is the capital of Australia” includes grader notes specifying that identifying Sydney must be scored as incorrect rather than partially_correct. Similarly, for “why is the Great Wall visible from the Moon”, the grader notes state that an acceptable response must explicitly reject the false premise rather than explain it.

To resolve which of two candidate answers is better, configure a pairwise judge. The pairwise judge receives two competing candidate outputs alongside descriptive evaluation criteria and selects the preferred option. To mitigate position bias (where a model systematically favors whichever response appears first), the harness evaluates every candidate pair twice with the presentation order reversed. A candidate is declared the winner only if it prevails across both presentation orders; otherwise, the harness scores the comparison as a tie. The harness explicitly reports this position bias rate, ensuring that positional drift is quantified rather than ignored.

the arithmetic of an llm judge

Rubric scoring relies on standard weighted normalization, and tracing this arithmetic makes its mechanics fully transparent. Our delegation suite applies a rubric across four discrete criteria: factual correctness (scored 1 to 5, weight 3), completeness (scored 1 to 5, weight 2), clarity (scored 1 to 5, weight 1), and routing correctness (a boolean indicating specialist accuracy, weight 1).

Consider this response from the router when asked: “what is the derivative of x^3 + 2x? Show the rule you used”:

The derivative of $x^3 + 2x$ is $3x^2 + 2$. As detailed by the MathExpert, this result is obtained by applying the Sum Rule, the Power Rule, and the Constant Multiple Rule.

The llm judge populated the structured schema with the following scores: correctness 5, completeness 2, clarity 3, and right specialist true. Each integer score is normalized to a continuous scale between 0.0 and 1.0 by subtracting the scale minimum and dividing by the score range. Thus, a score of 5 becomes 1.0, 2 becomes 0.25, 3 becomes 0.5, and a boolean true normalizes to 1.0. We calculate the dot product of these values against their respective weights and divide by the total weight sum:

(3 × 1.0) + (2 × 0.25) + (1 × 0.5) + (1 × 1.0)     5.0
───────────────────────────────────────────── = ───── = 0.714
                 3 + 2 + 1 + 1                     7.0

With the suite’s pass_threshold configured at 0.75, an aggregate score of 0.714 triggers a test failure. The accompanying rationale explains the failure precisely: while the mathematical conclusion is accurate, the user explicitly requested a step-by-step demonstration of the differentiation rules. The model merely listed three rule names without showing their application. This represents an evaluative distinction that cannot be captured via regex scripts, yet is immediately obvious to a human reviewer.

A rubric is a schema, not a mood. Criteria become fields the provider’s API forces the judge to fill, weights turn the fields into one number, and the threshold turns the number into a decision. Every step is visible and every step is yours to set.

the judge may not be a model under test

Language models exhibit measurable self-preference bias, systematically assigning higher ratings to outputs produced by their own model family than an independent model assigns. Consequently, evaluating a Gemini-based agent using a Gemini judge invalidates the objectivity of the benchmark.

The harness actively prevents this configuration before runs execute. During the preflight check, the harness cross-references each judge model against every agent model declared in your active variants, traversing all nested subagent definitions. If an identical model is detected across both roles, the harness halts execution; if the models share an underlying provider family, it issues an explicit warning. Post-deployment evaluation suites enforce identical constraints against the execution traces loaded from production. If you must bypass this restriction, you can explicitly set allow_same_model = true on the judge configuration, with the standard expectation that the rationale is documented in an accompanying code comment.

In the production suites this module draws on, Gemini-based agent teams are graded by an independent judge, claude-sonnet-4-6. Two runs on the same test cases, each graded by a different judge model, show where the judges diverged:

judged by	code-01 (Fibonacci)	math-01 (derivative)
gemini-3.7-flash (the flash-low variant’s own model)	0.179, fail	0.750, pass
claude-sonnet-4-6 (independent)	0.179, fail	0.714, fail

Both judge models identified the missing Fibonacci implementation and produced identical scores. Their assessments diverged on the calculus problem. The two rows come from two runs of the same variant on the same case under identical rubric weights, and each run generated its own answer: the first told the user to see a step-by-step breakdown “above” that the user could not see, and the second credited “the MathExpert” for detail it never relayed. The independent judge scored clarity at 3, whereas the same-model judge awarded a 4. On a criterion with a weight of 1, that single point accounts for a 0.036 shift in the composite score, the margin separating 0.714 from the 0.75 pass threshold. Because the two answers differ, the pair shows the direction of the effect without isolating it; the next tightening is to re-judge one stored record with both judges, so that the answer stays fixed and only the judge changes. Across three evaluation cases, the same-model judge passed the lite-nothink router twice; the independent judge failed both instances shown above.

Treat your evaluation harness with the same empirical skepticism as any measurement system. The exact magnitude of self-preference bias varies across model families, rubrics, and task domains; the rule holds at every magnitude.

Judge independence: a judge model may not be a model under test. Prefer a different provider, not merely a different checkpoint, and keep a programmatic judge beside every rubric so at least one grader has no opinion to be biased by.

a difference you can believe carries a confidence interval

When variant B records a higher pass rate than variant A, one is tempted to conclude immediately that the new prompt or model is superior. To prevent premature conclusions, the harness computes a seeded 95% bootstrap confidence interval for every reported metric. Multi-variant suites generate a dedicated Comparisons summary that pairs test cases across variants, computing the exact delta along with its confidence interval and setting a significant boolean flag only when the interval strictly excludes zero.

Here are the results from an initial evaluation of our delegation router across three test cases:

| variant   | judge   | metric    | delta   | 95% CI          | cases | significant |
| flash-low | quality | pass_rate | +33.3%  | [+0.0, +100.0]  | 3     | no          |
| flash-low | routing | pass_rate |  +0.0%  | [+0.0, +0.0]    | 3     | no          |

Notice that despite a raw pass rate increase of +33.3%, the harness marks the change as not statistically significant. The confidence interval spans from +0.0% to +100.0%. Statistically, three cases provide insufficient power to distinguish between a genuine improvement and random variance. The absence of a significance flag accurately reflects the limited scale of your sample.

Conversely, you must also understand how small sample sizes can artificially trigger significance. In a separate two-case evaluation, the harness reported a delta of +100.0% with a confidence interval of [+100.0, +100.0] and marked significant: yes. Because both test cases moved positively in unison, every bootstrap resample produced an identical value, collapsing the interval to a single point that technically excluded zero. The significance flag accurately reported that the calculated interval omitted zero, but it cannot assess whether two test cases constitute a meaningful sample. Always evaluate sample size and statistical flags in tandem. A bootstrap confidence interval resamples only the test cases included in the dataset. It quantifies how much the observed metric might move across those specific cases, and it says nothing about task variations or edge cases omitted from your suite.

Multiple trials fulfill a complementary purpose. Repeatedly evaluating identical test cases measures an agent’s internal stability, reported as trial_agreement. If an agent succeeds on trial one but fails on trial two, a single-run evaluation would have obscured that instability. A low agreement score indicates that your evaluation reflects a wide probabilistic distribution rather than a reliable operational capability.

a rubric must be calibrated against human agreement

Because llm judges are language models applying qualitative criteria, you must verify their alignment with human judgment before granting them deployment authority. The harness treats human labels as an integrated workflow rather than an ad-hoc exercise.

The harness’s label <run> --sample 20 command extracts a deterministic random sample of execution records from a run, presenting the user input, the expected reference, and the agent’s output while hiding the automated judge’s verdict. You record your assessment with a single keystroke (p for pass, f for fail). Your annotations are saved directly to the run directory; passing the --persist flag stores them in the central database so they persist across subsequent re-evaluations. Its calibrate <run> command then calculates raw agreement alongside Cohen’s kappa, an established inter-rater reliability metric that corrects for the agreement two raters would reach by chance, given each rater’s own pass rate. The resulting kappa score maps to standard statistical tiers: slight, fair, moderate, substantial, or almost perfect.

What “corrects for chance” means is worth one line, because this number decides whether a judge may gate a deploy. Count how often the judge and the human reached the same verdict: that is the observed agreement. Then work out how often two unrelated raters, each keeping its own pass rate from the real pair, would have landed together by luck alone: that is the chance agreement. Kappa is what survives of the first once the second is taken out of both the score and the room left to score in:

$$\kappa = \frac{\text{observed agreement} - \text{chance agreement}}{1 - \text{chance agreement}}$$

Read that as: agreement beyond luck, over the agreement luck left available. Two raters who both pass nearly everything agree constantly and demonstrate nothing, which is how a judge posts a high raw agreement and a kappa near zero — exactly what the routing judge does three paragraphs below.

The harness further categorizes disagreements into two distinct operational classes. A lenient disagreement occurs when the automated judge passes an output that a human reviewer failed; this represents an unmitigated quality defect that reaches production users. A harsh disagreement occurs when the judge fails an output that a human validated; this creates unnecessary engineering triage without impacting end users.

Annotating three records from our delegation router evaluation revealed a foundational calibration insight. The llm judge agreed with the human annotator across all three instances. In contrast, the programmatic routing judge (which relies solely on deterministic structural checks) achieved a raw agreement of 0.667 and a Cohen’s kappa of 0.0 (“slight”). Its single disagreement was the Fibonacci case, and its rationale was three words long:

case_id     code-01           judge: passed        human: failed
note        "the code is missing"
rationale   "all checks passed"

Every deterministic check passed successfully: the router delegated to CodeExpert, the relay executed without error, and no runtime exceptions occurred. The programmatic judge was completely accurate regarding the operational parameters it was designed to observe, yet entirely blind to the missing implementation. This concrete divergence demonstrates why effective evaluation stacks require both programmatic and rubric-based judges: calibration identifies the specific failure modes your deterministic scripts did not check.

Three annotated records do not constitute an adequate calibration sample, and our harness does not flag that: it computes kappa over however many labels exist, and a gate accepts a three-label kappa as calibration. Its uncalibrated warning appears only when a judge has no human labels at all. Twenty is the default sample size of the label command, and it is a starting sample. Twenty binary labels still give a coarse measurement, since a single disagreement shifts observed agreement by five percentage points. Agreement is also specific to a task and a rubric, so you gather fresh labels and compute kappa again whenever you modify rubric criteria, change scoring weights, or point the judge at a different task.

A judge is calibrated or it is an unvalidated heuristic. Collect human labels with the verdict hidden, calculate Cohen’s kappa alongside the direction of the lenient and harsh disagreements, set a floor per judge, and do not let a judge below its floor decide a deploy. A judge with no labels has no kappa to compare against that floor, so in your release pipeline treat missing calibration as a blocking failure.

The routing judge’s three records, from the module’s run. The person failed code-01 and passed the other two; the routing judge passed all three. They agree on two of three: observed agreement 0.667.

The judge passes everything (a rate of 1) and the person passes two in three. Two unrelated raters with those rates both pass 1 × 2/3 of the time and both fail 0 × 1/3 of the time: chance agreement 0.667. So kappa = (0.667 − 0.667) / (1 − 0.667) = 0.0. A judge that passes everything agrees with whoever passes most things, and kappa takes that agreement away.