The grader and the ledger

Here is a real answer from the starter pack’s delegation router, asked to write a Fibonacci function with a short docstring:

The coding task has been completed by CodeExpert. Let me know if you need any further assistance!

No code. No docstring. No error either: the run completed, the orchestrator delegated to the right specialist, the specialist answered, and the relay of that answer back to the user was structurally sound. Every signal a running system watches was green. What arrived was a well-formed sentence containing nothing.

Module 1.01 explains why this shape of failure is the common one. The model predicts the most probable continuation, and after a delegation, a polite handoff sentence is an extremely probable continuation. Nothing in the machinery holds a belief about whether the code came with it. The failure is quiet by construction, and quiet failures are found by exactly one thing: something that reads the output and grades it.

You have built that something once already. In module 2.02 you debugged one output by hand, named the failing variable, and then handed the judgment to a critic agent that scored a draft as a number a loop could gate on. We take that same move and do two things with it. First we run the critic over a whole dataset under configurations you control, with enough statistics that a claim of improvement becomes a difference you can defend. Then we turn the same graders on production, over a ledger that has been recording what your agents actually did since the day you switched it on. By the end you will be able to say, of any change you make to an agent, whether it got better, by how much, with what confidence, and whether it may ship.

a suite is a directory

An eval suite is three things in one folder: a dataset, the variants you want to compare, and the judges that grade the results. In the melchizedek observatory each of those is a block in one suite.toml, and everything the harness does follows from them.

The dataset is a JSONL file, one case per line. A case is an input (or a list of turns for a conversation), an optional expected value, and tags:

{"id": "code-01", "input": "Write a Python function that returns the nth Fibonacci number, with a short docstring.", "expected": {"delegate": "CodeExpert"}, "tags": ["code"]}
{"id": "chat-01", "input": "Hi there! How's it going today?", "expected": {"delegate": null}, "tags": ["chat"]}

The variants are the inputs you control. A variant is a syndicate file plus overrides: a model, a thinking level, a temperature, a sentence appended to every agent’s prompt, a patch to one named agent. The overrides are applied in memory to the loaded definition, so the file on disk is never touched, and they reach through nested references into subagents defined in other files. Two variants of the same router differing by one line is the normal case:

[[variants]]
label = "lite-nothink"
syndicate = "delegation.yaml"
[variants.overrides]
model = "gemini-3.1-flash-lite"
thinking = "off"

[[variants]]
label = "flash-low"
syndicate = "delegation.yaml"
[variants.overrides]
model = "gemini-3.7-flash"
thinking = "low"

The judges turn each result into a verdict: passed or failed, a score between zero and one, named fields, and a written rationale. A suite normally stacks several, and the report gives each one its own column.

What actually runs is the product: variants times cases times trials. A trial is a repetition of the same case under the same variant, and it exists because a model asked the same question twice does not always answer the same way. Two trials over ten cases and two variants is forty runs, and forty runs of a small model is cheap enough to stop thinking about. A twelve-run smoke test of the fact-checker suite cost $0.00068, which puts a full pass of its thirty labeled claims, two variants and two trials deep, at a hundred and twenty runs for about seven tenths of a cent. Price it once, then run it on every prompt edit instead of every release.

One design decision inside the harness matters more than any of its features. The Python side owns suites, datasets, scoring and reports, and it never runs inference. Every syndicate call and every judge call goes through a small Node bridge that compiles the agent graph with lib/compile.ts, the same compiler the A2A server calls when it serves that syndicate to real users. An eval that builds its own copy of the agent measures the copy. This one measures the thing you deploy.

four ways to grade an answer

The judge kind follows from what the question can be answered with, and the order below runs from the most certain to the least.

A programmatic judge is a script. It extracts something from the result, such as a JSON field, a regex capture, the route the dispatcher chose, or the subagent that was consulted, then compares that with the case’s expected value, or asserts a property of the run regardless of the answer. Those assertions are the checks: valid JSON, no error, no emoji, under four thousand characters, delegated to the expected specialist, latency under fifteen seconds. When a boolean label is being compared, the report gives you accuracy, precision, recall, F1 and a confusion matrix for free, plus how often the agent agreed with itself across trials. When a check does not fit a configuration line, a Python file next to the suite exports one function and does the work. The fact-checker suite uses one to score calibration: a checker that is confident and wrong is worse than one that is unsure and wrong, so the script fails any run where confidence is at least 0.9 and the verdict is false.

Use a script wherever a script will do. It is free, it is deterministic, and it never has an opinion.

An llm judge exists for the questions a script cannot reach: is this code correct, does this reply actually contain what was asked, is it longer than it needs to be. You write a rubric: a list of criteria, each with a description, a scale, and a weight. The harness compiles that rubric into a structured-output schema. The judge is an ordinary agent with that schema attached, routed through the same model registry as any syndicate, so a Gemini model, a Claude model, or a local open-weight model can all take the job. Structured output is enforced by the provider’s API, so the fields come back under the names the rubric declared rather than as prose you would have to parse.

A golden judge is the llm judge with an answer key. Each case carries a reference answer a person wrote, the judge sees it beside the agent’s reply, and the default rubric asks for correctness as an enum of correct, partially correct and incorrect, plus completeness, and lists of contradictions and omissions by name. Wording may differ; facts may not. The reference can be a plain string or an object carrying notes for the grader, which is how the concierge suite handles its two awkward cases: for “what is the capital of Australia” the notes say that Sydney is incorrect rather than partially correct, and for “why is the Great Wall visible from the Moon” the notes say a correct answer must reject the premise instead of explaining it.

A pairwise judge answers the only question that matters at the moment you are choosing between two versions: which of these two answers is better. It shows the judge both replies under free-text criteria and asks it to pick. Every pair is judged twice with the order swapped, because a judge shown the same two answers in a different order will sometimes change its mind, and a winner that does not survive the swap is recorded as a tie. How often that happens is reported as the position bias rate, so the bias is measured rather than assumed away.

the arithmetic of a rubric

A rubric score is arithmetic, and it is worth watching one being computed. The delegation suite’s rubric has four criteria: correctness on a one-to-five scale at weight three, completeness one-to-five at weight two, clarity one-to-five at weight one, and a boolean for whether the right specialist handled it at weight one.

Here is the router’s reply to “what is the derivative of x^3 + 2x? Show the rule you used”:

The derivative of $x^3 + 2x$ is $3x^2 + 2$. As detailed by the MathExpert, this result is obtained by applying the Sum Rule, the Power Rule, and the Constant Multiple Rule.

The judge filled the schema with correctness 5, completeness 2, clarity 3, right specialist true. Each integer scale normalizes onto zero-to-one by subtracting the floor and dividing by the range, so 5 becomes 1.0, 2 becomes 0.25, 3 becomes 0.5, and the boolean becomes 1.0. Multiply by the weights and divide by their sum:

(3 × 1.0) + (2 × 0.25) + (1 × 0.5) + (1 × 1.0)     5.0
───────────────────────────────────────────── = ───── = 0.714
                 3 + 2 + 1 + 1                     7.0

The suite’s pass_threshold is 0.75, so 0.714 is a fail, and the rationale says why: the answer is mathematically right, the user asked to see the rule applied, and what came back names three rules without showing one of them being used. That is the judgment a script could not have made and a person would have made instantly.

A rubric is a schema, not a mood. Criteria become fields the provider’s API forces the judge to fill, weights turn the fields into one number, and the threshold turns the number into a decision. Every step is visible and every step is yours to set.

the judge may not be a model you are testing

A model shown its own output rates it higher than an independent model does. The effect has a name, self-preference bias, and it is the reason a Gemini judge over a Gemini agent is not an evaluation.

The observatory refuses that arrangement before it costs you a run. Preflight compares each judge’s model against every agent’s model in every variant, following nested references down through subagents, and fails the run if any of them match; when the models merely share a provider family it warns instead. Post-deploy suites run the same check against the models recorded on the loaded turns. Overriding it is possible and deliberate: set allow_same_model = true on the judge, and the convention is to write the reason in a comment beside it.

The shipped suites judge Gemini agents with claude-sonnet-4-6. What that buys is visible in two runs of the same suite differing in who graded. Both of the cases below ran under both judges, on the same cheap variant:

Both judges caught the empty Fibonacci reply and scored it identically. They parted on the derivative. The same reply, the same rubric, the same weights: the independent judge scored clarity 3 where the same-family judge scored 4, and one point of clarity on a weight-one criterion is 0.036 of the final score, which is exactly the distance between 0.714 and the 0.75 line. One point, on one criterion, on one case, moved the verdict from pass to fail. Across the three cases it saw, the same-family judge passed that router twice; the independent judge, over the two cases above, passed it never.

Grade this instrument as you would any other. Two runs of three and two cases demonstrate the mechanism and do not measure the size of the bias, which varies by model pair, by rubric, and by task. Take the rule from them, and not the number.

Judge independence: a judge model may not be a model under test. Prefer a different provider, not merely a different checkpoint, and keep a script judge beside every rubric so at least one grader has no opinion to be biased by.

a difference you can believe

Two variants produce two pass rates and the temptation is immediate: the second number is higher, therefore the change worked. Every figure in an observatory report carries a seeded bootstrap 95% confidence interval to make that inference harder to reach by accident, and when a suite has more than one variant the report adds a Comparisons section that pairs the variants case by case and reports the difference with its own interval and a significant flag, set when the interval excludes zero.

The first honest run of the delegation suite compared the two routers over three cases:

| variant   | judge   | metric    | delta   | 95% CI          | cases | significant |
| flash-low | quality | pass_rate | +33.3%  | [+0.0, +100.0]  | 3     | no          |
| flash-low | routing | pass_rate |  +0.0%  | [+0.0, +0.0]    | 3     | no          |

A thirty-three point improvement that the harness declines to call an improvement. The interval runs from zero to a hundred, which is the arithmetic saying that with three cases the data is consistent with no difference at all and with a total rout, and you cannot tell which. Three cases will almost never be significant, and the flag saying so is the harness reporting the size of the experiment you actually ran.

The flag has a matching failure at the other end, and it is worth seeing once so you never trust the word by itself. A later two-case run of the same suite reported a delta of +100.0% with a CI of [+100.0, +100.0] and significant: yes. Both cases moved the same way, so every bootstrap resample landed on the same number, so the interval collapsed onto a point that excludes zero. The flag reports what it says it reports, that this interval does not contain zero, and it makes no claim at all about whether two cases were enough to ask the question. Read the number of cases in the same glance as the flag.

Trials do a different job. Repeating a case measures whether the agent agrees with itself, and the report gives that back as trial_agreement. An agent that passes on trial one and fails on trial two has told you something a single run would have hidden: the behavior you are grading is not stable, and averaging it is describing a distribution rather than a capability.

agreement with people

A rubric judge is a model with an opinion, and before you let an opinion gate a deploy you should know how often it matches yours. The observatory’s answer is to make hand-labeling a first-class command rather than a good intention.

observatory label <run> --sample 20 walks a seeded sample of the run’s records and shows you the input, the expected value, and the output. It does not show you what the judge decided. You answer p or f, and your labels are written to the run directory and, with --persist, to a table where they outlive every future re-grading of the same turns. Then observatory calibrate <run> reports, per judge, the raw agreement and Cohen’s kappa, which is agreement corrected for how much of it you would expect from two grading processes that share the same pass rate and are otherwise unrelated. The reading is the standard one: slight, fair, moderate, substantial, almost perfect.

Kappa also splits the disagreements into the two kinds that carry different consequences. A lenient disagreement is the judge passing what a human failed, and it is the one that reaches users. A harsh disagreement is the judge failing what a human passed, and it costs you work rather than trust.

Labeling three records of the delegation run produced a result worth more than the effort it took. The rubric judge agreed with the human on all three. The programmatic routing judge, whose three checks are cheap and deterministic, reported agreement of 0.667 and a kappa of 0.0, “slight”. Its single disagreement was the Fibonacci case, and its rationale was three words long:

case_id     code-01           judge: passed        human: failed
note        "the code is missing"
rationale   "all checks passed"

Every check did pass. The router delegated to CodeExpert, the relay did not fall back, the run produced no error. The script was correct about everything it was capable of seeing, and structurally blind to the only thing that had gone wrong. This is the argument for stacking a rubric judge on top of a script judge, stated by the numbers rather than asserted: calibration is how you discover which of your graders cannot see the failure you care about.

Three labels are not a calibration, and the harness says so out loud rather than quietly using them: a gate configured with a kappa floor prints uncalibrated and warns when the judge has no labels behind it. Twenty is the sample the label command defaults to, and twenty is roughly the point where kappa stops swinging on a single answer. Label twenty records the first time a rubric decides anything, and label twenty more each time you change the rubric, because changing the criteria makes a new judge.

A judge is calibrated or it is a guess. Collect human labels with the verdict hidden, read kappa and the direction of the disagreements, set a floor per judge, and do not let a judge below its floor decide a deploy.

the ledger

Everything so far grades runs you started. The other half of observability records the runs you did not.

Every model call in melchizedek emits a span and every syndicate turn emits a root span, and with one environment variable set they land in a Supabase ledger of five tables. Three of them carry traffic. adk_turns holds one row per turn: the input and output, the agent that answered, the route the dispatcher chose and how it decided, whether it fell back, the error if there was one, input and output and thinking tokens, total latency split into time spent in the model and time spent in tools, and the tool calls with their full responses, so a later grader can check an answer against the evidence it was built from. adk_telemetry holds one row per span, which is the per-call view: provider, model, tokens, latency, agent. adk_payloads holds the assembled prompt and the raw response for a model call, written by policy (always on errors and fallbacks, otherwise a deterministic ten-percent sample, or everything) and expiring after thirty days, because full prompts are the most sensitive thing the system stores. The remaining two tables belong to the graders: adk_verdicts stores what a judge decided about a turn, and adk_labels stores what a person decided.

The load-bearing part is the identity on every row. A turn carries its session_id, its user_id, its A2A task_id and its ADK invocation_id, plus a config_hash of the agent definition that produced it and the engine version that ran it. That is what turns storage into something you can ask questions of.

The questions that identity makes cheap are the ones you would otherwise answer with guesswork. Which specialists are actually handling your traffic, and how often does the router fall back to a default. What each agent costs per day, per model, in tokens. Whether anyone has ever asked this thing before: adk_turns carries a generated full-text search column over input and output, so observatory search "PEG ratio" finds every time the question came up and what the agent said. And because a turn stores the invocation id that ADK also writes onto its session events, one join rebuilds a conversation with the machinery of every turn beside it: what was asked, what was answered, which tools ran, and what they returned.

The provenance columns do the version control. config_hash is a digest of the compiled agent definition, so grouping turns by it shows you exactly what changed when a prompt edit went out: average latency before and after, fallback rate before and after, on real traffic rather than on your test cases.

Two cautions, because this is user data. adk_turns stores user text and adk_payloads stores full prompts. Both are locked to the service role by the hardening SQL that ships beside the schema, sessions expire after seven days, payloads after thirty, and turns are kept indefinitely unless you bound them. Deciding your retention is part of switching this on, not a thing to get to later.

grading what production already did

Because a stored turn can be shaped into the same record a fresh run produces, the same judges work on both. A post-deploy suite has no variants and no trials and runs no inference for the agent under test: it loads the newest exchanges from the database, excludes its own eval traffic automatically, and grades them.

The shipped suite runs two judges over recent turns. A hygiene judge checks the rules every reply must obey: no emoji, no horizontal rules, no disclaimer boilerplate, no run of blank lines, under four thousand characters, no error. A rubric judge reads a sample and scores whether the reply answered the question that was asked, whether its specific claims are tied to tool results or sources, whether the register is right, and whether it hedged itself into uselessness. Judge independence applies here too, against the models recorded on the loaded turns rather than against a variant list.

Its first run over five live exchanges scored four of five on each judge, and the failure was one both judges caught. Here is how the reply began:

Running the required X sweeps now across official, opposing camps, and
reaction.Fetching official and opposing-camp voices.CLAIMS: 1. ...

That is the arbiter from module 2.08, and what is glued to the front of its answer is its own progress narration, concatenated without so much as a space. The content underneath was fine, which is why the two judges disagreed about how bad it was: the helpfulness judge gave it 5 for grounding and 1 for register, landing at 0.607 and quoting the narration as its first issue, while the hygiene judge failed it on length alone, at 4,786 characters against a 4,000 limit. No error was raised anywhere in the system, the user got their answer, and the defect would have kept shipping until somebody happened to read one reply. A grader read every one.

Grade that judge too, while you have it open. Two of the three issues it listed were the harness’s own context lines, the ROUTE: and TOOL CALLS: headers the rubric asks to be shown, quoted back as though the agent had written them. The finding was right and a third of its evidence was not, which is the argument for reading rationales rather than only scores, and for labeling a sample by hand before a rubric gates anything.

replay: the same question, the same data, a new answer

The strongest form of this is a back-test. A replay suite turns stored turns into its dataset: each case is a real user input, the answer that user actually received becomes the baseline, and the tool responses recorded on that turn are injected when your candidate runs. The tool returns the recording instead of calling the world, so last month’s question is not being answered with this morning’s prices. A pairwise judge then asks, for every turn and in both orders, whether the new answer beats the one production gave.

What that isolates is the change itself. Both answers were built from the same data, so a difference between them is your prompt edit or your model swap and nothing else. The run result reports how many tools served recordings and how many had none and ran live, so the report can say how much of the comparison was grounded.

Its limits belong in the same breath as its claim, and the shipped suite states them in its own header: a replay is single-turn, so the conversation history that preceded the stored turn is not reconstructed; a tool the candidate calls that was not called in the recorded turn runs live; and long-term memory is not replayed, so a syndicate that leans on recall runs without it. A back-test that clears the gate has proven the candidate better on single-turn, tool-matched traffic. That is a real claim and a bounded one.

the gate

A change becomes user-facing at the moment you publish it, so that is where the question belongs.

observatory gate <suite> reduces a run, a baseline and a set of thresholds to an exit code. The thresholds live in the suite: how far a pass rate may fall against the baseline, how far a mean score may fall, the maximum tolerable error rate, a p95 latency ceiling, absolute floors per judge, and a kappa floor per judge. The baseline is resolved in order: one you name, else the one you pinned, else the previous run of that suite. Pinning writes a self-contained JSON file you can commit, so continuous integration can gate against it without the original run directory.

Then the deploy step consults it. deploy_agent.ts --gate <suite> runs the gate before publishing and refuses on a non-zero exit unless you pass --force. Here is the real gate output for the lite router, against the pinned baseline:

  check           variant       judge    value   threshold  result
  --------------  ------------  -------  ------  ---------  ------
  min_kappa       *             quality  -           0.400  n/a     uncalibrated
  max_error_rate  lite-nothink  -        0.0%         0.0%  pass
  max_p95_ms      lite-nothink  -        3530        20000  pass
  min_pass        lite-nothink  quality  0.0%        60.0%  FAIL    pass rate 0.0%
  max_drop_pass   lite-nothink  quality  66.7%       10.0%  FAIL    66.7% -> 0.0%
  max_drop_pass   lite-nothink  routing  0.0%        10.0%  pass
  ...
  [!!] judge 'quality' has no calibration (no human labels) — kappa floor 0.4 is unverified

  [xx] GATE FAILED (2 failing of 10 checks)

Read the whole table, including the parts that are not verdicts. The min_pass line is the clean finding: the cheap router clears 0% of the quality bar against a 60% floor, and the deploy stops. The max_drop_pass line beneath it needs the context that the baseline was pinned from a run graded by a different judge model, which is why every verdict carries a judge_hash: a comparison across a changed judge is not a comparison, and the honest move after changing a rubric or a judge model is to re-pin. And the warning at the bottom is the harness declining to pretend: a kappa floor over an unlabeled judge is a rule nobody has checked.

The gate asks one question at the one moment it can act: did this get worse than the version already serving users? A deploy step that cannot answer it is publishing on faith.

one run, end to end

Here is the whole loop on a real change, a proposal to move the router down to the cheaper model, compressed from the runs above. The numbers, the failing output and the gate table are verbatim.

build the smallest version of this

You do not need this harness to grade your own agents, and the portable parts are the ones worth taking. Start here, in this order, on whatever stack you already have.

Write ten cases in a JSONL file. Not a hundred; ten, drawn from real questions your agent gets, including the two that make you nervous. Ten cases with expected values beat a benchmark you never run.

Add the script judge first. Anything with a right answer, a required field, a format rule, or a hard bound belongs to a program, and every check you can state as a rule is one the model never gets a vote on. When you find yourself unable to state a rule, that is the signal to add a rubric.

Write the rubric as criteria with weights and a threshold, give it to a model from a different provider than the one you are grading, and keep the script judge running beside it. Then label twenty of its verdicts by hand with the judge’s answer hidden, and compute your agreement. Until you have done that once, you have a grader whose accuracy is unknown.

Run trials, read the confidence interval before you believe a difference, and pin a baseline the day something ships. Then put the comparison in the deploy path, so the question gets asked when it can still change the outcome.

The materials are in two places. The suite files we worked from are yours to copy: eval-suite-template.md carries suite.toml, rubric.toml, the ten cases, and the custom calibration judge, verbatim and with their comments intact. Reading those comments is most of the design. And the ledger half runs today from a clone of the melchizedek-agents repo: apply db/telemetry.sql and then db/hardening.sql to a Supabase project, set TELEMETRY_SUPABASE=true, and every turn your syndicates take starts recording itself:

npm run syndicate:delegation -- "Write a Python function that returns the nth Fibonacci number."
npm run telemetry:stats

One honest gap: the Python observatory itself is not in the public mirror yet, so the python3 -m observatory commands above run in the private framework. The suite format is the teachable artifact, and it is a specification rather than an implementation (a dataset, variants with overrides, a stack of judges, a threshold, and a gate), which is why it transfers to whatever runner you build behind it.

When you next change an agent prompt, notice which question you reach for first. The one worth reaching for names four things: which cases you ran it on, who graded them, how far you trust that grader, and what the interval was.