The production ledger and the deploy gate

the ledger: production turns in an append-only record

On 2026-08-22, a turn answered by a production fact-checking team’s X researcher carried its scratchpad glued to the answer. Nobody ran an evaluation suite, and the runtime raised no error. What would have to have been written down, at the moment it happened, for an automated judge to grade that output afterward, or for you to determine how often the dispatcher falls back across production traffic?

The answer is one row per turn in an append-only ledger. At its smallest, each turn is a single row organized into four field groups over a provenance band: who initiated the call (the identity columns), what crossed the boundary (the exchange), which path the dispatcher chose (the decision), and what the execution consumed (the cost). The provenance band beneath them records the compiled graph hash (config_hash) and the engine version. The figure isolates twelve representative fields and three sample queries; a production row carries about forty columns across a wider schema.

The real row carries about forty columns, and the ledger organizes them across five relational tables. In the agent framework this module measures, every model invocation emits an OpenTelemetry span, and every agent-team execution creates a root span. When enabled via environment variables, these records write directly into an append-only Supabase ledger. Three tables manage operational traffic. The adk_turns table stores complete turn records: user inputs and agent outputs, invoked subagents, dispatcher routing decisions, fallback triggers, error traces, token counts (separated into prompt, completion, and reasoning tokens), latency breakdowns (isolating model inference time from tool execution time), and tool call parameters and payloads. Capturing raw tool payloads allows subsequent evaluators to verify whether model claims are supported by the underlying retrieved context. The adk_telemetry table provides span-level visibility across providers, model checkpoints, token utilization, latency, and agent bindings. The adk_payloads table captures complete assembled prompts and raw provider completions, retaining records under explicit retention policies: logging 100% of errors and fallbacks alongside a deterministic 10% sample of normal traffic, with an automated 30-day TTL. The remaining two tables, adk_verdicts and adk_labels, store automated judge scores and human annotations respectively.

Relational integrity across these tables is anchored by the identity columns. Every turn records its session_id, user_id, A2A task_id, and the runtime’s invocation_id, alongside a deterministic config_hash derived from the compiled agent definition and the runtime engine version.

With structured identity columns and provenance in place, operational questions reduce to straightforward SQL queries. You can analyze routing distributions across specialist agents, measure how frequently dispatchers fall back to default agents, calculate daily token expenditure per agent, and conduct full-text searches across historical turns. For instance, the harness’s search "PEG ratio" command queries the generated search column in adk_turns to locate every historical user query on that topic alongside the exact responses provided. Because each turn records the invocation_id shared across ADK session events, a single join reconstructs full multi-turn conversational transcripts alongside their corresponding tool inputs, intermediate states, and execution latencies.

The config_hash column provides exact version tracking. Because it represents a cryptographic hash of the compiled agent graph, grouping historical turns by config_hash enables direct A/B comparisons across prompt updates, measuring shifts in end-to-end latency and fallback frequencies on live production traffic rather than synthetic benchmarks.

Because adk_turns and adk_payloads store user text and raw prompts, you must restrict database access to protect user privacy. A hardening script locks these tables to the database service role. Production deployments enforce strict retention schedules: active session records expire after seven days, raw payloads expire after thirty days, and core turn records persist according to your organizational data governance policies. Establishing explicit retention windows is an essential prerequisite to recording production turns in the ledger.

grading what production already did

Because historical production turns adhere to the same schema as test suite runs, our standard evaluation judges can grade the ledger directly. A post-deployment evaluation suite requires no variant definitions, executes no trials, and triggers no new model inference for the agent under test. Instead, it queries the most recent production interactions, automatically filters out evaluation runs, and applies the configured graders.

Our default post-deployment suite pairs two distinct judges over recent production turns. A programmatic hygiene judge enforces structural constraints: verifying the absence of raw emoji characters, markdown horizontal rules, generic boilerplate disclaimers, or excessive blank line sequences, while asserting that total length remains under four thousand characters without unhandled errors. In parallel, a qualitative rubric judge samples interactions to assess whether user intent was satisfied, verify that factual assertions are grounded in tool outputs, validate conversational tone, and penalize excessive hedging. The principle of judge independence applies equally here, verifying that the judge model does not match the model recorded in the production turn metadata.

An initial evaluation across five production turns yielded passing marks on four, with both judges flagging the same failure. Consider the opening lines of the failing output:

Running the required X sweeps now across official, opposing camps, and
reaction.Fetching official and opposing-camp voices.CLAIMS: 1. ...

The ledger records this turn under XResearcher, the fact-checking team’s X collector: its route column names XResearcher, chosen by the classifier. The agent concatenated its internal scratchpad status updates directly onto the user-facing response without spacing. While the underlying factual claims were sound, the two judges evaluated the presentation failure through different lenses. The rubric judge awarded a 5 for factual grounding but a 1 for conversational register (yielding an overall score of 0.607 and citing the concatenated status text in its rationale). The programmatic hygiene judge failed the response strictly on character limits, as the concatenated text expanded the response to 4,786 characters against a 4,000 character maximum. The system experienced no runtime crash and delivered the required analysis, yet a noticeable formatting defect was introduced. Automated evaluation ensures such regressions are flagged immediately rather than waiting for user bug reports.

Critically, you should inspect the judge’s own diagnostic rationale. In this instance, two of the three issues cited by the llm judge were harness metadata markers (ROUTE: and TOOL CALLS:), which the judge mistakenly attributed to the agent. While the final failure verdict was correct, a portion of the supporting rationale was flawed. This highlights the importance of regularly reviewing judge rationales and calibrating automated rubrics against human annotations before establishing deployment gates.

replay: back-testing against recorded production data

The most rigorous offline evaluation method is back-testing through replaying production traffic. A replay suite ingests historical production turns as its evaluation dataset: each real-world user prompt is an input case, the historical production output is the comparative baseline, and recorded tool responses are injected directly into the candidate agent’s execution. When the candidate agent calls a tool that has a recording, the harness intercepts the call and injects the recorded response. It matches recordings by tool name in recorded order, returning the next recorded response for that name regardless of the arguments the candidate passed, and once the recordings for a name run out it repeats the last one. For the replayed tools, last month’s financial query is answered with the prices recorded at the time of the original request. A pairwise judge then compares the candidate’s new output against the original production baseline across both presentation orders.

Injecting recorded tool responses narrows the variables separating the candidate output from the historical baseline. For the tools replayed from storage, both answers are built from the same tool outputs. Other differences remain: the model samples differently on each run, the conversation history is not reconstructed, and the harness matches recordings by tool name without checking the arguments. The run result counts the tool calls served from recordings and the unmocked calls that ran live, so you can see how much of the back-test was grounded in recorded data.

Replay evaluations operate within boundaries documented in the suite header: each replay evaluates a single turn without its prior conversation history, and persistent long-term memory is not replayed. When the candidate calls a tool that was not called during the original production turn, this harness runs that call live and counts it; it has no setting that refuses the call instead. A safer default is to fail closed: a tool call with no recording returns an error to the candidate agent and is reported in the result, so a back-test cannot write, send or pay against a live system. Passing a replay evaluation gate demonstrates that a candidate agent produces better outputs on single-turn, tool-matched production traffic, a bounded assertion.

the gate: the deploy decision against a baseline

Because a prompt modification or architecture update impacts users the moment it is published, automated evaluation gates belong directly in your deployment pipeline.

The harness’s gate <suite> command evaluates an active run against a designated baseline and a matrix of predefined thresholds, returning a standard process exit code. These quality thresholds are declared within the suite configuration: allowable pass rate degradation relative to baseline, allowable average score drops, maximum tolerable error rates, p95 latency thresholds, absolute score floors per judge, and minimum Cohen’s kappa requirements. When a judge has human labels and its kappa falls below the required floor, the harness fails the gate and marks the judge’s verdicts as untrusted. When a judge has no human labels, the harness emits an uncalibrated warning, records the kappa check as n/a without counting it, and lets that judge’s own pass-rate and score checks decide the gate. In your own deployment pipeline, enforce the stricter rule: treat an uncalibrated judge as a blocking failure, so that an unverified heuristic cannot approve a release. The gate resolves its comparative baseline hierarchically: prioritizing an explicitly passed baseline file, falling back to a committed pinned baseline, or defaulting to the most recent run of that suite. Pinning a baseline generates a self-contained JSON artifact that can be committed to version control, allowing continuous integration pipelines to enforce regression checks without maintaining historical run directories.

In the private framework, the deployment script integrates this gate directly: it validates the active agent graph against the gate before publishing and aborts the deployment on any non-zero exit code unless the operator passes a force flag.

Below is an authentic gate evaluation comparing the lightweight router against its pinned baseline:

  check           variant       judge    value   threshold  result
  --------------  ------------  -------  ------  ---------  ------
  min_kappa       *             quality  -           0.400  n/a     uncalibrated
  max_error_rate  lite-nothink  -        0.0%         0.0%  pass
  max_p95_ms      lite-nothink  -        3530        20000  pass
  min_pass        lite-nothink  quality  0.0%        60.0%  FAIL    pass rate 0.0%
  max_drop_pass   lite-nothink  quality  66.7%       10.0%  FAIL    66.7% -> 0.0%
  max_drop_pass   lite-nothink  routing  0.0%        10.0%  pass
  ...
  [!!] judge 'quality' has no calibration (no human labels) — kappa floor 0.4 is unverified

  [xx] GATE FAILED (2 failing of 10 checks)

Examine the structural details of this gate report. The min_pass check reveals that the lightweight router achieved a 0.0% pass rate on the quality rubric against a required 60.0% floor, halting the deployment. The subsequent max_drop_pass failure highlights an important operational detail: the baseline run was graded using a different judge model. Every evaluation records a judge_hash in its summary, naming the judge’s rubric, model and thresholds, so you can see the mismatch. The gate itself does not compare these hashes: it computed the 66.7% to 0.0% drop across two different judges. A pass-rate delta between runs graded by different rubrics or judge models mixes the change in the judge with the change in the agent. Re-pinning the baseline under the new judge does not separate them, because the new baseline run generates new answers. To compare two judges or two rubric versions, you regrade one fixed set of stored outputs with both; a post-deploy suite does this naturally, since it grades stored turns without new inference for the agent. Finally, the warning at the bottom indicates that the quality judge lacks human calibration data, so the configured 0.4 kappa floor was never checked. The harness reports that check as n/a and does not count it as a failure. The gate failed and exited 1 because the quality judge’s own floor and drop checks failed; had they passed, the gate would have passed with the uncalibrated judge still deciding.

The gate asks one question at the one moment it can act: did this get worse than the version already serving users? A deploy step that cannot answer it is publishing on faith.

build the smallest version of this

You do not need a complex enterprise platform to implement rigorous agent evaluation. The foundational patterns translate directly to any development stack. We recommend implementing these components incrementally, in the following order:

1. Write ten test cases

   First, curate a focused dataset of ten representative test cases in a JSONL file. Rather than attempting to assemble hundreds of cases immediately, select ten realistic prompts from your production domain, ensuring you include challenging edge cases that previously caused failures. Ten concrete test cases with explicit expected values provide more immediate utility than large, unmonitored benchmarks.

   Done when a JSONL file holds ten domain prompts, including previous edge-case failures, each paired with an explicit expected value.

2. Check deterministic rules in code

   Next, implement programmatic judges. Any invariant involving exact output matching, required JSON properties, schema validation, or strict execution boundaries should be evaluated deterministically. Every constraint enforced via code eliminates subjective model variance. When an assertion cannot be captured via deterministic rules, introduce a qualitative rubric judge.

   Done when a script checks every rule code can test, leaving only what code cannot capture to a rubric judge.

3. Calibrate a judge model

   Define your qualitative rubric with explicit scoring criteria, relative weights, and an overall pass threshold. Execute the judge using a model checkpoint from a different provider family than your agent, running it alongside your deterministic scripts. Next, manually annotate an initial sample of twenty evaluation records with the judge’s score hidden, and calculate your inter-rater agreement and Cohen’s kappa. Twenty records are a first calibration: a single disagreement shifts agreement by five percentage points, so you label again as the suite grows and whenever the rubric changes. Until you have human labels, your automated judge remains an unvalidated heuristic.

   Done when you labelled twenty records by hand with the judge’s score hidden and computed agreement and Cohen’s kappa.

4. Compare trials against baseline in CI

   Finally, execute multiple trials per case to measure consistency, verify statistical significance via bootstrap confidence intervals over a dataset that covers the tasks your change touches before accepting performance improvements, and pin an immutable baseline upon deployment. Make the gate fail when a judge it relies on has no human labels, and check that rule once against a run with no calibration. Integrate this comparison directly into your CI/CD deployment pipeline so regressions halt releases automatically.

   Done when the CI comparison accepts only changes significantly different from the pinned baseline and stops the release on a regression.

5. Start a telemetry ledger

   Create an append-only turns table with the four field groups and the provenance band from the ledger figure, restrict it and any raw-payload table to the database service role, and set retention windows before the first production turn arrives. Have the runtime write one row per turn, run a single delegation, and read the row back with a query that groups by config_hash.

   Done when a query reads back the rows that one agent run wrote to your ledger tables.

The evaluation suite structure (datasets, variant overrides, judges, thresholds and gates) is a specification that you can implement within your own continuous integration workflow.

When you next adjust an agent prompt or update a model configuration, anchor your evaluation around four essential questions: which specific test cases were evaluated, which model graded the outputs, what is the calibrated human agreement of that judge, and did the performance delta achieve statistical significance?

That closes the work. Part 3 opens the instrument you have been grading: our lesson on token-by-token execution follows one token through the machine, and every failure class you measured here becomes arithmetic you can read. Open the syllabus to continue.