# Eval suite template: a dataset, the variants, the judges, the gate The verbatim suite files melchizedek's observatory runs against the starter pack's delegation router, with their own comments intact. Reference material for curriculum module 3.01 at https://lyceumagents.com/curriculum/the-grader-and-the-ledger/ Read this as a specification rather than as code for one runner. The portable idea is the shape: a dataset of real cases, variants that differ by the one thing you changed, a stack of judges from the most deterministic to the least, a threshold that turns a score into a decision, and a gate that turns the decision into an exit code your deploy step consults. That shape transfers to any language and any framework; the TOML below is one spelling of it. Three rules survive the translation: 1. The judge model may not be a model under test. Prefer a different provider, not merely a different checkpoint. 2. Keep a deterministic judge beside every rubric, so at least one grader has no opinion to be biased by. 3. Label twenty verdicts by hand, with the judge's answer hidden, before any rubric is allowed to gate anything. License: use it, adapt it, learn from it. --- ## observatory/suites/llm_judge/suite.toml (verbatim) ```toml # ============================================================ # Boilerplate 1 -- LLM AS JUDGE: a rubric becomes a schema # ============================================================ # # The syndicate under test is the starter pack's delegation router # (config/agents/examples/delegation.yaml): an orchestrator that hands # coding questions to CodeExpert, maths to MathExpert, and answers small # talk itself. There is no single right answer to grade against, so a # judge model scores each reply on a rubric (rubric.toml). The rubric's # criteria become a structured-output schema; the judge fills it in; the # harness turns the fields into a weighted score and a pass/fail. # # Two variants compare models AND thinking levels on the same cases, with # two trials each, so the report shows which configuration answers better # and how stable each one is. A programmatic judge stacks on top: whether # the orchestrator delegated to the specialist the case expected, and # whether its relay of the specialist's answer failed (relay fallback) -- # things a script knows better than a judge. # # Run: python3 -m observatory run llm_judge # Copy: python3 -m observatory new my_rubric_eval --kind llm name = "delegation-quality" kind = "pre_deploy" description = "Delegation router graded on correctness, completeness, clarity and specialist routing by a judge model." [dataset] file = "cases.jsonl" # {"id", "input", "expected": {"delegate": "CodeExpert" | "MathExpert" | null}, "tags"} [run] trials = 2 concurrency = 4 # ── Variants ───────────────────────────────────────────────────────────── [[variants]] label = "lite-nothink" syndicate = "delegation.yaml" [variants.overrides] model = "gemini-3.1-flash-lite" # every agent in the graph thinking = "off" [[variants]] label = "flash-low" syndicate = "delegation.yaml" [variants.overrides] model = "gemini-3.7-flash" thinking = "low" # models = { RouterAgent = "gemini-3.1-flash-lite" } # per-agent override beats `model` # instructionSuffix = "Always show your working." # prompt experiment on every agent # generateContentConfig = { maxOutputTokens = 2048 } # ── Judge 1: the rubric ────────────────────────────────────────────────── [[judges]] kind = "llm" label = "quality" model = "claude-sonnet-4-6" # NOT a model under test: a judge grading its own model prefers it (self-preference). # The harness refuses a judge that matches any variant's model unless allow_same_model = true. rubric = "rubric.toml" pass_threshold = 0.75 # weighted, normalised 0..1 # ── Judge 2: facts about the run a script knows best ───────────────────── [[judges]] kind = "programmatic" label = "routing" [[judges.checks]] type = "delegated_to" from_case = "expected.delegate" # null in the case means: expect NO delegation [[judges.checks]] type = "no_relay_fallback" [[judges.checks]] type = "no_error" # ── Judge 3: head-to-head against the first variant ────────────────────── # Each (case, trial) pair is judged twice with the answers swapped; a winner # must survive both orders, otherwise it is a tie and the disagreement # counts toward `position bias` in the report. [[judges]] kind = "pairwise" label = "head-to-head" model = "claude-sonnet-4-6" # a different provider from both variants (gemini-*) baseline = "lite-nothink" # the variant to beat; or "stored" for a replayed production answer criteria = "Prefer the answer that actually contains what was asked (the code, the steps), is correct, and is no longer than it needs to be." tie_passes = true # ── Gate: what "did it get worse" means for this suite ─────────────────── # `observatory gate llm_judge` runs the suite, compares it with the pinned # baseline (`observatory baseline llm_judge `; else the previous run), # and exits non-zero on a regression. `scripts/deploy_agent.ts --gate # llm_judge` refuses to publish on that exit code. [gate] max_drop_pass = 0.10 # a variant's pass rate may not fall more than 10 points vs the baseline max_error_rate = 0.0 max_p95_ms = 20000 require_significant = false # true: only fail a drop the candidate's CI supports [gate.min_pass] quality = 0.6 # absolute floor per judge [gate.min_kappa] quality = 0.4 # the rubric judge must agree with humans (kappa) before it can decide; unlabeled = warned ``` --- ## observatory/suites/llm_judge/rubric.toml (verbatim) ```toml # A rubric is a list of criteria. Each becomes one field of the judge's # structured output; scales normalise to 0..1 and weights combine them. # # scale = "1-5" integer range (any "lo-hi") # scale = "bool" true/false # scale = ["a", "b", "c"] enum -- first is best unless enum_scores says otherwise # # `extra_fields` are passthrough schema fields the judge must fill but that # are not scored (lists of issues, quotes, labels). They land in the # verdict's fields and in the HTML report. instruction = """ The agent is a small team: a router that either answers directly (for greetings and small talk) or hands the question to a CodeExpert or a MathExpert and relays their answer. Grade the FINAL answer the user received. A reply that refers to content the user cannot see ("see the steps above", "as shown in the code") when no such content is present is INCOMPLETE, whatever else it says. The TOOL CALLS line shows which specialist, if any, was consulted; CONTEXT shows which one the test expected. """ include_tool_trace = true include_route = false context_fields = ["expected.delegate"] [[criteria]] name = "correctness" description = "Is every factual, mathematical or technical claim in the answer right? Wrong numbers, wrong code, or wrong explanations pull this down sharply." scale = "1-5" weight = 3 [[criteria]] name = "completeness" description = "Does the reply actually contain what was asked for -- the working, the code, the example -- rather than describing or referring to it?" scale = "1-5" weight = 2 [[criteria]] name = "clarity" description = "Is the answer well organised and no longer than it needs to be?" scale = "1-5" weight = 1 [[criteria]] name = "right_specialist" description = "Was the question handled by the specialist the CONTEXT expected (or answered directly when none was expected)? Judge from the TOOL CALLS line." scale = "bool" weight = 1 [extra_fields] issues = { type = "ARRAY", items = { type = "STRING" }, description = "Specific defects found, quoting the answer. Empty when none." } ``` --- ## observatory/suites/llm_judge/cases.jsonl (verbatim) Ten cases. A case is an `input`, an optional `expected` value the judges read through dotted paths, and tags you can filter on. Start your own file at this size, drawn from questions your agent actually gets, and include the two that make you nervous. ```json {"id": "code-01", "input": "Write a Python function that returns the nth Fibonacci number, with a short docstring.", "expected": {"delegate": "CodeExpert"}, "tags": ["code"]} {"id": "math-01", "input": "What is the derivative of x^3 + 2x? Show the rule you used.", "expected": {"delegate": "MathExpert"}, "tags": ["math"]} {"id": "chat-01", "input": "Hi there! How's it going today?", "expected": {"delegate": null}, "tags": ["chat"]} {"id": "math-02", "input": "Solve for x: 3x + 7 = 22. Show the steps.", "expected": {"delegate": "MathExpert"}, "tags": ["math"]} {"id": "code-02", "input": "Explain the difference between a list and a tuple in Python, with one example of each.", "expected": {"delegate": "CodeExpert"}, "tags": ["code"]} {"id": "math-03", "input": "What is 17 * 23? Show the steps briefly.", "expected": {"delegate": "MathExpert"}, "tags": ["math"]} {"id": "code-03", "input": "Why does a JavaScript for...in loop over an array give me the indices as strings? Show the idiomatic alternative.", "expected": {"delegate": "CodeExpert"}, "tags": ["code"]} {"id": "math-04", "input": "What is the area of a circle with radius 5? Give the exact value and a decimal approximation.", "expected": {"delegate": "MathExpert"}, "tags": ["math"]} {"id": "chat-02", "input": "Thanks, that's all I needed.", "expected": {"delegate": null}, "tags": ["chat"]} {"id": "code-04", "input": "Write a SQL query that returns the three most recent orders per customer from an orders table (customer_id, order_id, created_at).", "expected": {"delegate": "CodeExpert"}, "tags": ["code"]} ``` --- ## observatory/suites/programmatic/judge.py (verbatim) The escape hatch: when a check does not fit a configuration line, a script next to the suite exports one function. This one scores calibration for a boolean fact checker, where confident and wrong fails. ```python """ A custom programmatic judge -- the escape hatch when a check does not fit a config line. Contract: judge(case, result, context) -> dict case the dataset row as a dict: id, turns, expected, tags, meta ... result the RunResult from the engine: output, structured, toolCalls, route, tokens, latencyMs, error ... (lib/evals/types.ts) context {"suite": name, "record": the whole record, "fields": the outcomes of the config checks that ran before this script} Return any of: passed bool ANDed with the config checks score float replaces the checks' pass fraction when given fields dict merged into the verdict's fields (shown in reports) rationale str appended to the verdict's rationale This one scores CALIBRATION: a fact checker that is confident and wrong is worse than one that is unsure and wrong. The score is the Brier-style complement |confidence - correct|, so a confident right answer scores 1.0 and a confident wrong answer 0.0. """ from __future__ import annotations from typing import Any def judge(case: dict[str, Any], result: dict[str, Any], context: dict[str, Any]) -> dict[str, Any]: data = result.get("structured") or {} if not isinstance(data, dict) or "verdict" not in data: return {"passed": False, "score": 0.0, "fields": {"calibration": None}, "rationale": "no structured verdict"} expected = case.get("expected") verdict = data.get("verdict") try: confidence = float(data.get("confidence", 0.5)) except (TypeError, ValueError): confidence = 0.5 confidence = max(0.0, min(1.0, confidence)) correct = (verdict is expected) if isinstance(verdict, bool) else (str(verdict).lower() == str(expected).lower()) calibration = confidence if correct else 1.0 - confidence overconfident = (not correct) and confidence >= 0.9 return { "passed": not overconfident, "score": round(calibration, 3), "fields": {"calibration": round(calibration, 3), "confidence": confidence, "overconfident_wrong": overconfident}, "rationale": "confident and wrong" if overconfident else "", } ``` --- ## The gate, in one paragraph `gate` reduces a run, a baseline and the thresholds above to an exit code: `0` pass, `1` fail, `2` configuration error. The baseline resolves in order: one you name, else the one you pinned, else the previous run of that suite. Pinning writes a self-contained JSON file you can commit, so continuous integration gates without needing the original run directory. The publish step then consults it and refuses on a non-zero exit. Every verdict carries a `judge_hash` over the rubric, the model and the thresholds, because a comparison across a changed judge is not a comparison: change a rubric, re-pin the baseline.