Generate an image from a spec, then audit it blind

A generated image appears complete whether or not it satisfies its prompt constraints. A language model generates statistically probable continuations without maintaining an internal certainty metric, and a diffusion model rendering pixels carries no such metric either. A generated scene depicting two vases when three were requested is statistically coherent; no artifact or pixel flag marks the missing object. The verification check must therefore originate outside the generator, executed by an evaluation component that receives no prior conditioning regarding what to expect.

You will run that verification workflow on an image using the image production syndicate configuration in config/agents/examples/image_production.yaml. The architecture coordinates three distinct roles: a designer agent that writes a structured visual specification and halts before rendering until you supply an explicit approval, an observer agent whose tool receives the image file path and no contextual metadata, and an auditor agent that reads the specification and the inventory without viewing the image. We will draft the contract, approve the schema, trigger generation, inspect the resulting reports, compute the conformance score by hand from the individual field verdicts, and apply the structural fix required when a field returns as unverifiable. By completing this project, you will produce an image, verify its conformance using an independent numerical score, and establish an audit boundary that prevents expectation priming.

three minds, and what each one is allowed to hold

Three amphorae were asked for. The generator writes pixels and holds no count of what it drew. A grader who has read the spec looks at two vases and sees three, and so do you once you have read it. Who is allowed to hold what? The system resolves that design problem by dividing the labor across three bounded roles. The designer holds the request and the spec and never grades. The observer’s tool takes one argument, a file path, so the spec has no field to travel through. The auditor, executed as SpecAuditor, reads the spec and the inventory as text and has no vision tool. This division prevents expectation priming, where reading the spec beforehand leads an observer to report items that the generator failed to draw.

One statement about this page’s figure and numbers, made once. The figure draws the three side by side; they run in sequence: designer to observer to auditor and back. The audit in step four shows three of the eight fields from the course’s run of this request, and its 57 is 4 ÷ 7 measured from that run.

step one: run the syndicate and read its two phases

Execute the syndicate:

npm run syndicate:image

The image model requires a Gemini API key on a paid tier. The free tier returns an API error at the generation step; however, the two-phase workflow executes regardless, allowing you to practice drafting and validating specifications without an active paid key. Before submitting a prompt, open config/agents/examples/image_production.yaml and examine the system instructions assigned to the designer agent. The system prompt defines three distinct execution phases, and explicitly restricts tool usage during the initial phase:

    ### PHASE 1: DESIGN & CONFIRMATION (First Turn or Payload Tweaking)
    When the user first requests an image, or when they request changes to an
    existing design, you must **NEVER** call the 'generate_image' tool.

The designer outputs a structured specification and a confirmation request on its first turn, never an image. The syndicate enforces a deterministic sequence rather than a single-turn generation loop. The first stage requires a human operator to verify the specification fields before the system executes an expensive generation API call.

step two: write a contract of fields a verdict can check

Request an arrangement containing countable elements. The reference run for this project requested three phosphor-green amphorae on a stone shelf against a dark virtual terminal space. The discrete count of three provides an objective validation target: a numeric count is verifiable, whereas a stylistic adjective is subjective. Define a subject with a specific count, a distinct palette, a background described in a concise phrase, an explicit medium, and a target aspect ratio. Submit your request into the terminal session and inspect the returned output. The agent returns a structured JSON payload containing meta, visual_elements, technical_specifications, composition, generation_parameters, and a master prompt string.

Before approving the spec, evaluate it by separating fields that state empirical criteria from those that express subjective tone. A field such as primary_focus: three amphorae, evenly spaced provides an observable target that an inventory can corroborate. A field such as mood: contemplative cannot be verified by an inspection parser. Continue prompting the designer in the conversation loop until at least five fields contain countable or empirically verifiable values, removing qualitative adjectives that mimic functional constraints.

Write the contract before anything renders. A field omitted from the specification cannot be evaluated as a match or a mismatch. If a visual requirement cannot be stated in concrete terms that an independent observer can verify, the constraint remains uncommitted, and the rendering step will drift unchecked.

Approve the design explicitly by entering one of the accepted confirmation phrases recognized by the prompt instructions, such as build it.

step three: generate once, and find the file

Upon confirmation, phase two executes the generate_image tool using the validated prompt string, style definition, aspect ratio, and color palette. The tool writes the generated image to the outputs/ directory inside the repository and returns the relative file path. The designer reports this path without providing descriptive commentary. The system prompt explicitly prohibits the designer from narrating the output. When a generating agent evaluates its own artifacts, it summarizes its latent generation targets rather than the pixels written to disk. Open the generated file directly in an image viewer to confirm write success, then close the window. Having read the specification, your own visual evaluation is already primed: human observers conditioned on a prompt tend to project specified details onto ungrounded images.

step four: read the blind inventory, then the audit

Instruct the designer to review the generated output and observe the execution trace. The designer first invokes inspect_image, passing only the file path. Because the tool schema accepts exactly one argument, the spec cannot leak into the observer’s context window. The returned inventory provides an unconditioned list: identified subjects with exact counts, dominant color values, background geometry, illumination vectors, visible rendering artifacts, and explicit notations of perceptual uncertainty. The observer emits no evaluative judgment, because its context contains no spec against which to compare.

The designer then routes execution to SpecAuditor, providing two text inputs verbatim: the spec and the observer’s inventory. An output schema constrains the auditor to a leaf agent; it operates without sub-agents or tool access, and it receives no image data. The auditor returns a structured JSON evaluation:

{
  "fields": [
    { "field": "subject_matter.primary_focus", "expected": "three amphorae, evenly spaced",
      "observed": "three amphorae on a horizontal shelf, evenly spaced", "verdict": "match" },
    { "field": "technical_specifications.medium", "expected": "CRT vector-graphics render",
      "observed": "photograph of a physical monitor displaying line art", "verdict": "mismatch" },
    { "field": "generation_parameters.aspect_ratio", "expected": "16:9",
      "observed": "(not reported)", "verdict": "unverifiable" }
  ],
  "conformance": 57,
  "recommendation": "regenerate — reinforce technical_specifications.medium"
}

The payload allocates one entry per field, pairing the expected value with the observed visual element and assigning one of three verdicts: match, mismatch, or unverifiable. The conformance metric reflects the percentage of checkable fields that matched, while the recommendation identifies a single target field to reinforce. The trace is from the course’s run: the image model rendered a photograph of a physical CRT monitor instead of vector graphics, and the observer recorded the monitor casing, knobs, and cables.

step five: compute the score yourself

Calculate the conformance metric directly from your field outputs. Subtract the number of fields labeled unverifiable from the total number of evaluated fields to determine the checkable count. Divide the count of match verdicts by this checkable total:

conformance = matches ÷ (fields - unverifiable)

the course's run:  4 ÷ (8 - 1) = 57%

The automated auditor output calculated 57 percent for that test run. Verify the corresponding arithmetic in your own output trace. If the reported conformance score diverges from the quotient of its individual field lines, the discrepancy indicates a prompt drift or parsing error within the auditor instructions, detectable only by manual verification.

Next, isolate the root cause for each recorded mismatch. A mismatch originates from one of three failure modes, distinguishable by comparing the expected and observed lines quoted in the verdict record:

1. Observation failure: If the image contains three amphorae but the observer inventory reports two, the observer failed its extraction step. Blind evaluation removes expectation priming, but it does not eliminate raw vision-model miscounting.
2. Evaluation failure: If the observer inventory recorded the scene accurately but the auditor classified the entry as a mismatch, the text evaluation logic failed to compute semantic equivalence.
3. Generation failure: If both text records are accurate yet contradict each other, the image generator failed to satisfy the spec. This failure mode provides the signal for refinement in step seven.

Separating these three failure modes across distinct agents makes the pipeline observable and debuggable.

A verdict traces to two quotable lines. Every match or mismatch pairs an expected specification entry with an observed inventory line. Reading that pair indicates whether the observation failed, the evaluation logic failed, or the generated pixel output failed.

step six: close an unverifiable field the right way

Your audit report will typically contain at least one unverifiable entry. The aspect ratio field often triggers this verdict because the default inventory protocol catalogs scene contents without reading pixel geometry. Developers frequently attempt to resolve this by passing the image directly to the auditor agent. That approach compromises the architecture: an auditor exposed to image pixels becomes vulnerable to visual expectation priming. The durable engineering resolution requires expanding the observer’s inspection protocol.

This protocol is implemented in the tool source at lib/tools/inspectImageTool.ts, with an exportable reference documented in the visual inventory protocol. Update the protocol instructions to require that the observer extract the image pixel dimensions and calculate their reduced aspect ratio. Re-run the review against the existing image file, and observe the unverifiable verdict resolve deterministically into either match or mismatch. An unverifiable verdict signals an incomplete observation contract, requiring an expanded protocol rather than a bypass of the evaluation gate.

step seven: change one field and regenerate

The auditor recommendation isolates a single priority field. Return to phase one, reinforce that specific constraint within the spec, provide approval, generate a new image, run the evaluation review, and recalculate the score. Systematic prompt tuning applies equally to image generation pipelines: adjust one variable, hold the remaining configuration constant, and measure the delta in the conformance metric. If the conformance score improves, the modification resolved the constraint. If the score remains unchanged, the adjusted field failed to alter the generation parameters. Because the underlying image model processes only the compiled prompt string, you should inspect that string to confirm the parameter mapped correctly.

equip yourself with the materials

1. The syndicate, verbatim: image-production-syndicate.md.
2. The observer’s protocol and the audit schema, portable to any pipeline: visual-inventory-protocol.md.
3. A spec template with the checkable fields marked, the run sheet, and the scoring arithmetic as a fill-in: image-spec-audit-kit.md.
4. The module: multi-modal agents.

apply it to your own ecosystem

Production teams regularly approve multimodal artifacts through unconstrained manual inspection: ecommerce product photos checked against catalog listings, presentation slides checked against brand guidelines, analytical charts checked against source data tables, or interface screenshots evaluated against design mockups. In each case, an unwritten specification is reviewed by a human operator who already anticipates what the asset should depict.

Select one production artifact from your domain and implement this architectural separation:

1. Formulate the spec as explicit fields. A catalog listing requires an exact product count, a specified background hex value, and an explicit camera angle. A chart requires an exact title string, labeled axes, a defined series count, and an explicit date range. An interface screenshot requires a deterministic layout hierarchy, exact button copy, and defined design tokens. Eliminate subjective qualitative adjectives.
2. Execute the observer blind. Pass the asset file path alone to an inspection tool equipped with a protocol that extracts counts, geometric relationships, and readable text strings. The inspect_image tool implementation and download protocol apply across any standard image format.
3. Perform the audit in text. Supply the spec and the generated inventory to the auditor, enforcing the three-state verdict schema: match, mismatch, or unverifiable.
4. Score ten artifacts that your team previously approved through manual visual inspection, and record every field marked unverifiable.

That list highlights the unspoken assumptions that escaped explicit validation. Expand the observation protocol to capture those parameters, and evaluate the subsequent batch of artifacts against verifiable, programmatic contracts.