Build a truth-arbitration bot

When you ask a language model what happened in the world this week, it generates text in the same fluent register whether the event exists in its training weights or not. The physical mechanism is straightforward: the network produces statistically probable token continuations across an activation vector and holds no internal certainty score to verify factual grounding. A paragraph describing a verified negotiation and a paragraph describing an invented event carry identical grammatical confidence. The truth-arbitration bot you build here isolates these tasks: the agent writing the final answer has no tools to look up records directly, forcing every factual assertion to originate in an external report compiled by dedicated collector agents.

We implement this architecture in Augustin, a three-agent syndicate: two collectors that search and report what they find, and one arbiter that judges from those reports alone. You build it on one model, Gemini, running all three agents, and two search tools: x_api_search reaches X directly through its own API, and web_search/web_extract reach the open web. You build it by executing the decisions embedded in its configuration file in order. Each decision is forced by the one before it: the information channels determine the required tools, the tools dictate which keys you provision, and two provider limits shape the topology and the prompt. By the end of this project, you will run the arbiter against a live claim from your own feed, read the corroboration labels it assigns, and implement the same gate for your own domain.

the build is a chain, and each link is forced by the last

One statement about this page’s figure and numbers, made once. The figure draws the seven decisions as a straight chain. In the build, a limit found at step five sends you back to step three to change a model, which changes the key at step four. The labels read in step five come from your own runs, and the one dated fact on the page (that web_extract landed upstream on 2026-09-02, package 0.9.6) is measured.

step one: decide which channels the world speaks through

An automated fact-checker operates on two distinct evidence modalities that present differing reliability characteristics despite both arriving as tokenized text. The first modality is the live claim stream: immediate discourse, source attribution, and competing public framings. The second modality is the documented record: wire reports, primary regulatory filings, official transcripts, and verified datasets. The live stream moves rapidly without verification; the documented record moves deliberately with identifiable institutional accountability. Because neither stream independently confirms what is verifiably established, our syndicate reads both sources and rejects any factual assertion not explicitly present in at least one channel.

Interrogating the documented record requires two separate computational operations. A search tool queries an index and returns truncated snippet text selected by the provider. Reading a full primary source requires an independent HTTP fetch, implemented here as web_extract: the agent passes a specific URL, and the tool fetches the document and extracts normalized text content, processing up to five pages per call. Search yields index snippets, whereas web_extract yields the complete document body; your web collector requires both capabilities.

Record your target channels explicitly before modifying your configuration. In this implementation covering global events, our channels are X posts and the open web. In your own operational domain, these channels will reflect your internal data sources, while the downstream architectural dependencies remain identical.

step two: the tool table decides the model

When you declare x_search — xAI’s own X-search mechanism — on a Gemini collector, the run completes, the report is empty, and nothing in the execution trace explains the missing records. A declared tool that the provider does not carry produces no error at all, only an empty sweep, so the trace looks like a quiet day on X rather than a misconfiguration.

Every entry declared under an agent’s tools: list is a string key that lib/toolRegistry.ts resolves to a concrete runtime implementation. Some of those declarations are sentinels: names that signal a provider-native search mechanism rather than engine code, and a sentinel declared against a provider that does not carry that mechanism returns the silent no-op above. x_api_search and web_extract are the opposite kind: engine code that calls an external API directly from the server, so they fetch identically no matter which model is reading their output. The tool table maps how each string key resolves across the providers this system uses.

This mapping dictates the model choice for each role in the pipeline.

The X collector reaches X through x_api_search, a client-side tool, so it runs on any provider. Nothing about the X channel forces a particular model any more — the tool calls X’s own search API from the server, not from a provider’s agentic tool-calling surface. (x_search, xAI’s own sentinel for the same job, still exists and still forces a grok-* model — it is the cautionary case above, not the reference configuration’s choice.) The reference configuration deploys gemini-3.8-flash, the same model as the other two agents.

The web collector requires web_search and web_extract. Any supported cloud provider meets this requirement. The reference configuration deploys gemini-3.8-flash, which the runtime environment recommends for tool calling, because gemini-2.5-flash returns a 400 error status against this engine’s tool-call schema.

The arbiter requires no tools. Its architectural requirement centers on context adherence: it must maintain a complex system instruction and enforce a rigid output schema while evaluating two extensive incoming reports. Allocate your most reliable schema-adhering model to this role. The reference configuration assigns gemini-3.8-flash with reasoning enabled.

The tool decides the model. Identify the specific tools an agent must invoke, determine which model prefixes support those interfaces in the tool table, and select the smallest model that satisfies the operational requirements. Selecting a model before defining tool requirements leads directly to configuration failures where agents declare capabilities they cannot physically execute — and a client-side tool like x_api_search removes that constraint for the role it serves, rather than pinning it to one more prefix.

step three: each key unlocks one prefix

API credentials must reside exclusively in your local .env file, never inside system prompts or interaction logs. Initialize your environment configuration from the provided template, populating only the keys required by your selected model prefixes:

cp .env.example .env

Our baseline deployment requires two credentials, and only one of them is a model key. GOOGLE_GENAI_API_KEY, available through the Google AI Studio free tier, powers all three agents. X_BEARER_TOKEN, a read-only app token from the X developer portal, authorizes x_api_search — it unlocks the tool, not a model, so it costs nothing in any provider’s account. Validate network connectivity and authentication before invoking full multi-agent sweeps:

npm run demo:models

This verification script dispatches a single test prompt to a lightweight agent on every provider with an active key in your environment. It logs the generated response alongside token counts and latency metrics, bypassing unconfigured providers. A provider that outputs no data indicates a missing or malformed key, isolating credential failures prior to execution.

step four: two provider limits shape the topology and the prompt

Two constraints decide the shape of this build. The first decides how many agents there are. The second decides what every message to the syndicate carries.

Gemini grounds effectively or enforces a schema, but rarely achieves both in a single inference call. Early iterations of our production pipeline demonstrated that when a single Gemini call attempts to execute search grounding while enforcing a strict structured schema, the network defaults to generating completions directly from static weights. Decoupling these responsibilities across two discrete agents resolves this failure mode. The collector executes searches and outputs an unstructured plain-text report, while the arbiter enforces the rigid two-part schema without invoking external tools.

x_api_search is a boolean keyword search over X’s last seven days, one page per call. It is not semantic: the words you send are the words it matches, so a query needs several formulations — official-account names, opposing camps’ own phrasing, the reaction terms people actually use — rather than one well-worded attempt. Because the window is fixed at seven days regardless of what the prompt asks for, temporal scoping has to be established directly within the prompt instead: every user submission to Augustin is prefixed with [System Context: Current Date is …], instructing both collectors to interpret “today” relative to that timestamp. (If you swap in the optional grok-based X channel from the variants download, the trade flips: x_search’s own date bounds and handle lists come from the environment, its web search accepts domain filters but ignores date boundaries the same way, and reasoning stays continuously active at medium effort with no option to disable it — a latency cost worth accounting for in that variant’s execution budget.)

A tool name the registry does not carry is dropped. Every string declared in an agent’s tools: array is validated against lib/toolRegistry.ts, a lookup map linking configuration keys to runtime tool instances. Unrecognized names trigger a warning and are omitted from the agent’s schema. Inspect the registry directly before relying on a tool declaration. The upstream repository integrated web_extract on 2026-09-02 (package version 0.9.6); earlier checkouts restrict Augustin’s web collector to search snippets alone. Exposing an existing engine tool to agents requires adding one import and a single map entry to that file. Maintain strict data-instruction boundaries for all retrieved content: incoming document text constitutes untrusted data to analyze, never executable instructions.

step five: run the arbiter and read its labels

The reference implementation incorporates all architectural decisions outlined above. Execute the syndicate against a live claim from your own feeds, supplying the temporal context marker that grounds both collectors:

npm run syndicate:augustin -- "[System Context: Current Date is 2026-09-02] What is actually known about <the claim in your feed> right now?"

Evaluate the generated response against its output contract before examining the factual claims. The output begins with a concise introductory summary, followed by a structured listing containing exactly one factual assertion per line, concluding with source attribution in parentheses. Next, audit the corroboration labels. An entry bearing an unqualified source attribution passed the corroboration gate, indicating agreement across both channels or verification via an authoritative primary source. An entry labeled unverified relies on a single uncorroborated source. The tags disputed and contradicted by indicate points of direct conflict between channels, recording the contradiction itself as the primary finding. If a collector yields an empty sweep, the system must report that the external record is thin, without speculating on missing data.

The corroboration gate: Claims confirmed across multiple channels or substantiated by primary documentation qualify as bare facts. Attested statements lacking secondary confirmation remain attributed claims. Single-source statements are tagged as unverified. Conflicting statements are reported directly as contradictions.

Execute the identical query across three independent runs. Because stochastic token sampling introduces variance in query generation and source retrieval, compare the assigned corroboration labels across runs rather than analyzing phrasing alone. If a claim appears as an established fact in one run but is tagged as unverified in the next, the underlying empirical evidence is thin.

audit the answer with the three questions

Across every architectural variant, diagnose incorrect or degraded completions by auditing three operational questions:

First, was the necessary evidence present in the context window? Inspect the raw collector reports to confirm whether the specific claim was retrieved, and verify that the system context date line was passed into the prompt array.

Second, did the completion present ungrounded assertions with high stylistic confidence? Grammatical confidence is merely a property of the token distribution; verify every claim against the citations enclosed in parentheses.

Third, where was the verification check applied, and did the autoregressive generation bypass it? In this design, the corroboration gate is an instruction within the arbiter’s prompt, evaluated token by token during text generation. A prompt instruction alters generation probabilities without providing deterministic execution guarantees. Your manual audit, reconciling final labels directly against the raw collector reports, produces the ground truth measurement that detects gate failures.

equip yourself with the materials

1. The shipped syndicate, line for line: augustin-arbiter.md.
2. An appendix for later, when you want to run this on other providers: three model variants, the registry patch, and a three-draw run sheet: truth-arbiter-variants.md.
3. The module that studies the design: multi-agent fact-checking.

apply it to your own ecosystem

Every technical domain maintains an active claim stream alongside an authoritative documented record, though the underlying data sources vary widely:

• In product engineering, the claim stream consists of customer support tickets and community discussions, while the documented record consists of production release notes and issue trackers.
• In scientific research, the claim stream consists of preprint servers and academic social feeds, while the documented record consists of peer-reviewed literature and experimental repositories.
• In personal operations, the claim stream consists of messaging threads, while the documented record consists of shared calendars and payment receipts.

Construct an arbitration bot for one of these environments:

1. Define your two distinct channels and implement the tool interface required to access each. A discussion forum offering an HTTP API requires a custom tool integration, following the architectural pattern demonstrated in the financial analyst project. An enterprise wiki connects via an MCP server. A local directory of unstructured documents maps to collections_search on Grok or web_extract targeting locally served endpoints.
2. Consult the tool table to select the appropriate model prefix for each collector’s toolset.
3. Maintain the arbiter as a tool-free agent. Adapt the corroboration gate’s taxonomy to your domain: reported by one customer, reproduced by support, contradicted by the changelog. Define deterministic promotion criteria for each state: a reported bug becomes an established fact only when correlated with an open tracking issue or a reproduction commit, remaining tagged as reported by one customer until that criterion is satisfied.
4. Execute your pipeline against an unverified operational claim across three consecutive runs, tabulating the corroboration labels assigned across trials.

Deliver the resulting arbitration report to the domain expert who manually evaluates these claims, and determine which assertions their manual workflow would have classified incorrectly. That empirical baseline provides the true evaluation metric for this system.