Building memory systems for agents
by the end you can:
- Distinguish session state from long-term semantic memory
- Trace the lifecycle: conversation → distillation → embedding → recall
- Decide what deserves to be remembered: distinguish facts and trends from noise
- State the ethics: fabricated memory is harm; a memory store is a PII store
The language model operates as a fundamental amnesiac. Its weights remain frozen during inference, its context window constantly slides forward, and every session is born entirely blank. Memory, therefore, is not an inherent feature of the model; it is an architectural structure engineered around the model. Building this structure requires solving two distinct problems: judgment (determining exactly what deserves retention) and doctrine (establishing how recalled facts may be trusted). This module examines Melchizedek’s most sophisticated memory implementation: the Patient Advocate, a system designed to maintain a persistent, reliable record across multiple patient sessions.
the lifecycle of a fact
An effective memory system utilizes two distinct stores with two specific jobs.
Session persistence (adk_sessions) maintains the raw, running transcript. This ensures the conversation survives a system restart verbatim. It is heavy, detailed, and strictly short-lived.
Long-term memory (adk_memory_facts) preserves the underlying meaning of the session. When an interaction concludes, a distillation mechanism compresses the transcript into discrete, structured records. Each record is a single line carrying five load-bearing fields alongside its text: the date it is about (always absolute — “last Tuesday” is stored as its calendar date), the source who asserted it (the patient, a named clinician, a document), its exact values with units, its status (active or historical), and one to five index keys — the names, medications, and metrics the record is filed under. The system embeds the entire line as a 768-dimension vector within pgvector, and simultaneously parses those fields into queryable columns. (The how-agents-remember post details these specific tiers.)
Observe a single fact navigating this entire arc:
Structure earns its keep at recall time. A search runs on three axes at once: semantic similarity (the embedded query against the embedded records), index keys (a question naming “metoprolol” finds the records filed under that key even when the phrasing diverges), and dates (a question about “the March labs” boosts records dated to that month). Pure similarity search finds what a query resembles; the structured axes find what it references.
Structure also lets the record evolve. When the user corrects something — a dose raised, an appointment moved — the distiller emits a correction that names the outdated claim, and the store retires the old record: marked superseded, linked to its replacement, kept as inspectable history. Unresolved contradictions receive the opposite treatment: both records are stored alongside a note naming the conflict, because a memory system that silently picks a side is falsifying the chart.
The system initiates recall through two distinct channels based on initiative. preload_memory functions ambiently: before the model processes a new message, the system automatically searches the store and injects the highest relevance hits directly into the context window. Conversely, load_memory functions deliberately: it is a specific tool the agent explicitly calls when the preloaded context proves insufficient (e.g., “Retrieve the specific lab values from March”). Both channels operate simultaneously within the specimen’s YAML configuration:
memory_system: "long-term"
orchestrator:
tools:
- "preload_memory" # ambient: relevant facts injected every turn
- "load_memory" # deliberate: explicit search into the store(For a secondary application of this pipeline, examine the repository’s memory-syndicate example—it utilizes identical mechanics within a strategic planning context.)
the distillation judgment
While raw storage remains inexpensive, cognitive recall is highly costly. Every retained fact competes fiercely for the similarity search’s top slots and consumes the context window’s strict token budget. Therefore, the distiller’s primary and most vital function is refusal.
The system actively preserves: dated clinical facts regarding the subject, explicitly stated preferences, core identities, and critical trend material. An isolated creatinine level of 1.4 mg/dL holds one meaning; understanding it as the third consecutive rise provides a completely different clinical picture. The system actively discards: pleasantries, logistical noise, and any information the model can easily re-derive from its general training data.
Judgment extends beyond selection into formation. A fact that survives the sieve must be written to survive time: its values keep their exact units (never rounded, never restated from the model’s own knowledge), its dates become absolute at write time, and its source is named — because “hold the dose below 55 bpm” means something entirely different coming from this patient’s cardiologist’s discharge orders than coming from nowhere. A stored number without provenance is a number the agent will later repeat with unearned authority. Alongside the discrete records, one narrative episode summary per session preserves the connective tissue the individual facts cannot: what was discussed, decided, and left open.
The definitive test for retention is simple: define the exact future question this specific fact answers. If no clear question exists, the system refuses storage.
Hold the sieve yourself. Evaluate these eight statements and make the call:
the doctrine
A memory system operating without a written doctrine is simply a liability equipped with a search index. Consequently, the Patient Advocate’s instruction dedicates an entire block specifically to governing how memory may be utilized. Here are the load-bearing requirements, verbatim:
<memory_doctrine>
This record is your ground truth. Treat it the way a great nurse treats a chart:
- Anchor every answer to THIS patient's specific history. Never provide generic advice when their record supports precise, tailored advice.
- EVERY FACT HAS PROVENANCE: each record carries the date it is about, who asserted it, its exact values with units, and whether it is still current. When you use a remembered fact, cite its provenance — "per the discharge orders of 2026-07-03".
- When the user shares a new clinical fact, acknowledge explicitly that you are adding it to their record — restating the date, the exact value with its units, and who said it.
- CORRECTIONS SUPERSEDE: the old fact becomes history — never quote a superseded value as current.
- CONTRADICTIONS SURFACE: name the conflict out loud and ask which is right. A quietly merged contradiction is a corrupted chart.
- When the record is silent on something material, state that silence clearly and ask. Never invent a remembered fact. A fabricated memory in a medical context constitutes a direct harm, not a convenience.
- Track TRENDS, not just isolated values.
</memory_doctrine>The doctrine pairs with a safety boundary that memory makes newly dangerous: numbers belong to their clinician. A numeric action threshold — a hold-the-dose cutoff, a when-to-call vital sign — may appear in the agent’s output only as a quote from this patient’s record, attributed to the clinician or document that set it, with its date. An agent that supplies a “typical” cutoff from general medical knowledge has issued a prescription in disguise; the correctly designed agent says the record holds no such instruction and hands the patient the exact question that gets one.
The silent-record rule elevates the confabulation lesson from Module 01 to its highest stakes. An invented fact masquerading under the authority of a historical record—“I remember her potassium was fine”—represents the most dangerous output this course will teach you to prevent.
Furthermore, you must recognize that the entire long-term store functions as a PII database by definition. It holds diagnoses, names, and dates deliberately keyed to specific users. The input-stage hygiene practiced in Module 01 must follow the data directly into the vector store. Strict retention limits, immediate deletion protocols, and absolute per-user isolation are foundational memory features, not retroactive compliance tasks.
Observe the doctrine actively governing a live session — watch for three moves: recalled records cited with their sources, hold-dose numbers attributed to the named cardiologist rather than stated as universal cutoffs, and a user correction superseding the old dose while retiring it to history:
Equip yourself with the materials. Review the advocate’s complete instruction set: patient-advocate-prompt.md. Download the doctrine generalized into a fill-in template applicable to any domain, containing explicit record rules, distillation laws, and privacy mandates: memory-doctrine-template.md.
Executing memory-backed syndicates requires specific infrastructure setup absent from earlier modules: a Supabase project equipped with pgvector. This includes two specific tables and one similarity function, with the SQL provided verbatim in the Melchizedek README. You will also require Node 20+ and your Gemini key.
A model inherently forgets. A record persistently remembers. A doctrine definitively decides which memory earns the right to speak. Master all three, and your agent transcends the limitation of starting over; the second mind maintains the chart, freeing the first mind to actively think. Proceed to Module 05 to grant this mind vision and learn the discipline of refusing to trust it.