Building memory systems for agents

An agent that greets you by name, recalls your preferences, and refers to a decision you made last month is reading a record of your earlier conversations. Its weights are the same weights every other user is talking to. Something outside the model read those conversations, decided which parts were worth keeping, wrote them down, and pushed the relevant ones back into the context window before this request ran. That machinery is what we build here, and one sentence carries the whole of it: a model keeps nothing between requests, so memory is a filing system you build around it. Every design decision in that filing system is yours: what gets filed, in what form, how it is found again, and what the agent is allowed to say about it.

You already hold the reason the filing system has to exist. In module 1.01 we established that the weights are fixed while you talk to the model and that nothing is written down at the end: close the conversation and every trace of it is gone. Module 1.04 opened that up and proved it from the inside: a model runs on two sets of numbers, a frozen set shared by everyone and a live set built from your text and discarded when the answer ends, and whatever a model works out lives in that live set for one request only. Nothing here depends on that proof; the filing system follows from the plain fact.

We work through it on one specimen, the Patient Advocate, an agent that helps a family keep track of a relative’s medications and readings between clinic visits. The stakes there make every judgment call visible: a remembered dose that is wrong can put someone in the hospital. By the end you will be able to follow one fact from the moment it is spoken to the moment it is recalled weeks later, decide what a store should keep and what it should drop, keep the store from silting up with restatements, and write the rules that stop an agent inventing a memory it never formed.

the lifecycle, end to end

Here is the whole path before we name any part of it.

A patient tells the agent something during a conversation: a dose changed, a reading was taken, a cardiologist said to call if a number goes above a threshold. That conversation is stored as raw text, turn by turn, so a server restart cannot lose it. That store is the session state, and it is the easy half.

When the session ends, a background process reads the whole transcript and keeps only what deserves keeping. Most of what was said goes: greetings, restatements, thinking aloud. What survives is a small number of discrete facts, each stripped of the conversation around it and rewritten to stand on its own with a date, a source, and exact values. That pass is called distillation, and the model that runs it is the distiller.

Each surviving fact is turned into a list of numbers that captures its meaning and written to a database alongside its text and its metadata. The list of numbers is an embedding; the database is long-term memory.

Two weeks later the patient asks something related. Before the model sees the question, the system searches that database, by meaning, by category tag, and by date, and places the matching facts into the context window. The model then answers with those facts in front of it, exactly as if you had pasted them in by hand.

Look at what that last step is. In module 1.01 we called it grounding when you supplied verified material, records, source files, policy documents, into the context window before the model generated anything. Recall is the same move with one difference: the material was written by the system itself, from an earlier conversation, and fetched by a search instead of by you. That is the whole trick, and it is worth extracting, because everything else in the module leans on it.

Recall arrives as text in the window. A remembered fact reaches the model the only way any fact reaches it: as text in the context window, on this request. Recall is grounding pointed at a store the system writes to itself. The weights never change, so what an agent “remembers” is exactly what the store held and the search returned, and nothing more.

two tiers, two jobs

Why two stores? People routinely build only one of them and are surprised by what breaks, so the split is worth understanding as an engineering decision rather than a naming convention.

Session state (adk_sessions) holds the raw conversation, turn by turn, exactly as it happened. Its job is continuity within an interaction: a network drop or a server restart does not lose the last twenty minutes. It keeps everything and judges nothing.

Long-term memory (adk_memory_facts) holds distilled facts across sessions. Its job is continuity between interactions, and it is deliberately lossy.

Why not keep the full transcripts forever and search those? Because of context overflow from module 1.01: everything the model can consider shares one budget of tokens, and when input exceeds it a raw API call returns an error while a framework holding rolling history trims the oldest tokens out of scope without a word. That budget is the binding constraint here. Every token of recalled material competes with the system instruction, the tools, and the current question for the same space. Searching raw transcripts returns whole conversations, most of which is filler, and the useful sentence arrives buried in it. Distillation exists to make recall dense: fifteen words that carry the fact, instead of four hundred that contain it. The counts are illustrative; the ratio is the point.

These two stores are two of the three settings of the memory_system key you met in module 2.04. The third, internal-only, keeps neither: the context resets after each turn. session-only keeps the transcript for the length of a session, and long-term adds the distilled store. The Patient Advocate runs on long-term, and that is the tier the rest of this module builds.

a fact, from spoken to stored

the five fields

A fact that will be read weeks later, by a model with no memory of the conversation that produced it, has to carry its own context. Five fields do that work, and each one prevents a specific failure.

An absolute date. 2026-03-15, never “yesterday” or “last Tuesday.” A relative date is meaningless the moment it leaves the conversation that anchored it, and a model reading “yesterday” in a stored record resolves it against the only date its window now holds, which is today’s.

Provenance. Who asserted this: patient, discharge_summary, dr_smith. This is what lets the agent say where an answer came from instead of stating it as fact, and it is what makes a contradiction resolvable later.

Exact values with units. 120/80 mmHg, 50 mg. A bare number is ambiguous context, and ambiguous context is where confident wrong answers come from.

Status. active, superseded, or contradicted. Facts change, and status is how a change registers as a replacement: the new value is written active, the old one marked superseded, so a search can tell current from history instead of returning both as equals forever.

Index keys. Category tags like metoprolol or blood_pressure, used for exact filtering alongside meaning-based search.

Read a stored record as a sentence that has to survive on its own, in a window that holds nothing else from the day it was written. Each of the five fields answers, in advance, a question the model would otherwise answer by guessing.

what the embedding does

An embedding is a list of numbers computed from the fact’s text, positioned so that texts with similar meanings land near each other. Here it is 768 numbers long. Nothing about that length is special; what matters is the property that closeness in the list tracks closeness in meaning.

That property is the whole reason the store is searchable in a useful way. A patient asking “has my blood pressure been getting worse?” uses none of the words in a record reading “BP 148/92, taken at home, 2026-03-15.” Keyword search finds nothing. Meaning-based search finds it, because the two texts sit near each other in that numeric space.

The framework stores each embedding beside the record text and the indexed metadata columns, in PostgreSQL, an ordinary general-purpose database; the pgvector extension is what lets it hold these number lists and rank rows by closeness.

three ways of matching, at once

Is meaning alone enough? Try it and you will see that it is not. Search by meaning for “blood pressure” and you get every blood pressure record ever stored, ranked by similarity, including the ones from two years ago, and excluding nothing.

So the search runs three ways at the same time: by meaning, which finds what the question is like; by index key, which finds what it refers to; and by date, which finds when.

Watch one fact travel the whole arc: spoken once, distilled at session end, filed under its keys, recalled two weeks later.

Read the two numbers in that trace as the two halves of the search. The recall came back at 0.87 similarity together with a key match on supabase: the question resembled the record, and the record was filed under the word the question used, so it rose above everything that merely sounded similar. The restatement the user asked for scored 0.97 against the record it repeated, and the store kept one copy. Hold onto that second number; when we clean the store below, you will see the door it was stopped at.

two channels back into the window

Recalled facts reach the model in one of two ways, and the difference is worth knowing because it changes how you configure an agent.

preload_memory is ambient. Before the model processes each new message, the framework searches the store and injects the top matches into the context automatically. The agent does not ask; the facts are already there when it reads the message.

load_memory is deliberate. It is a tool the agent calls when the preloaded facts are not enough and it needs a targeted search against specific dates or keys. This is the action surface from module 2.01, the tools an agent can call, pointed at your own database.

memory_system: "long-term"
orchestrator:
  tools:
    - "preload_memory"   # ambient: relevant facts injected every turn
    - "load_memory"      # deliberate: targeted search the model invokes

Read that as: this syndicate keeps a distilled store, its orchestrator gets the top matches pushed into every turn without asking, and it also holds a tool for going back to the store with a sharper question. Both lines are that mechanism in configuration: two routes for text to reach the window, and no third route.

Ambient recall keeps ordinary conversation smooth without a tool call on every turn. Deliberate recall handles the questions ambient search will miss, which are usually the specific ones: a date range, a particular medication, a comparison across three readings. If your agent keeps answering “I don’t have that” about things it plainly stored, check whether it has the second channel at all.

deciding what deserves to be kept

Vector storage is cheap. Context space and retrieval accuracy are not. Every irrelevant fact your search returns competes for a ranking slot and spends tokens that the actual answer needed.

So distillation is a judgment, and it is the part of a memory system that most rewards care. Four categories earn their place:

Dated events with values. Measurements, completed tasks, changes to the system: anything where what happened and when both matter.

Explicit preferences. Directives the user actually stated about how they want to be dealt with.

Core identity data. Names, roles, static attributes that stay true.

Trend material. Sequential measurements where the change is the information. Three consecutive elevated readings mean something no single reading does, and that only works if all three are kept with their dates.

Everything else goes: greetings, restatements, the user thinking out loud, the agent’s own explanations.

Practice the call below. Some of these are obvious and some are genuinely arguable, which is the point:

keeping the store clean

The lifecycle you traced has a failure mode that only shows itself in production, so we meet it through the store where it was found. A deployed assistant’s memory had grown to 43 rows. Read closely, those rows held roughly seven distinct facts. One balance had been recorded six different ways. One preference, seven. Thirteen separate session summaries narrated the same session, each with a slightly longer list of details than the last. Recall returns a fixed shortlist, ten candidates here, so by the time the store looked like that, most of every shortlist was one fact wearing different sentences, crowding out the records the user actually needed.

How does a store get that way when every part of it is working? Two mechanics compounded, and neither is a bug in any single part. A served agent is stateless between requests, so it runs the distillation pass after every completed task, and each pass reads the whole session; a conversation of twenty turns gets distilled twenty times, over a transcript that grows each time. And the guard that should have caught the repeats compared strings byte for byte, while the distiller is a model, sampling each summary token by token, so no two passes over the same fact land on identical text. Each new phrasing of the same balance sailed past a comparison built for identical text.

The repair is one mechanism at each end of the pass.

Ingest incrementally. The pipeline keeps a high-water mark per session, how far into the transcript the last distillation reached, and each new pass reads only the turns added since. The mark lives in process memory rather than in a table of its own, which is a deliberate economy: after a restart, one session gets re-read once, and the check below absorbs the repeats that single re-read produces.

Deduplicate by meaning. Before a new record is inserted, the store is probed with the same meaning-based search that recall uses. A candidate that lands above 0.93 similarity to a record already stored is a restatement, and it is dropped. That number belongs to the embedding model in use here; a different model spreads its scores differently, so re-measure the threshold on your own records rather than copying it across. That is the door the 0.97 restatement in the trace was stopped at. Two conditions guard the guard. The categories must match: a session summary that mentions a dose change must never suppress the dose record itself, however similar their sentences read. And a superseded record never suppresses anything; history cannot veto the present.

The threshold is a judgment, not a law of nature. Raise it and more restatements survive; lower it and genuinely distinct facts begin suppressing each other. So which way should a borderline call fall, including the case where the probe itself throws an error? Weigh the two mistakes. A duplicate that gets through costs one slot on a ten-record shortlist. A real fact that gets dropped costs the user something they told the agent, and it is gone without anyone noticing. The costs are not symmetric, and the rule follows from that.

A duplicate is cheap; a lost fact is not. A duplicate costs a shortlist slot; a lost fact costs the user something they said. Every doubt resolves toward storing, and a dedup probe that fails stores the record anyway.

Tell the distiller what your domain keeps. The prompt that does the distilling is shared by every long-term syndicate, and its examples are medical because the Patient Advocate came first. But what deserves keeping is a domain judgment, which is why the schema in module 2.04 carries memory_extraction_rules, a per-syndicate field whose text is appended to the shared prompt at run time. Its natural shape is two lines. Never store the values that go stale on their own, a live price, a current reading the system re-fetches anyway, because a stored copy can never be used, only crowd out records that can. Always store what the user asserted, decided, or committed to. A syndicate that declares nothing gets the shared prompt exactly as it was, so the field costs existing agents nothing.

Clean up by deleting, not by retiring. Supersession, which the doctrine below builds on, is the right treatment for facts that were true: the old dose really was 25 mg, and history should say so. Junk is different. A restatement was never information, and marking it superseded does not remove it from the search: the store’s similarity query filters by user alone and applies status afterwards, in the re-ranking, so a retired row still occupies one of the ten candidate slots it was polluting. For junk, the framework ships a small admin tool, npm run memory, that lists a user’s records, groups likely restatements, and deletes the ones you name. It is a dry run unless you confirm, and every delete is scoped to one user’s silo.

The habit that follows: read your store, not just your traces. Once a month, list one user’s records and count the distinct facts against the rows. If the rows are winning, the shortlist is already paying for it.

the memory doctrine

Module 2.01 established that doctrines follow capabilities: an agent earns a new instruction block the moment it gains a new power, because a power without a written rule is one nobody decided about. Memory is exactly such a power, and it is a dangerous one, because a fabricated recollection is far more convincing than a fabricated fact.

A memory doctrine governs how remembered facts are cited, updated, and handled when the store is silent.

<memory_doctrine>
1. EVERY FACT REQUIRES PROVENANCE: Every stored record carries a date, source,
   and exact value with units. When stating a remembered fact, cite its source
   explicitly ("According to your discharge summary dated 2026-07-03…").
2. CORRECTIONS SUPERSEDE: When a user provides updated data, mark the old
   record `superseded` and store the new value as `active`. Never quote a
   superseded value as current.
3. EXPOSE CONTRADICTIONS: If two stored records conflict, state the conflict
   and ask the user. Never silently guess which record is correct.
4. STATE SILENCE CLEARLY: If the record holds no data for a request, say so.
   Never invent a historical fact.
</memory_doctrine>

Read that as four answers to four questions the model will otherwise answer by guessing. Where did this come from? Say so, with the date. Which of two values is current? The one marked active, and say the other was superseded. What if two records disagree? Say that they do, and ask. What if there is nothing there? Say that there is nothing there.

Rule 4 is the one that carries the most weight, and it is worth seeing why before we make a law of it. In module 1.01 we named confabulation: asked for something it does not hold, a model produces a fluent answer shaped exactly like a true one, and its confidence tracks how clear the pattern was in training, never whether the claim is true. Here the stakes are raised, because the shape of a correct answer is a specific, dated, plausible thing you told it once. A patient has no way to distinguish a real recollection from an invented one; both arrive with the same confidence and the same specificity. So the doctrine has to make “the record is silent” the expected thing to say, exactly as the grounding doctrine in module 2.01 did for a missing policy.

A silent store is reported as silent. A memory system must never invent a historical record. If the store holds no matching data, the agent states that the record is silent.

Rule 2 deserves a note on what actually happens in the database. When a dose changes from 25 mg to 50 mg, the old record is not deleted. Its status becomes superseded and the new record is written as active. The old value stays as audit history; when a search surfaces it, it arrives relabelled as superseded so it can never masquerade as current, and you keep the ability to answer “what was I taking in March?” without ever quoting it as today’s dose.

The doctrine pairs with a boundary that memory makes newly necessary. Once an agent can recall a threshold a clinician set, it can also be asked for one the clinician never set, and the shape of the two answers is identical. So the rule has to be about where a number may come from.

Numbers belong to their clinician. A numeric action threshold, a hold-the-dose cutoff, a when-to-call vital sign, may appear in the agent’s output only as a quote from this patient’s record, attributed to the clinician or document that set it, with its date. An agent supplying a “typical” cutoff from general medical knowledge has written a prescription, whatever the surrounding language says.

Watch the doctrine govern a live session. Look for three moves: recalled records cited with their sources, hold-dose numbers attributed to the named cardiologist rather than stated as universal, and a user correction superseding an old dose while retiring it to history:

Two things in that run are worth reading back against the mechanism. The recall step returned eight records ambiently, before the agent asked for anything: that is preload_memory, and it is why the first answer could cite dates without a tool call. And the correction shows rule 2 at the row level: the 25 mg record was retired, kept, and the 50 mg record written as active, so the store can still say what he was taking in early July.

the store is a PII database

Say this plainly, because it is easy to build a memory system without noticing you have done it.

A long-term memory store holds diagnoses, names, dates, and preferences, deliberately keyed to individual people. That is a personal-data database by any definition that matters, and it acquired that status the moment it started working well.

In module 1.01 we placed the first human check at the input stage: you control what enters the window, and you strip the credentials and sensitive data that should never be sent. A memory store moves that stage inside the system, because now the system itself decides what enters the window, from a store it filled. So the same discipline has to follow the data all the way in. Retention limits with an actual expiry. A deletion path that really removes records rather than hiding them, which is the same distinction as deleting junk rather than retiring it. Per-user isolation enforced in the query, so that no search can cross between users even when a bug tries. These are features you build alongside the memory system, in the same commit, not compliance work you schedule for later.

equip yourself with the materials

1. The complete system prompt for the specimen: patient-advocate-prompt.md.
2. A domain-agnostic doctrine template: memory-doctrine-template.md.
3. Run it live with npm run syndicate:advocate. Long-term memory needs a PostgreSQL database (a free Supabase project works) with pgvector; the adk_sessions and adk_memory_facts schema is verbatim in the repository’s DOCUMENTATION.md §4, followed by db/hardening.sql. You will also want Node 22+ and your Gemini key.

what you can do with this today

That mechanism is a diagnostic you can run on any agent that claims to remember. When it gets a remembered fact wrong, ask where in the lifecycle the fact was lost, because there are only four places it can go wrong.

Was it ever stored? If the agent never had it, the distiller dropped it, or the fact was one of the kinds the store never keeps. Read the transcript, then read the store. If a durable fact was spoken and is not in the rows, the fix is in the distillation prompt or the memory_extraction_rules for this syndicate.

Was it stored in a form that can be read cold? Date, source, exact value with units, status. A record that only makes sense next to the conversation that produced it is a record that will be misread. Write facts that can stand alone.

Was it found? A stored fact that never reached the window failed at recall. Check whether the shortlist was full of restatements, whether the question’s words matched a key, and whether the agent had the deliberate channel for a sharper search. Decide your forgetting rules before your keeping rules: what gets dropped, what supersedes what, and what expires. Systems that only ever accumulate get slower and less accurate at the same time. Supersede history; delete junk, because a retired row still spends a candidate slot every time the store is searched.

Was it in the window and still misstated? Then the doctrine failed. A superseded value quoted as current, a silence filled with an invention, a threshold stated without its clinician: each is one rule of the doctrine, and each is countable against the transcript.

And treat the store as personal data from the first commit. Deletion, retention, and isolation are cheap to build in and expensive to retrofit.

Here is the whole lifecycle on one card, in the vocabulary you now own:

1. SESSION STATE            adk_sessions
   The raw transcript, turn by turn. Survives a restart. Judges nothing.

2. DISTILL                  at session end, incrementally
   Read only the turns since the high-water mark.
   Keep: dated events with values · explicit preferences ·
         core identity · trend material.  Drop everything else.
   Each kept fact rewritten to stand alone:
         absolute date · provenance · exact value with units ·
         status · index keys.

3. DEDUPLICATE              before insert
   Probe the store by meaning. Same category, active, and at or
   above 0.93 similarity → a restatement, dropped.
   Any doubt, any probe error → store it; a lost fact costs more.

4. STORE                    adk_memory_facts
   The embedding (768 numbers) beside the text and the indexed
   metadata columns. Corrections supersede; junk is deleted.

5. RECALL                   before the model sees the question
   Similarity search → a shortlist of ten → re-ranked by keys
   and dates. Ambient via preload_memory; deliberate via load_memory.
   Matching facts land in the context window as text, like any fact.

6. ANSWER                   under the memory doctrine
   Cite the source and date. Never quote a superseded value as
   current. Expose contradictions. If the store is silent, say so
   and invent nothing. Numbers belong to their clinician.

Next we extend an agent’s reach past its own database, connecting it to live external systems over the Model Context Protocol, and confront what it means to let an agent act on the far side of that connection. By the end of it you will be able to give an agent tools it was never built with, and say exactly what has to sit outside the model for that to be safe. Open Module 2.06 to continue.