Storing and recalling agent memory

When an agent greets you by name, recalls a preference you stated weeks ago, or references an earlier medical decision, the underlying model executes against the exact same frozen weights for every user who queries it. An external system examined those prior conversations, determined which details were worth saving, recorded them in a structured store, and injected the relevant records back into the context window right before running the current turn. That external architecture is what we build in this module. The governing principle is straightforward: a model keeps nothing between requests, so memory is a filing system you build around it. Every engineering choice in that filing system belongs to you: what gets saved, how it is structured, how it is retrieved, and what strict rules dictate how the agent may speak about it.

The transformer architecture requires this external scaffolding because of how its parameters execute. During inference, the model operates on the weights, which remain static across every session, alongside the activations—the vectors built from your text for this run and discarded when the answer ends. Because the runtime clears those vectors the moment generation finishes, any intermediate state the model computes exists only for that single call. A memory system is the practical engineering response to that physical constraint.

We ground every concept in a concrete specimen: the Patient Advocate, an agent designed to help families track a relative’s clinical readings, medications, and physician instructions between appointments. High-stakes clinical environments make memory mechanics easy to evaluate, because an incorrectly recalled dosage or a missed warning sign can cause direct physical harm. By the end of this module, you will be able to trace a piece of information from the moment a user speaks it to its retrieval weeks later, evaluate what a store should retain versus discard, maintain database hygiene so candidate lists do not get clogged with repetitive summaries, and enforce the runtime rules that prevent an agent from hallucinating recollections it never actually recorded.

One statement about this module’s numbers and pictures, made once. Several numbers come directly from running systems: the 43-row store that held roughly seven facts comes from a production audit, and the 0.93 gate, the 768-number embedding, the 24-candidate similarity pool, and the shortlist of ten come from the implementation this module runs. The 0.87 recall score and the 0.97 restatement score are the values from the recorded run this module follows. The comparison of fifteen words against four hundred is a shaped illustration of information density where no formal count was taken. The two architecture diagrams also idealize the operational machinery. The lifecycle figure follows one fact through one clean pass, where the distiller is a model that rephrases what it keeps and recall returns a ranked shortlist rather than one record. The three-channel figure draws similarity, keys, and dates as equals meeting at a point, whereas the running system produces a similarity candidate list first and re-ranks it by key and date, so a record the similarity pass never surfaced cannot be rescued by a key.

the memory lifecycle: stored outside the model, recalled as text

We trace the complete path a single piece of information travels before breaking down each individual component.

During a consultation, a patient shares three clinical updates with the agent: a changed prescription, a new blood pressure reading, or a cardiologist’s instruction to call if a specific vital sign exceeds a given threshold. The system appends every turn of this dialogue to a raw text log so that an unexpected network interruption or server restart will not erase the ongoing discussion. That raw record is your session state, and it represents the simpler half of the architecture.

Once the session concludes, an automated background job reviews the transcript to isolate durable information. Most conversational turns are deliberately discarded, including polite greetings, clarifying restatements, and diagnostic thinking aloud. The facts that survive are distilled into standalone, structured records, each stripped of conversational filler and annotated with an absolute date, a source, and precise numerical values with units. We call this step distillation, and the model configured to execute it is the distiller.

Every distilled fact is converted into an embedding, a numerical vector that captures its meaning, and written alongside its text representation and metadata attributes into long-term memory (adk_memory_facts).

Two weeks later, the patient returns with a related question. Before the agent generates its response, recall runs a cosine-similarity search over long-term memory, raises the score of any record filed under a key the question names or dated to a month or year the question mentions, and inserts the highest-ranked records directly into the context window. The model then answers the patient with those verified records sitting in its active context, executing exactly as if you had pasted those clinical notes into the prompt by hand.

Notice what this final step accomplishes. Grounding is the practice of placing verified source documents, policies, and records into the context window before generation begins. Recall is grounding pointed at a store the system writes to itself across sessions. That is the foundational mechanism of agentic memory, and understanding it clearly explains every design decision that follows.

Recall arrives as text in the window. A remembered fact reaches the model the only way any fact reaches it: as text in the context window, on this request. Recall is grounding pointed at a store the system writes to itself. The weights never change, so what an agent “remembers” is exactly what the store held and the search returned, and nothing more.

one fact · spoken, kept, and brought back

session

the raw transcript, turn by turn

adk_sessions: survives a restart, holds everything, judges nothing

distill

most of it is dropped

what stays is rewritten to stand alone: date, source, exact value

store

adk_memory_facts

the embedding, the text, and the indexed metadata columns

recall

search by meaning, by key, and by date

matching facts are placed in the context window

answer

grounded in the record, cited to its source

the weights never changed at any point in this diagram

memory splits into session state and long-term facts

Why do we maintain two separate data stores instead of one? Engineers frequently attempt to build memory with a single monolithic log, only to run into reliability problems in production. Treating this division as an intentional architectural pattern clarifies why both tiers are necessary.

Session state (adk_sessions) preserves the raw, turn-by-turn conversational history exactly as it unfolds. Its purpose is maintaining immediate continuity within an ongoing interaction. If a user’s connection drops or a backend worker restarts, the last 20 minutes of conversation remain intact. This store is unopinionated: it logs everything and makes no editorial judgments.

Long-term memory (adk_memory_facts) stores carefully distilled facts across different sessions. Its purpose is establishing continuity over extended periods, and it is designed to be intentionally lossy.

Why not simply keep complete conversational transcripts indefinitely and perform semantic searches over those logs? The answer comes down to context overflow. Every token an agent evaluates draws from a finite context budget. When that budget is exceeded, raw API calls fail with context limit errors, while frameworks using rolling buffers silently truncate the earliest tokens. Recalled memory tokens must share that exact same window with your system prompt, tool definitions, and the user’s latest query. If you search raw conversational logs, retrieval brings back sprawling dialogue full of pleasantries and filler, burying the critical fact you actually needed. Distillation ensures maximum information density: a standalone 15 words that convey the core finding rather than 400 words of conversational padding.

These two stores are two settings of the team definition’s memory_system parameter. Setting the parameter to internal-only writes to neither store: the runner holds the conversation in process memory, so the turns of one run still see each other, and nothing survives when the process exits. The session-only setting writes the raw transcript to adk_sessions, so the thread survives a restart and continues when a later turn names the same session id, while long-term enables both session logging and the distilled vector store. The Patient Advocate operates on long-term, which is the tier we will implement across the rest of this module.

a stored fact must stand without its conversation

the five fields of a memory record

When an extracted fact is retrieved weeks later by a model that has no memory of the original interaction, that record must be completely self-contained. We structure each record with five essential fields, each designed to eliminate a specific mode of failure:

An absolute date. Always record explicit dates like 2026-03-15, never relative markers like “yesterday” or “last Tuesday.” Relative expressions lose all meaning once severed from the conversation in which they were spoken; when a model encounters “yesterday” in a stored snippet, it inevitably interprets the timing relative to today’s date.

Provenance. Always document the origin of the information: patient, discharge_summary, or dr_smith. Provenance enables the agent to attribute facts transparently rather than presenting them as unverified assertions, and it provides the necessary context to resolve conflicting data later.

Exact values with units. Always capture complete measurements, such as 120/80 mmHg or 50 mg. Omitting units creates ambiguous records, and ambiguity in context is the primary catalyst for confident hallucinations.

Status. Every record must be tagged with its lifecycle state: active, historical, or superseded. Facts evolve over time, and tracking status allows updates to function as traceable replacements. The distiller marks what is currently true active and what the conversation itself ended historical; when a correction arrives, the store marks the previous entry superseded and the new figure enters as active. A contradiction receives no status of its own: the store keeps both records and adds one record naming the conflict. Status gives the retrieval pipeline and the model a field that separates current regimens from past treatments. How much that field protects an answer depends on whether retrieval filters on it, which we return to under the memory doctrine.

Index keys. Categorical identifiers such as metoprolol or blood_pressure allow our search pipeline to run strict deterministic filters alongside semantic vector matches.

Think of each stored record as an independent statement that must make complete sense on its own, even when loaded into a context window that contains no other information from the day it was written. Each of these five fields preemptively answers a question the model would otherwise have to guess.

embeddings retrieve by meaning rather than vocabulary

An embedding is a list of numbers made from a text snippet, 768 of them here; at its smallest it is a list of three, and two lists are close when their dot product is large. Formally it is a vector of floating-point numbers mathematically positioned such that statements with similar meanings end up close together in high-dimensional space. In our system, this vector contains 768 numbers. What the dimensionality buys is the mathematical property: geometric proximity directly reflects semantic similarity.

This property transforms how effectively our agent can search its history. If a patient asks, “has my blood pressure been getting worse?”, their query shares almost no vocabulary with a stored record stating “BP 148/92, taken at home, 2026-03-15.” A standard keyword match would return zero results. Semantic search, however, retrieves the record effortlessly because the mathematical representations of the question and the clinical entry sit near each other in vector space.

Our framework stores each 768-dimensional embedding right alongside the raw text and indexed metadata columns in PostgreSQL, using the pgvector extension to rank candidates by cosine similarity, scored as 1 - (embedding <=> query_embedding). Every similarity figure in this module sits on that one scale: the 0.87 of a recall that hits, the 0.97 of a restatement, the 0.93 gate that drops it.

retrieval combines semantic, key, and date matching

Relying exclusively on cosine similarity creates practical failures in production. If you execute an unconstrained vector search for “blood pressure,” the database returns every reading ever recorded, ordered solely by topical closeness. That candidate list includes entries recorded years ago while burying recent clinical measurements.

To solve this, recall ranks records on three channels: cosine similarity to locate conceptually relevant records, index keys to favor records filed under an entity the question names, and dates to favor records from a month or year the question mentions. The runtime applies the three in sequence. Cosine similarity runs first across the vector index and returns a pool of 24 candidates for this user. The runtime then adds a fixed boost to each candidate whose index key appears in the text of the question (0.12), whose year the question names (0.08), or whose month the question names (0.10), adds a smaller boost (0.05) to active records, re-sorts the pool, and keeps the top ten. Dates adjust the ranking and filter nothing out: a record from a year the question never mentions keeps its place on similarity alone. If the initial similarity pass misses a relevant record, an exact key match cannot recover it.

One recorded run follows a fact and a preference a user asks the agent to keep through their lifecycle: a Supabase project name and a preference for Claude as the critic model. The user states them in conversation, the system stores them upon session completion, indexes them under specific tags, and recalls the project name two weeks later during a follow-up inquiry. These records describe a project’s infrastructure; the Patient Advocate’s clinical store runs the same storage and recall mechanics on its records.

The metrics in this run demonstrate how recall ranks competing candidates. The relevant record achieved a 0.87 similarity score alongside an exact match on the supabase index key, and that match fired because the word “Supabase” appears in the question itself. The key channel reads only the text of the question: had the user asked which project holds the sessions table without naming Supabase, the record would have competed on similarity alone. Because the question aligned in embedding space and also named the record’s key, the runtime ranked the record above broader conversational candidates. When the user later prompted the system with a restatement of the same fact, deduplication scored it at 0.97 cosine similarity against the existing record, above the 0.93 threshold, and dropped the duplicate entry rather than creating a second copy. We examine the mechanics of that 0.93 deduplication threshold when we inspect database hygiene.

recall enters the window by preload or by a deliberate tool

When retrieving information from long-term memory, facts can enter the agent’s context window through two distinct mechanisms, and choosing the right one directly impacts how you configure your system.

preload_memory operates ambiently. Before the model begins processing an incoming user message, the framework automatically searches long-term memory and injects the top-ranked records into the prompt. The model does not have to request these records; they are already present when it reads the query.

load_memory operates deliberately. It is an explicit tool that the agent invokes whenever preloaded context is insufficient and a targeted search across specific dates or index keys is needed. It is an action surface like any other: a structured tool that queries your internal database on demand.

memory_system: "long-term"
orchestrator:
  tools:
    - "preload_memory"   # ambient: relevant facts injected every turn
    - "load_memory"      # deliberate: targeted search the model invokes

Read that as: this team keeps a distilled store, its orchestrator gets the top matches pushed into every turn without asking, and it also holds a tool for going back to the store with a sharper question. Both lines are that mechanism in configuration: two routes for text to reach the window, and no third route.

Ambient injection keeps everyday conversations fluid and natural without requiring repetitive tool invocations on every turn. Deliberate recall handles focused requests that ambient search typically misses, such as evaluating historical trends over a specific date range, auditing dosage adjustments across multiple months, or comparing three successive vitals. If your agent routinely claims it has no record of facts you know were stored, verify that its deliberate retrieval tool is properly registered and accessible.

fact triage keeps dated values and drops conversational noise

Vector storage in a database is computationally inexpensive; active context space and retrieval precision are not. Every unneeded fact returned by a memory search consumes an available ranking slot and burns through token capacity that should have been reserved for the final response.

For this reason, distillation requires disciplined filtering. Four distinct categories of information justify long-term retention:

Dated events with values. Concrete metrics, medical changes, and completed actions where both the timing and the exact outcome are essential.

Explicit preferences. Clear, user-provided instructions regarding how they wish to be supported, notified, or addressed.

Core identity data. Foundational attributes that remain stable over time, including names, family relationships, and primary clinical diagnoses.

Trend material. Chronological observations where the trajectory of the data provides the clinical signal. Three consecutive elevated blood pressure readings carry diagnostic meaning that a single isolated measurement lacks, but that trend is only visible if each distinct reading is preserved with its exact timestamp.

Everything else should be discarded: conversational greetings, restatements, informal brainstorming, and the agent’s own explanatory prose.