Agent Memory: How LLM Agents Remember and Forget

Imagine a hotel receptionist who is brilliant, charming, knows every rule of the hotel by heart — and wakes up every morning with no memory of yesterday. Regulars introduce themselves again every single day. The workaround the hotel invents is a notepad: before each shift, someone writes down everything the receptionist needs to know today, and the receptionist reads it before greeting anyone.

That receptionist is a large language model. The model’s weights know the world the way the receptionist knows hotel rules — baked in during training, frozen after. But between two API calls, nothing persists. There is no save button inside the model. ChatGPT “remembering” your name is not the model remembering — it’s engineering around the model: something wrote your name down, and something pastes it back into the notepad every morning.

That notepad is the context window (the fixed number of tokens the model can read per call), and it has three properties that shape everything in this post:

1. It’s working memory, not storage. Whatever isn’t re-sent on the next call is gone.
2. It’s rented by the token. Every memory you paste in costs money and latency, on every single call.
3. It degrades when full. As the context grows, the model’s ability to find things in it drops — people call this context rot, and it falls out of how attention works: every token competes with every other token for a fixed budget of attention.

The version everyone builds first#

My first agent did what every first agent does — kept the whole conversation and re-sent it every turn:

history = []

def chat(user_message: str) -> str:
    history.append({"role": "user", "content": user_message})
    reply = llm.complete(system_prompt, history)   # send EVERYTHING, every time
    history.append({"role": "assistant", "content": reply})
    return reply

• history is the notepad, growing without limit.
• llm.complete re-reads the entire notepad on every turn — the model itself remembers nothing between calls.
• Nothing ever leaves history, and nothing survives a process restart.

For a demo this is perfect. Then real usage arrives and three clocks start ticking. Cost: by turn forty the prompt carries every “sounds good!” from turn three, and you pay for all of it, every turn. Latency: prompts grow, responses slow. Quality: the model starts missing things that are literally in the prompt, because they’re buried under ten thousand tokens of chit-chat — and the day the conversation exceeds the window, the oldest messages silently fall off the edge, which is how an agent forgets the user’s name it was told an hour ago.

All three clocks run on the same missing capability: the model has no write operation. Nothing inside it can decide that a fact matters. So unless something outside decides what to keep, where to keep it, and when to bring it back, nothing is ever truly remembered.

Which means somebody has to keep the notepad — and that turns out to be four jobs: write down the few things worth keeping, paste back only what earns its rent, throw out what stopped mattering, and tidy the page while nobody’s asking.

Memory is a data pipeline, not a database#

The mistake I brought into this topic was thinking “agent memory” was a product you pick — a vector database (a store you search by meaning instead of by key), and done. Reading the actual systems cured that. A vector database is a shelf. Memory is the entire librarian job around the shelf, and it has exactly four verbs:

flowchart TB
  subgraph hot["Every turn (hot path)"]
    U["user turn"] -->|"extract(turn)"| W["write path"]
    W -->|"add / update facts"| S[("memory store")]
    S -->|"top-k memories"| R["read path"]
    R -->|"pasted into prompt"| L["LLM call"]
  end
  subgraph cold["On a schedule (background)"]
    C["consolidation"] -->|"merge repeats into one fact"| S
    F["forgetting"] -->|"evict what decayed"| S
  end

If a backend analogy helps (it carried me through the whole topic): the write path is ingestion with a filter, the read path is a query with a ranking function, forgetting is cache eviction, and consolidation is log compaction. None of these ideas are new — what’s new is that an LLM makes the judgment calls inside them.

One verb at a time from here, each with code that runs. Everything below is from my agent-memory-lab repo — plain Python, no dependencies, no API keys (the LLM-shaped pieces are deterministic stand-ins with the real-API seam marked), so the whole lifecycle runs offline in one python demo.py.

What’s on the shelves — the four kinds of memory#

The verbs move things around. Before walking them, it’s worth knowing what the things are, because “memory” is not one substance — it’s four, each doing a different job, each failing differently. The vocabulary comes from cognitive science by way of CoALA (the 2023 Princeton paper that mapped human memory categories onto language agents), and by now it’s the shared language of the whole field — Letta, Mem0, and LangChain all organize their docs around it.

One shelf sits below all of them: the weights themselves — parametric memory. That’s the receptionist’s actual training: language, world knowledge, hotel rules. Facts can be pushed in there by fine-tuning, but for per-user memory it’s the wrong shelf three times over: writing is slow and expensive, you can’t inspect what was stored, and you can’t surgically delete one user’s data when they ask. So everything below is deliberately outside the weights, in stores you can read, audit, and erase.

flowchart LR
  RAW["raw turn"]
  subgraph shelves["Long-term shelves"]
    EP["episodic: events, timestamped"]
    SE["semantic: timeless facts"]
    PR["procedural: skills + playbooks"]
  end
  subgraph promptbox["This turn's prompt"]
    WM["working memory = context window"]
  end
  RAW -->|"write path"| EP
  RAW -.->|"direct fact write"| SE
  EP -->|"consolidate"| SE
  EP -->|"reflect"| PR
  EP -->|"retrieve"| WM
  SE -->|"retrieve"| WM
  PR -->|"retrieve"| WM

Working memory is the context window itself — what you hold in your head while dialing a phone number. It’s the only memory the model can actually think with: every other type is inert storage until something pastes it in here. It’s also the scarcest and the most fragile (rented per token, rots when full), which reframes the entire topic — the memory game is really “who earns a slot in working memory this turn?” The advanced end of this type is working-memory management: MemGPT’s core-memory blocks are labeled, size-capped regions of the context the agent edits with tools, and Claude Code’s auto-compaction is a forced summarize-and-restart when the window fills.

Episodic memory is events with timestamps — the diary. “User reported order #4411 broken on Aug 1.” “Tried the regex fix; tests failed.” It’s what you need when history itself is the value: a support agent recalling past tickets, a coding agent remembering which three approaches already failed so it doesn’t loop, Reflexion storing lessons from failures, ChatGPT’s reference-to-past-chats. Two advanced moves hide here. First, episodic records can be replayed as few-shot examples — a past solved case pasted into the prompt teaches better than any instruction. Second, episodic is by far the noisiest type — it’s the shelf that forgetting and consolidation exist to clean.

Semantic memory is timeless facts — no event attached. “User is pescatarian.” “The deploy target is Vercel.” Highest value per token of anything you can inject, which is why it’s what ChatGPT’s “memory updated” writes, what a CLAUDE.md file is, what Zep stores as graph edges, and what a RAG knowledge base holds at document scale. Facts are born two ways: written directly at extraction time, or distilled out of many episodic records by consolidation. The advanced problem is shape: keep facts as one profile document (compact, but every update is a risky merge into existing text) or as a collection of independent items (easy to add, hard to deduplicate)? And staleness lives entirely on this shelf — the vegetarian-to-pescatarian collision from the sequence diagram above is a semantic-memory problem by definition.

Procedural memory is how-to — riding a bicycle: you can’t recite it, you perform it. For agents it takes two forms: refined instructions (a system prompt or playbook that improves with experience), and skills as executable code — Voyager’s library of verified Minecraft functions is the famous one, and every Claude Code skill or saved workflow is the same idea in production clothes. This is the only type that makes an agent better rather than merely better-informed, so it compounds — and it ages differently: a skill doesn’t decay with time, it breaks when the environment changes underneath it. Evict on failure, not on a clock. The research frontier here is pulling skills back into the weights (fine-tuned executor models) — the parametric shelf reclaiming what the code shelf proved out.

Which shelf leads depends entirely on what the agent is:

In the lab, the shelves are the Kind enum on MemoryRecord — and tracing who creates each kind is a nice way to internalize the flow: the extractor only ever mints EPISODIC, the consolidator is the only place SEMANTIC is born, and PROCEDURAL sits in the enum unused. That last one is honest: the lab has no skill library yet, and building one is on my experiment list precisely because it’s the type my flagship lane (forgetting) interacts with least obviously.

Write path — deciding what deserves to exist#

Here’s the part every explainer hand-waves: how does a memory actually get created? A conversation turn is mostly noise. “Nice weather today, haha” must not become a memory; “I always prefer a refund over store credit” must. Someone has to judge that, at write time:

def extract(self, turn: str, now: datetime) -> list[MemoryRecord]:
    out = []
    for sentence in re.split(r"(?<=[.!?])\s+", turn.strip()):
        importance = self._judge_importance(sentence)   # LLM call in production
        if importance < self.threshold:
            continue        # most of a conversation should NOT be remembered
        out.append(MemoryRecord(
            id=str(uuid.uuid4()),
            kind=Kind.EPISODIC,
            content=sentence.strip(),
            embedding=self.embedder.embed(sentence),
            created_at=now,
            importance=importance,
        ))
    return out

• _judge_importance is the judgment call. In production it’s one LLM call — “extract durable facts, preferences, decisions from this turn; rate importance 0 to 1.” The lab uses transparent regex heuristics so the pipeline runs offline.
• The threshold line is the whole philosophy: remembering is opt-in. A memory system that keeps everything is just the naive history list wearing a database.
• importance gets judged once, at write time, and stored on the record. The read path will reuse it forever — the Generative Agents paper (the famous simulated-town experiment from Stanford) literally prompts a model to rate memories 1 to 10, where “brushing teeth” earns a 2 and “asking your crush out” earns an 8.
• embedding is the sentence turned into a vector (a list of numbers where similar meanings land close together) — this is what makes the read path’s “is it about this?” question computable.

But extraction is only half the write path. The harder half is what happens when a new fact collides with an old one. Watch one fact live its life:

sequenceDiagram
  participant U as User
  participant A as Agent
  participant S as Store
  U->>A: "I'm vegetarian"
  A->>A: extract(turn)
  A->>S: search(similar to "diet: vegetarian")
  S-->>A: nothing similar found
  A->>S: ADD("user is vegetarian", importance 0.8)
  Note over U,S: three weeks later
  U->>A: "actually I eat fish now"
  A->>A: extract(turn)
  A->>S: search(similar to "diet: eats fish")
  S-->>A: hit: "user is vegetarian" (0.86 similar)
  A->>S: UPDATE(old fact becomes "user is pescatarian")

That decision at the end — the industrial version of it is Mem0’s write path, where an LLM tool-call picks one of ADD / UPDATE / DELETE / NOOP for every extracted fact against its nearest stored neighbors. Zep’s temporal-graph store makes the other interesting choice: it never overwrites — the old fact gets stamped invalid_from: today and kept, so the agent can still answer “what did I believe in July?” Same collision, two philosophies: overwrite the past, or version it.

Read path — deciding what earns prompt space#

At read time the question flips: of everything in the store, which two or three memories are worth their token rent in this prompt? The recipe that stuck (from the Generative Agents paper, reused everywhere since) scores every candidate on three signals:

Relevance asks is it about this? (similarity between the query vector and the memory vector). Recency asks is it fresh? (exponential decay since last use — the paper decays by 0.995 per hour). Importance asks did it ever matter? (that write-time judgment). One signal alone fails in an obvious way: pure relevance surfaces a year-old address the user has since changed; pure recency surfaces this morning’s chit-chat; pure importance surfaces the user’s wedding anniversary in a debugging session.

def retrieve(self, query: str, k: int, now: datetime) -> list[Hit]:
    q = self.embedder.embed(query)
    hits = []
    for r in self._records.values():
        relevance = max(0.0, cosine(q, r.embedding))
        hours_idle = (now - r.last_accessed_at).total_seconds() / 3600
        recency = 0.5 ** (hours_idle / self.half_life_hours)
        score = self.w_rel * relevance + self.w_rec * recency + self.w_imp * r.importance
        hits.append(Hit(r, score, relevance, recency, r.importance))

    hits.sort(key=lambda h: h.score, reverse=True)
    top = hits[:k]

    for h in top:                      # retrieval doubles as rehearsal
        h.record.last_accessed_at = now
        h.record.access_count += 1
        h.record.stability *= 1.5
    return top

• cosine measures the angle between two vectors — small angle, similar meaning. It’s the entire magic of “search by meaning” in one function.
• recency is a half-life, exactly like caching: a memory untouched for one half-life scores 0.5, for two half-lives 0.25.
• The Hit carries all three sub-scores, not just the total. When retrieval misbehaves — and it will — the first debugging question is which signal put this memory here, and logging the breakdown answers it for free.
• The last block is the sneaky-important part: a retrieval that gets used is a rehearsal. It resets the memory’s decay clock and grows its stability. That one side effect is what connects the read path to forgetting — useful memories keep earning their place; ignored ones quietly age out.

Forgetting — the chapter nobody writes#

Here’s what surprised me most in the reading: of the major memory systems, none has a real forgetting policy. Mem0 never deletes. Zep invalidates but keeps everything forever. Generative Agents’ memory stream just grows. The one hint in the literature is Letta evicting “about 70% of messages” on overflow without saying which 70%. Meanwhile the store grows, retrieval gets noisier, and stale facts about a user’s old job compete with fresh ones.

Every backend engineer has already run a forgetting policy in production. It was called cache eviction, and the classics translate directly:

def ebbinghaus(retention_threshold=0.25, base_half_life=timedelta(hours=24)):
    """retention = exp(-t / S): t = time since last access, S = stability.
    Importance gives S a head start; every rehearsal multiplies S."""
    base_hours = base_half_life.total_seconds() / 3600

    def policy(records, now):
        victims = []
        for r in records:
            hours_idle = (now - r.last_accessed_at).total_seconds() / 3600
            s = r.stability * base_hours * (0.5 + r.importance)
            retention = math.exp(-hours_idle / s)
            if retention < retention_threshold:
                victims.append(r.id)
        return victims
    return policy

• The formula is the Ebbinghaus forgetting curve — the 1885 psychology result that memory retention decays exponentially, and that each rehearsal makes the next decay slower (the spacing effect). Here it’s just TTL with a twist: the TTL grows every time the memory proves useful.
• stability is that twist. TTL evicts by age, LRU (evict the least-recently-used — the default policy in Redis-style caches) evicts by last touch; Ebbinghaus evicts by earned trust — an important fact retrieved weekly becomes effectively permanent, while an equally important fact never retrieved again dies in a week.
• The policy returns victims instead of deleting directly — in a real system you’d log them, or demote them to cold storage, before dropping them. Deleting is the one operation you can’t debug afterwards.

One more thing the decay clock quietly assumes: it’s an episodic-memory tool. The other shelves die by different causes — a semantic fact is killed by collision (a newer fact contradicts it: UPDATE or invalidate), and a procedural skill is killed by failure (the environment changed and it stopped working). Running time-decay over facts and skills evicts things that were perfectly good, which is a bug wearing a policy’s clothes.

Consolidation — sleep for agents#

The fourth verb runs on a schedule, not on a turn. After a week, the store holds five separate episodic records that all say some version of “user prefers refunds over store credit.” Each one is weak on its own; together they’re one strong fact. Consolidation is the nightly job that notices the repetition and distills it:

for cluster in self._cluster(episodic):           # group near-duplicates by cosine
    if len(cluster) < 2:
        continue
    distilled = self.summarize([r.content for r in cluster])   # LLM call in production
    store.write(MemoryRecord(
        kind=Kind.SEMANTIC,
        content=distilled,
        importance=min(1.0, max(r.importance for r in cluster) + 0.1),
        source_ids=[r.id for r in cluster],       # provenance: auditable back to sources
        ...
    ))
    for r in cluster:
        store.forget(r.id)

• The kind changes from EPISODIC (things that happened) to SEMANTIC (facts distilled from them) — which is, not coincidentally, how psychologists describe what sleep does to human memory.
• source_ids is the line I’d fight for in a code review: the summary keeps pointers to the raw records it came from. When a consolidated “fact” turns out wrong, provenance is the difference between debugging and shrugging.
• A repeated fact gets a small importance bump — something the user said five times matters more than something said once.
• The backend name for this whole move is log compaction (Kafka’s keep-only-the-latest-value-per-key cleanup): many raw events in, one current fact out, old entries dropped.

The same four verbs, four famous shapes#

Everything above is one tiny system, but the four verbs are a lens that makes the whole landscape readable. Every well-known memory architecture is a different answer to “which verb do you make central?”:

Squint and a second pattern appears: each school is one memory type taken to its extreme. Paging is working-memory management made total; Mem0 and Zep are semantic-fact stores with opposite collision philosophies; the scored diary is episodic memory with a good librarian. The missing row is procedural — its flagship is Voyager’s skill library, which nobody has yet folded into a general memory product. That empty seat at the table is worth remembering.

None of this stays in the papers. Three of these four shapes are running on my machine today. ChatGPT’s and Claude’s built-in memory are the write-pipeline shape (a profile document plus retrieval over past chats — and ChatGPT writes memories in the hot path, which is why you sometimes see “memory updated” mid-conversation). Claude Code runs the paging-plus-notes shape: a CLAUDE.md file read at session start is the paged-in memory block, and its auto-compaction is consolidation under deadline. And Anthropic’s own engineering guidance for long-running agents — compaction, structured note-taking, sub-agents that return only summaries — is the four verbs applied to a single long context instead of a database.

The honest numbers — when a memory system loses#

This is the section I couldn’t find in any vendor post, and the numbers were sitting in the vendors’ own papers.

Mem0’s paper benchmarks itself against simply stuffing the entire conversation history into the prompt. Full-context wins on accuracy: 72.9% vs 66.9% for Mem0 on the LoCoMo benchmark (long multi-session conversations with questions about earlier sessions). What memory wins is everything else: 1.44s vs 17.12s p95 latency (p95 = the time your slowest 1-in-20 request takes) and roughly 90% lower token cost (~2k tokens per query instead of ~26k). Zep’s paper has the same shape hiding in it: up to +18.5% on LongMemEval overall, while losing 17.7% on single-session questions — when everything relevant already fits in the context, retrieval can only remove information.

So the honest framing is: a memory system is a lossy compressor you buy for latency, cost, and persistence — not for accuracy. That reframes when to use one:

• Don’t build one when conversations are short-lived, everything fits comfortably in the window, and nothing needs to survive the session. A long context plus a plain notes file is the strongest baseline in the field — slightly embarrassingly, the most-adopted memory standard of 2026 is a markdown file read at session start.
• Build one when facts must survive across sessions, when the history is far bigger than the window, or when the token bill of re-sending everything is the actual problem.
• And know the new attack surface you bought. A store the agent reads from is a store an attacker wants to write to: the PoisonedRAG result needs just 5 poisoned texts to hijack answers to a target question 90% of the time, and BadRAG got a 98.2% retrieval hijack rate by controlling 0.04% of the corpus. Memory write paths need the same trust boundaries as any other write API.

The notepad, condensed#

• The model has no write operation. All “memory” is engineering outside it: a store, plus four verbs.
• Four shelves with four failure modes: working memory rots when full, episodic drowns in its own noise, semantic goes stale by collision, procedural breaks when the environment shifts. Diagnose memory bugs by asking which shelf misbehaved.
• Write path: extract few, judge importance once, resolve collisions (ADD / UPDATE / DELETE / NOOP — or version the past like a temporal graph).
• Read path: relevance x recency x importance; log the breakdown per hit; retrieval doubles as rehearsal.
• Forgetting: cache eviction with a psychology upgrade — decay that slows every time a memory proves useful. Nobody ships this yet; that’s a gap, not a settled answer.
• Consolidation: log compaction with an LLM as the compactor; keep provenance.
• Memory trades accuracy for latency, cost, and persistence. Full context beat the best memory system on accuracy in that system’s own paper — 72.9% vs 66.9% — while memory won 10x on speed and cost.

What I still don’t know — and what I want to measure rather than read: which forgetting policy actually wins on a real benchmark? TTL, LRU, and Ebbinghaus-with-rehearsal are all sitting in the lab repo behind one interface, and LongMemEval exists precisely to grade this. If the memory literature has an unguarded open goal, it’s that nobody has published that head-to-head.