AI Agent Memory: Types, Architecture and Tools

LLMs don't remember anything between calls. Every bit of "memory" an agent has is something your code stored and chose to put back into the prompt. That makes memory an engineering problem, not a model feature. This guide covers the memory types that matter, the architectures that implement them, and how two popular frameworks do it. Every claim links to a primary source.

Why agents need memory at all

The CoALA paper puts it plainly: "Language models are stateless: they do not persist information across calls," whereas language agents "may store and maintain information internally for multi-step interaction with the world" (Sumers, Yao et al., Cognitive Architectures for Language Agents).

Bigger context windows don't remove the problem:

• Long histories hurt quality. The LangGraph docs note that a long history may not fit the window at all, and even when it does, "most LLMs still perform poorly over long contexts," getting "distracted" by stale content while costing more and responding slower (LangChain memory overview).
• The middle of a long context gets lost. The MemGPT authors cite research showing long-context models struggle to use extra context well and recall the start and end of the window better than the middle (Packer et al., MemGPT).

The four memory types

The most useful taxonomy comes from CoALA, which borrows it from cognitive architectures like Soar (CoALA). LangChain's docs use the same mapping (memory overview).

1. Working memory (short-term)

What it is: the "active and readily available information" for the current decision cycle: perceptual inputs, retrieved knowledge, and active goals. CoALA stresses that working memory is a data structure "that persists across LLM calls," not just the prompt. Each LLM call is built from a subset of it, and the output is parsed back into it (CoALA).

In practice: this is your agent's run state: message history, tool results, the current plan. In LangGraph it's the graph state, kept per thread by a checkpointer (LangGraph persistence).

2. Episodic memory

What it is: "experience from earlier decision cycles," such as past trajectories or event histories, which can be pulled back into working memory to support reasoning (CoALA).

In practice: LangChain notes episodic memory is "often implemented through few-shot example prompting." You store past successful runs and pull in the most relevant ones as examples. The hard part is choosing which examples to use for a given input (memory overview).

3. Semantic memory

What it is: "an agent's knowledge about the world and itself." Retrieval-augmented generation (RAG) over a document corpus is a read-only semantic memory, but agents can also write new knowledge into it (CoALA).

In practice: facts about users, projects, and decisions. Don't confuse semantic memory (a kind of content) with semantic search (a retrieval technique). The LangChain docs call this distinction out explicitly (memory overview).

4. Procedural memory

What it is: how the agent does things. CoALA splits it into implicit knowledge in the LLM weights and explicit knowledge in the agent's code. It also warns that letting an agent write to procedural memory "is significantly riskier than writing to episodic or semantic memory," because it can introduce bugs or let the agent get around its designers' intentions (CoALA).

In practice: your system prompt, tools, and code. LangChain notes it's uncommon for agents to change their weights or code, but "more common for agents to modify their own prompts." One way is reflection: feed the current instructions plus user feedback back to the model and save the revised prompt (memory overview).

(The "Risk" column is our editorial summary, not a quote from the sources.)

Architecture patterns

Profile vs collection (semantic memory)

LangChain describes two shapes (memory overview).

Profile: a single JSON document per entity that you update over and over.

• Upside: it gives the model the full picture of the entity in one place.
• Downside: updates become "error-prone as the profile gets larger."

Collection: many small memory documents.

• Upside: each one is easier to generate, and this "tends to lead to higher recall downstream."
• Downside: you now have to delete and update items, and some models "over-insert" while others "over-update."

Hot path vs background writes

Again from LangChain (memory overview):

• Hot path: the agent decides to save a memory before it responds.
  • Upside: memories are available immediately and the user can see what was saved.
  • Downside: more latency, and the agent has to juggle memory work with its main task.
• Background: a separate job extracts memories asynchronously.
  • Upside: no added latency, and memory logic stays separate from the app.
  • Downside: you have to decide how often and when to trigger it.

Start with background writes for anything user-facing where latency matters.

Tiered memory (MemGPT)

MemGPT treats the context window like RAM and outside storage like disk, an operating-system analogy (MemGPT). The main context has three parts:

• read-only system instructions
• a fixed-size, writable "working context"
• a FIFO message queue

Messages live in "recall storage," and arbitrary text lives in "archival storage." The model moves data between tiers itself through function calls.

When the prompt goes past a warning threshold (the paper's example is 70% of the window), the system adds a "memory pressure" message so the model can save what matters. At the flush threshold, it evicts messages and writes a recursive summary.

In the paper's deep-memory-retrieval test on multi-session chat, MemGPT raised GPT-4's accuracy from 32.1% to 92.5% over a baseline that only saw a lossy summary of past sessions (MemGPT, Table 2). That's one benchmark under the paper's setup, not a general guarantee.

Retrieval scoring: recency, importance, relevance

CoALA describes how Generative Agents (Park et al., 2023) pull episodes using a mix of recency (rule-based), importance (rated by reasoning), and relevance (embeddings) (CoALA). CrewAI's memory uses the same three-signal idea, described in the next section.

How frameworks implement it

LangGraph

LangGraph splits memory into two primitives (persistence docs):

• Checkpointers hold short-term memory scoped to one thread, picked by thread_id. InMemorySaver loses everything on restart, so use PostgresSaver in production.
• Stores hold long-term memory across threads.

Long-term memories are JSON documents saved under a namespace (like a folder, often including a user ID) and a key. You can search them by vector similarity plus content filters (memory overview). Deciding what to remember is your job.

CrewAI

CrewAI has a unified Memory class that "replaces separate short-term, long-term, entity, and external memory types" (CrewAI memory docs). Key behaviors:

• Scopes: memories sit in a tree of paths like /project/alpha that works like a filesystem. If you don't pass a scope, an LLM infers one. The docs recommend keeping it 2–3 levels deep.
• Composite recall score: semantic_weight * similarity + recency_weight * decay + importance_weight * importance. Decay is 0.5^(age_days / half_life_days). The default weights are 0.5, 0.3, and 0.2, with a 30-day half-life.
• Consolidation: when a new memory is more than 0.85 similar to an existing one, an LLM decides whether to keep, update, delete, or insert. That handles stale and duplicate facts for you.
• Recall depth: shallow is plain vector search with no LLM call (the docs say about 200 ms). deep (the default) runs a multi-step flow with LLM query analysis for longer queries.
• Privacy controls: you can tag records with a source and mark them private so only matching sources can recall them.
• Defaults to know:
  • embeddings: OpenAI text-embedding-3-large
  • analysis LLM: gpt-4o-mini
  • storage: LanceDB on disk
  • Memory content goes to the LLM for analysis, so use a local model for sensitive data.

A practical starting recipe

1. Make working memory durable first. Use a database-backed checkpointer or persisted state, not an in-memory one.
2. Trim or summarize history before it hits the context limit. MemGPT's warn-then-flush approach is a good model to copy.
3. Add semantic memory as a collection of small facts, scoped per user or project, and write it in the background.
4. Rank retrieval by more than similarity. Recency and importance cut down on stale recall.
5. Plan for updates and deletes from day one. Contradictory facts are the most common memory bug. Consolidation (CrewAI) or an explicit update step (LangGraph) handles them.
6. Treat procedural memory as code. If an agent rewrites its own prompt, version it and review the change.

For framework trade-offs beyond memory, see our LangGraph vs CrewAI comparison.

Agent Signal is a weekly newsletter for people building AI agents. Subscribe to get the next deep dive.