AG // agentology
🧠 Anatomy

Memory Systems

How agents remember—from ephemeral context windows to persistent knowledge stores, and the mechanisms that connect past experience to present action.

#architecture #persistence #retrieval #context

Memory systems are the mechanisms through which agents maintain information across time—connecting past observations and actions to present decisions. They represent the extension of cognition beyond the model’s native context window.

The Anthropological Lens

Human societies have developed sophisticated memory systems:

  • Oral tradition: Stories, songs, and rituals that encode knowledge
  • Writing: External storage of information
  • Institutions: Organizations that maintain knowledge across generations
  • Technology: Libraries, databases, the internet

Individual human memory is limited; cultural memory extends it. Agent memory systems face analogous challenges: how to persist information beyond the immediate, and how to make it accessible when needed.

The Context Window Problem

Language models have a fundamental limitation: the context window. All reasoning must occur within this fixed space of tokens.

graph TB
  subgraph CW["CONTEXT WINDOW<br/>everything must fit"]
      SP[System Prompt]
      CH[Conversation History]
      RC[Retrieved Context]
      CI[Current Input]

      SP --> CH --> RC --> CI
  end

  style CW fill:#0a0a0a,stroke:#00ff00,stroke-width:2px,color:#cccccc
  style SP fill:#0a0a0a,stroke:#00ff00,stroke-width:1px,color:#cccccc
  style CH fill:#0a0a0a,stroke:#00ff00,stroke-width:1px,color:#cccccc
  style RC fill:#0a0a0a,stroke:#00ff00,stroke-width:1px,color:#cccccc
  style CI fill:#0a0a0a,stroke:#00ff00,stroke-width:1px,color:#cccccc
context_window_constraint // ~128K tokens

Memory systems extend this by storing information externally and selectively retrieving it when relevant.

Types of Agent Memory

Working Memory

The active context window—what the agent is currently “thinking about.” Limited but immediately accessible.

Characteristics:

  • Fast access
  • Limited capacity
  • Lost between sessions
  • No retrieval needed

Episodic Memory

Records of specific experiences—conversations, tasks, observations. The agent’s autobiography.

Implementation:

  • Conversation logs
  • Action trajectories
  • Timestamped records

Use cases:

  • “What did we discuss last Tuesday?”
  • “How did I solve this problem before?”
  • “What has this user asked for in the past?”

Semantic Memory

General knowledge extracted from experiences—facts, concepts, relationships. The agent’s understanding of the world.

Implementation:

  • Knowledge graphs
  • Vector databases
  • Structured databases

Use cases:

  • “What do I know about this topic?”
  • “What are the properties of X?”
  • “How are A and B related?”

Procedural Memory

Knowledge of how to do things—skills, procedures, strategies. The agent’s capabilities.

Implementation:

  • Tool definitions
  • Code libraries
  • Learned heuristics

Use cases:

  • “How do I accomplish this task?”
  • “What’s the best approach here?”
  • “What steps does this require?”

Memory Architecture

graph TD
  CORE["AGENT CORE<br/>language_model"]
  CORE -->|read/write| WM["WORKING MEMORY<br/>current_context"]

  WM --> EM["EPISODIC<br/>logs<br/>transcripts<br/>history"]
  WM --> SM["SEMANTIC<br/>vector_db<br/>knowledge<br/>facts"]
  WM --> PM["PROCEDURAL<br/>tools<br/>functions<br/>skills"]

  style CORE fill:#0a0a0a,stroke:#00ff00,stroke-width:2px,color:#cccccc
  style WM fill:#0a0a0a,stroke:#00ff00,stroke-width:2px,color:#cccccc
  style EM fill:#0a0a0a,stroke:#00ff00,stroke-width:1px,color:#cccccc
  style SM fill:#0a0a0a,stroke:#00ff00,stroke-width:1px,color:#cccccc
  style PM fill:#0a0a0a,stroke:#00ff00,stroke-width:1px,color:#cccccc
layered_memory_architecture

Retrieval Mechanisms

Having memory isn’t enough—agents must retrieve the right memories at the right time.

Recency-Based

Prioritize recent information. Simple but loses long-term context.

Relevance-Based (RAG)

Retrieve information semantically similar to the current query:

  1. Embed the current context as a vector
  2. Find similar vectors in memory
  3. Include matching content in context

Importance-Based

Weight memories by significance—errors, successes, surprises, explicit markers.

Associative

Follow links between related memories, like human associative recall.

Hierarchical

Organize memories in summaries and details; retrieve at appropriate granularity.

Memory Operations

Encoding

Transforming experiences into storable form:

  • Raw storage (full transcripts)
  • Summarization (compressed representations)
  • Extraction (key facts and entities)
  • Embedding (vector representations)

Consolidation

Processing memories for long-term storage:

  • Deduplication
  • Integration with existing knowledge
  • Importance weighting
  • Pruning obsolete information

Retrieval

Accessing relevant memories:

  • Query formulation
  • Similarity matching
  • Ranking and filtering
  • Context injection

Forgetting

Intentional removal or decay:

  • Privacy requirements
  • Obsolete information
  • Capacity management
  • Error correction

Memory and Identity

Memory is central to identity. An agent with persistent memory develops something like continuity:

  • Remembers past interactions
  • Builds on previous work
  • Maintains consistent preferences
  • Develops “relationships” with recurring users

Challenges

Capacity

Memory grows without bound. What to keep? What to forget?

Retrieval Quality

Finding the right memories at the right time is hard. Too little context is useless; too much is overwhelming.

Consistency

Memories should be coherent. Contradictory memories create confusion.

Privacy

Stored information may contain sensitive data. Memory creates liability.

Staleness

Old memories may no longer be accurate. The world changes; memories don’t.

Trust

Should all memories be trusted equally? Some may be incorrect or adversarial.

The Collective Memory Parallel

Multi-agent systems have collective memory challenges similar to human organizations:

  • Shared knowledge bases: What does the group know?
  • Communication: How is knowledge transmitted?
  • Consistency: Do agents have compatible memories?
  • Specialization: Who knows what?
  • Institutional memory: What persists as agents change?

Organizational memory research offers insights for multi-agent memory design.

Future Directions

Memory systems for agents are rapidly evolving:

  • Learned retrieval: Models that learn when and what to remember
  • Structured memory: Beyond vector similarity to richer knowledge structures
  • Active forgetting: Intelligent pruning and consolidation
  • Memory-augmented architectures: Deep integration of memory with model computation
  • Shared memory protocols: Standards for multi-agent memory

The goal: agents that truly learn from experience, building knowledge over time.

See Also