From Context to Memory: Your Agents Need a Real Memory Layer — Anders Swanson, Oracle

Overview

• Speaker: Anders Swanson
• Channel: AI Engineer
• Main topic: Agent Memory Architecture, Retrieval, and Database Integration
• Purpose: To educate AI engineers on how to transition from transient context windows and ad-hoc scratchpads to a robust, governed, multi-model database memory layer for AI agents. Anders Swanson, developer advocate for Oracle AI Database, explains why context windows and scratchpads (like markdown files or temp files) are insufficient as agent memory. He outlines the core patterns of modern agent memory systems—memory formation, hybrid retrieval, consolidation, temporality, regular evaluation and feedback, and governance controls—and demonstrates how a multi-model database can serve as a robust, scalable memory layer.

Topic Map

The Problem with Context Windows and Scratchpads

• Explanation: Discusses why context windows (which are inputs for a single run, not persistent experiences) and file-based scratchpads (markdown notes, temp files, grep) fail as memory layers for autonomous agents across multi-tenant or long-running workflows.
• Key claims:
  • Context windows are input for the current run and are tossed away at session end, leading to repeated work across sessions.
  • Large context windows dilute signals with irrelevant information (noise, logs, prompt instructions).
  • Scratchpads (files, wikis, grep) do not scale in multi-tenant environments and lack complex hybrid retrieval.
• Examples:
  • Agent Session A reproduces a failure and finds root cause; Session B starts fresh and has to re-read files and guess again.
  • Claude Code or Codex re-reading the same file repeatedly because it has no memory of past runs.
• Terminology:
  • context window
  • scratchpad
  • persistent experience
  • managed state
  • governed recall
• Why it matters: Without persistent memory, agents repeat mistakes, waste tokens, hallucinate more often, and lack fidelity.

Where Modern Agent Memory Systems Converge

• Explanation: Outlines the core recurring pattern across modern agent memory implementations: memory formation, hybrid retrieval, consolidation, temporality, regular evaluation and feedback, and governance controls.
• Key claims:
  • Memory formation turns raw transcripts into distilled, structured experiences and facts, not raw transcript dumps.
  • Hybrid retrieval fuses vector similarity, lexical search, entity search, graph traversals, and temporal/recency signals.
  • Consolidation resolves duplicates and conflicts so the system evolves instead of becoming stale.
  • Temporality tracks creation time, access time, and validity windows of memories.
  • Governance controls handle redaction of secrets, scoping to agents/workflows, and suppressing stale or poisoned data.
• Examples:
  • Redacting PII before embedding or persisting memory.
  • Tracking valid_from and valid_to timestamps on memory records.
• Terminology:
  • memory formation
  • hybrid retrieval
  • consolidation
  • temporality
  • governance controls
  • vector similarity
  • lexical search
• Why it matters: These six patterns form the foundation for building production-grade agent memory that scales and avoids failure.

Walking the Agent Memory Loop

• Explanation: Walks through the operational lifecycle of agent memory: Run, Record, Recall, Adapt.
• Key claims:
  • During a run, an agent executes tasks and generates useful facts.
  • Facts are recorded via redacting, enriching, and storing in a persistent layer.
  • Subsequent sessions recall related context to reconstruct context cards.
  • Feedback scores adapt the memory over time based on successful use or failure.
• Examples:
  • An agent using prior evidence from a past run to make the next task easier and avoid dead ends.
• Terminology:
  • memory loop
  • context card
  • provenance
  • feedback score
• Why it matters: Connecting runtime execution to persistent storage via a structured loop enables continuous improvement across agent sessions.

Memory Read, Write, and Storage Layers

• Explanation: Details the architectural breakdown of agent runtime, memory API layers, and multi-model database storage platforms.
• Key claims:
  • Agent runtime invokes memory as a function via hooks, MCP, skills, or REST APIs.
  • Memory layer API handles write policy, recall assembly, redaction/review, and feedback/eval.
  • Multi-model databases store memory rows, vector embeddings, full-text search indices, graph links, and episodes/facts in one place.
• Examples:
  • Using Oracle AI Database to handle vector search, JSON documents, relational tables, and graph relationships in a single platform.
• Terminology:
  • memory layer API
  • multi-model database
  • vector embeddings
  • full-text search
  • entity graph
  • episodes and facts
• Why it matters: A multi-model database eliminates the need to stitch together multiple disparate data stores for different memory search modalities.

What Does Memory Look Like? Typed Schemas

• Explanation: Breaks down memory into distinct types inspired by human memory: episodic, semantic, procedural, and working state.
• Key claims:
  • Episodic memory stores specific past events, transcript messages, and evidence spans.
  • Semantic memory stores facts, preferences, user preferences, and entities.
  • Procedural memory stores validated routines, reusable rules, processes, and codebases.
  • Working state stores short-term current work, temporal facts, and validity windows.
• Examples:
  • Storing a reusable build step rule in procedural memory for future agent runs.
• Terminology:
  • episodic memory
  • semantic memory
  • procedural memory
  • working state
• Why it matters: Typing memories allows agents to bucket information into different areas and apply targeted search techniques.

Storage Schema and Intentional Memory Formation

• Explanation: Explains the developer-facing model for memory storage objects and the step-by-step memory persistence workflow.
• Key claims:
  • Memory objects include content, metadata JSON, vector embeddings, text search indices, graph links, temporal facts, and audit events.
  • Good memories are intentionally formed through a pipeline: redact, enrich, scope, embed, episode, policy, save.
  • Redaction must happen before embeddings, enrichment, and persistence to strip PII and secrets.
• Examples:
  • A Python data class definition showing type Memory with ID, content, metadata, and embedding fields.
• Terminology:
  • storage schema
  • redaction
  • enrichment
  • scope
  • audit events
• Why it matters: Intentional formation prevents prompt injections, data leaks, and unstructured junk from poisoning the memory layer.

Memory Retrieval: Hybrid Search and Ranked Fusion

• Explanation: Explains how memory recall is a query problem solved via hybrid search fusing vector distance, full-text search, graph traversal, and feedback signals.
• Key claims:
  • Hybrid search combines vector similarity search, lexical keyword search, entity search, and graph traversal.
  • Results from different search branches are assigned weights, normalized, fused, thresholded, and reranked.
  • The output is a context card containing scored evidence with full provenance.
• Examples:
  • Combining VECTOR_DISTANCE, contains operator, entity hits, and fact hits with weighted SQL scoring.
• Terminology:
  • hybrid search
  • ranked fusion
  • context card
  • reranking
  • provenance
• Why it matters: Relying on a single search method (like vector similarity alone) is insufficient for comprehensive context reconstruction.

Consolidation, Error Handling, and Feedback Loops

• Explanation: Discusses failure modes in agent memory (stale memory, poisoned memory, bad scope, conflicts, over-recall, irrelevance, data leakage, silent drift) and countermeasures.
• Key claims:
  • Stale memory occurs when project tooling changes (e.g., switching from npm to another build system); systems must expect stale facts over recall drift.
  • Poisoned memory happens when untrusted sources plant durable prompt injection instructions.
  • Memory conflict happens when two memories give conflicting instructions for the same task.
  • Feedback and evaluations close the memory loop by tracking whether memories were helpful, stale, or harmful to adjust rankings.
• Examples:
  • Logging into ChatGPT and accidentally getting another user's message history due to missing scope controls.
• Terminology:
  • stale memory
  • poisoned memory
  • bad scope
  • memory conflict
  • over-recall
  • data leakage
  • silent drift
  • feedback loop
• Why it matters: Without error handling and feedback loops, growing memory systems degrade in quality, leading to hallucinations, incorrect tool use, and security leaks.

Key Points

Context windows and scratchpads are not true memory

• Explanation: Context windows are single-run inputs that get discarded, and file-based scratchpads like markdown files or grep do not scale or support complex hybrid retrieval.
• Evidence: Agents using tools like Claude Code re-read the same files repeatedly because they lack historical persistence.
• Practical implication: Engineering teams must build dedicated persistent memory layers backed by structured databases rather than relying on growing context windows.

Memories must be typed and intentionally formed

• Explanation: Treating memory as raw transcript dumps leads to noise and failure. Memories should be categorized into episodic, semantic, procedural, and working state.
• Evidence: The memory persistence workflow requires explicit steps: redact, enrich, scope, embed, episode, policy, and save.
• Practical implication: Implement strict redaction and scoping on write to prevent PII leaks and prompt injection attacks.

Hybrid search and ranked fusion are essential for recall

• Explanation: Vector similarity alone is not enough; robust memory retrieval requires fusing vector search, lexical search, graph traversals, and temporal data.
• Evidence: SQL queries combining vector distance, full-text search, and entity graph traversal yield superior context cards with scored evidence.
• Practical implication: Use multi-model databases that support vectors, JSON, full-text, and graph capabilities natively.

Feedback and evaluation close the memory loop

• Explanation: Memory systems cannot be static repositories; they must track usage, penalize unhelpful or harmful memories, and reward successful ones.
• Evidence: If an agent uses a memory and crashes or succeeds, that feedback score must update the memory's ranking and lifecycle state.
• Practical implication: Build evaluation metrics into your memory layer to combat silent drift, stale facts, and over-recall.

Frameworks, Models & Processes

Memory Persistence Workflow

• How it works: A structured pipeline applied when an agent runtime decides to create a memory, ensuring safety, structure, and provenance.
• Components:
  • redact
  • enrich
  • scope
  • embed
  • episode
  • policy
  • save
• When to use: Whenever an AI agent generates new insights, facts, or workflow steps that need to be persisted for future sessions.

Typed Memory Model

• How it works: Categorizes agent memory into distinct psychological and functional types to optimize search and retrieval strategies.
• Components:
  • Episodic memory
  • Semantic memory
  • Procedural memory
  • Working state
• When to use: When designing the database schema and storage taxonomy for agent memory systems.

Examples & Case Studies

An agent working on customer support tickets reads documents, runs queries, reproduces a failure, and finds the root cause.

• Illustrates: Valuable work performed during a single agent session.
• Lesson: Without a memory layer, all this synthesized procedure is tossed away at the end of the session, forcing future agents to repeat the work.

A project switches its package manager from npm to another build system.

• Illustrates: Stale memory and quiet drift in agent memory.
• Lesson: If old memories referencing npm are still recalled with high weight, the agent will make repeated mistakes until consolidation and temporal validity windows are applied.

Actionable Takeaways

• Immediate:
  • Stop relying solely on large context windows and markdown scratchpads for agent state.
  • Implement redaction before embedding and persisting memory records.
  • Adopt hybrid search combining vector, lexical, and graph queries for agent memory recall.
• Strategic:
  • Design agent memory systems around formation, retrieval, consolidation, temporality, evaluation, and governance.
  • Leverage multi-model databases to store vector embeddings, JSON metadata, full-text indexes, and entity graphs in a single place.
  • Close the memory loop by feeding usage feedback back into memory rankings and lifecycle decisions.
• Questions to investigate:
  • How do we automatically detect and prune stale procedural memory when codebases change?
  • What are the latency tradeoffs of running multi-branch hybrid search during agent reasoning steps?
  • How can multi-tenant isolation be strictly enforced across shared agent memory graphs?

Claims Worth Verifying

• Multi-model databases eliminate the need for separate vector databases and graph databases in agent memory architectures. (architectural claim)
• Redaction before embedding is sufficient to prevent all prompt injection attacks in persistent memory layers. (security claim)

Notable Quotes

"A context window is not memory. Context is input for the current run, not persisted experience." (at 0:56) "Memory really cannot begin until these useful experiences from our agents are transformed into managed state." (at 2:26) "Good memories are very intentional. So you need to decide the lifecycle, the scope, the different relationships and associated metadata." (at 13:42) "Feedback is what closes the loop from memory. Memory quality needs to be measurable." (at 25:30)

Compressed Summary

• Context windows are transient run inputs, not persistent agent memories.
• File-based scratchpads and grep do not scale for multi-tenant agent recall.
• Modern agent memory requires formation, hybrid retrieval, consolidation, temporality, evaluation, and governance.
• Memories should be typed into episodic, semantic, procedural, and working state categories.
• Multi-model databases enable unified storage of vectors, text indices, entity graphs, and metadata for robust hybrid search.
• Keywords: agent memory, hybrid retrieval, multi-model database, context window, vector search, governance
• Core insight: AI agents require a governed, multi-model memory layer featuring typed storage, hybrid search, and feedback loops to evolve beyond transient context windows and avoid repeated failures.