AI Engineer · Published 2025-07-22

Stop Using RAG as Memory — Daniel Chalef, Zep

privateReview: unreviewedConfidence: high
memoryknowledge graphragembeddingsagentsagent-memoryknowledge-graphvector-databasesemantic-searchentity-schematemporal-graphontologiescontext-retrieval
Open on YouTube ↗

Summary

Overview

  • Speaker: Daniel Chalef
  • Channel: AI Engineer
  • Main topic: Why traditional RAG and vector database approaches fail for AI agent memory, and how domain-specific knowledge graphs solve the problem.
  • Purpose: To demonstrate the limitations of vector-based RAG memory in AI agents and present Zep's domain-specific knowledge graph approach for agent memory. Daniel Chalef explains that there is no one-size-fits-all memory for AI agents. Relying on simple fact-dumping into vector databases (like Redis) leads to semantic search pollution, retrieving irrelevant facts, and hallucinations. Using Zep's temporal knowledge graph framework, developers can model agent memory after their specific business domain using explicit entity types, schemas, and business rules, enabling precise retrieval and context management.

Topic Map

The Problem with One-Size-Fits-All Memory

  • Explanation: Standard memory architectures for AI agents treat memory as a flat collection of facts stored in vector databases, which leads to poor retrieval relevance.
  • Key claims:
    • There is no one-size-fits-all memory.
    • Vector databases dump facts without causal or relational context.
    • Semantic search results in retrieving arbitrarily similar but business-irrelevant facts.
  • Examples:
    • A media assistant remembering random facts like dog names ('Melody') and wake-up times when asked about music preferences.
  • Terminology:
    • vector database
    • semantic similarity
    • semantic recall
    • fact-dumping
  • Why it matters: Poor memory relevance causes AI agents to hallucinate, give inaccurate responses, and fail at maintaining coherent multi-turn context.

Semantic Similarity vs. Business Relevance

  • Explanation: Semantic similarity in embedding space does not equate to business relevance, as embeddings are simple projections without relational or causal structures.
  • Key claims:
    • Semantic similarity is not business relevance.
    • Embeddings lack causal and relational context.
    • Applications need domain-aware memory rather than better semantic search.
  • Examples:
    • Searching for favorite tunes returns irrelevant facts because of semantic closeness in embedding space.
  • Terminology:
    • embedding space
    • causal relationship
    • domain-aware memory
  • Why it matters: Developers must align agent memory structures with their specific business domains rather than relying on generic vector search.

Zep's Domain-Specific Knowledge Graph Solution

  • Explanation: Zep provides a temporal knowledge graph framework that allows developers to model user memory using explicit entity schemas, types, and business rules.
  • Key claims:
    • Developers can define custom entity types and schemas using TypeScript, Pydantic, or Zod.
    • Memory context can be filtered concurrently by specific node types and business rules.
    • Knowledge graphs evolve continuously from every user interaction.
  • Examples:
    • A financial coach application tracking financial goals, debts, and income sources explicitly in Zep's graph.
  • Terminology:
    • knowledge graph
    • temporal knowledge graph
    • entity schema
    • node types
    • ontology
  • Why it matters: Enables developers to retrieve precise, structured, and business-relevant memory context for AI agents.

Key Points

Vector databases fail at agent memory

  • Explanation: Dumping conversation facts into vector stores causes irrelevant recall because semantic similarity does not understand business logic.
  • Evidence: Media assistant recalling dog names when asked about music preferences.
  • Practical implication: Avoid using raw vector RAG for long-term agent memory without structured filtering.

Memory must reflect the business domain

  • Explanation: Custom entities, edges, and business rules allow agents to store and retrieve structured domain objects.
  • Evidence: Financial coach schema defining FinancialGoal, DebtAccount, and IncomeSource.
  • Practical implication: Define explicit schemas for what your agent needs to remember about users.

Zep outperforms full-context retrieval

  • Explanation: Zep's temporal knowledge graph architecture achieves higher accuracy on benchmarks compared to full-context or standard vector methods.
  • Evidence: Zep SOTA paper benchmarks showing accuracy improvements of up to 18.5%.
  • Practical implication: Use structured graph memory to improve agent reliability and response quality.

Frameworks, Models & Processes

Zep Temporal Knowledge Graph Framework

  • How it works: Extracts entities and facts from user interactions, builds a continuously evolving knowledge graph, and allows developers to query and filter memory using domain schemas and node labels.
  • Components:
    • Entity schemas
    • Node labels
    • Temporal graph
    • Concurrent graph search filters
  • When to use: When building AI agents that require accurate, domain-specific long-term memory and context retrieval.

Examples & Case Studies

A media assistant mixes up music preferences with dog names and wake-up times.

  • Illustrates: The failure of flat vector search memory to maintain domain-specific relevance.
  • Lesson: Irrelevant facts pollute memory and lead to hallucinated or inaccurate agent responses.

A financial coach app defines entity schemas for financial goals, debts, and income sources in TypeScript.

  • Illustrates: How developers can model memory around business domains using Zep.
  • Lesson: Explicit schemas enable precise, multi-query filtering of agent memory.

Actionable Takeaways

  • Immediate:
    • Stop treating vector stores as a complete solution for agent memory.
    • Define custom entity schemas for your agent's domain.
  • Strategic:
    • Model memory around business logic and relational graphs rather than raw semantic similarity.
    • Adopt temporal knowledge graphs for improved agent accuracy and lower hallucination rates.
  • Questions to investigate:
    • How does Zep handle updates and contradictions in temporal knowledge graphs?
    • What is the performance overhead of knowledge graph extraction compared to flat vector embedding?

Claims Worth Verifying

  • Zep achieves accuracy improvements of up to 18.5% over baseline methods on LongMemEval. (benchmark)

Notable Quotes

"I'm here today to tell you that there's no one size fits all memory." (at 0:15) "Semantic similarity is not business relevance." (at 5:24) "We need a solution. We need domain aware memory, not better semantic search." (at 5:46)

Compressed Summary

  • Flat vector RAG leads to semantic pollution and hallucinations in agent memory.
  • Semantic similarity does not equal business relevance.
  • Zep provides a temporal knowledge graph framework allowing developers to define domain entity schemas and business rules.
  • Structured graph filtering enables precise multi-query memory retrieval for AI agents.
  • Keywords: memory, knowledge graph, rag, embeddings, agents
  • Core insight: AI agent memory must be modeled around domain-specific schemas and knowledge graphs rather than relying on flat vector similarity RAG.

Core insights

5
Mental Modelmedium noveltystrong evidence

Semantic similarity is not business relevance. Embedding-space closeness cannot substitute for domain-specific meaning when retrieving agent memory, so vector memory alone retrieves arbitrarily similar but business-irrelevant facts and can cause hallucinations.

Why it matters

Engineers commonly treat vector search as the default memory/relevance layer. This insight reframes retrieval relevance as a domain modeling problem: semantic distance is only a proxy, not the ground truth for what an agent should remember.

Generalization

Any retrieval system for agent context that relies only on embedding distance should add domain constraints, relational structure, or business rules before surfacing results to the model.

Semantic similarity is not business relevance.
Open source video
Dumping conversation facts into vector stores causes irrelevant recall because semantic similarity does not understand business logic.
Open source video
Architecturehigh noveltymoderate evidence

Agent memory should be modeled as an explicit domain ontology: custom entity types, relations, schemas, and business rules defined by developers, rather than as a flat collection of extracted facts.

Why it matters

This moves memory design from generic infrastructure choice to application architecture. The memory layer becomes a typed, queryable model of the business domain instead of an unstructured text store.

Generalization

Before choosing a memory technology, product engineers should specify the entities and relations that matter in their domain; the memory schema is an application-level contract, not an afterthought.

Developers can define custom entity types and schemas using TypeScript, Pydantic, or Zod.
Open source video
Model memory around business logic and relational graphs rather than raw semantic similarity.
Open source video
Mechanismmedium noveltystrong evidence

A temporal knowledge graph can provide the missing causal and relational context for agent memory. Because embeddings do not record how facts connect to entities, events, or business decisions, a graph representation is an architectural substitute for flat vector fact storage.

Why it matters

The value of graph memory is not storing more data; it is preserving the relational structure that determines why a fact is relevant to the current context. Relation edges are actionable context.

Generalization

When memory must support reasoning over a user's evolving state, store connections and temporal structure explicitly rather than hoping semantic similarity will reconstruct them ad hoc.

Embeddings lack causal and relational context.
Open source video
Vector databases dump facts without causal or relational context.
Open source video
Mechanismhigh noveltymoderate evidence

Knowledge graph memory can be continuously built from interactions and queried through concurrent filters on entity types and business rules, yielding precise, multi-query context selection rather than one broad semantic top-k.

Why it matters

The practical retrieval pattern is not better embeddings; it is structured graph traversal constrained by domain predicates, which aligns retrieved context with the agent's current task.

Generalization

Implementations of long-term memory should support multiple simultaneous retrieval predicates over typed entities and relationships, not a single cosine-similarity query over raw conversation history.

Memory context can be filtered concurrently by specific node types and business rules.
Open source video
Knowledge graphs evolve continuously from every user interaction.
Open source video
Empirical Resultmedium noveltyweak evidence

Structured temporal knowledge graph memory can beat both full-context and vector retrieval on long-term memory benchmarks, with reported accuracy improvements up to 18.5%; the methodology is not disclosed in the summary, so this is a claim to verify.

Why it matters

It quantifies the practical payoff of domain-aware memory: structured retrieval over extracted graph state can improve answer correctness relative to simpler memory baselines.

Generalization

Memory architecture choices should be benchmarked against full-context and vector-only baselines; business-structured retrieval may provide substantial gains without extending context windows.

Zep achieves accuracy improvements of up to 18.5% over baseline methods on LongMemEval.
Open source video

Deep dives

4

Semantic similarity vs. business relevance in agent memory retrieval

Research question

How can we reliably measure and anticipate when embedding-space similarity diverges from business relevance in agent-memory retrieval?

Why

If retrieval surfaces semantically similar but business-irrelevant facts, agent context is polluted and hallucinations follow. Understanding this divergence is necessary to design domain-aware memory systems rather than relying on better embedding similarity.

Semantic similarity is not business relevance.
Open source video
Semantic search results in retrieving arbitrarily similar but business-irrelevant facts.
Open source video
Source video

Temporal knowledge graph consistency under user corrections and contradictions

Research question

What mechanisms allow a temporal agent memory graph to remain consistent when users contradict or change previously stored facts?

Why

A graph that evolves from every interaction is only trustworthy if it can handle retractions, updates, and conflicting statements. The source material claims continuous evolution but leaves conflict resolution unspecified, which is a critical gap for production agents.

Knowledge graphs evolve continuously from every user interaction.
Open source video
Vector databases dump facts without causal or relational context.
Open source video
Source video

Schema granularity tradeoffs for domain-specific agent memory

Research question

How does schema richness affect retrieval precision and coverage on long-horizon agent tasks, and what is the right balance?

Why

The recommendation to model memory with explicit entity schemas gives no guidance on how detailed the schema should be. Too little structure reproduces the flat-fact problem; too much structure risks missing unanticipated facts.

Developers can define custom entity types and schemas using TypeScript, Pydantic, or Zod.
Open source video
there is no one-size-fits-all memory
Open source video
Source video

Methodological rigor for knowledge-graph memory benchmark claims

Research question

What experimental controls, baselines, and dataset disclosures are necessary before accepting claims that knowledge graph memory improves long-term agent accuracy?

Why

The 18.5% accuracy improvement claim is presented without disclosed methodology. Replication and fair comparison against full-context and vector baselines are required for the community to adopt graph memory as an evidence-based choice.

Zep achieves accuracy improvements of up to 18.5% over baseline methods on LongMemEval.
Open source video
Source video

Article ideas

4

RAG Is Not Memory: Why Vector Similarity Pollutes Agent Context

Agents should not use semantic vector similarity as their memory layer because embedding-space closeness is uncorrelated with business relevance; memory must be constrained by domain meaning to avoid irrelevant recall and hallucinations.

Angle

Use the media-assistant failure (dog names and wake-up times surfacing for music questions) to demonstrate that top-k vector recall is a context-pollution risk, not a memory architecture.

Source video

What Graph Structure Adds to Agent Memory That Embeddings Cannot

Embedding vectors encode lexical/semantic proximity but not causal, relational, or business-logic context; persisting relationships in a graph gives agents the structural cues needed to retrieve the right memory at the right time.

Angle

Argue from the architectural claim that embeddings lack relational context, and then explain how typed edges and temporal graph evolution substitute a full flat fact store.

Source video

Your Agent Memory Schema Is a Product Decision, Not an Infrastructure Detail

Defining custom entity types, relations, and business rules in code makes memory an application-level contract; teams that design schemas before choosing storage will get more maintainable and relevant agent memory than teams that depend on generic vector recall.

Angle

Treat TypeScript/Pydantic/Zod schemas as typed API contracts for the memory layer, with the financial-goals/debt/income example as evidence of domain modeling.

Source video

Verify the 18.5%: Benchmarking Memory Architectures Honestly

Before adopting knowledge-graph memory because of headline accuracy numbers, teams should demand disclosed baselines and experimental controls; honest comparisons against full-context and vector-only memory will determine whether structured retrieval is genuinely superior.

Angle

Skeptical methodology piece around the reported LongMemEval result, using pass-1 insight that the methodology is undisclosed and the claim is weak until verified.

Source video

Project ideas

4

Relevance-vs-Similarity Retrieval Probe

beyond-evals

Entity-constrained graph filtering yields higher business-relevance precision at top-k than pure vector top-k on the same conversation history and question set.

Proof of concept

Generate a simulated financial-coach conversation history containing goals, debts, incomes, and intruder facts with high semantic similarity to music or lifestyle queries. Build two retrieval paths: (a) flat vector top-k over extracted facts; (b) typed graph memory with entity nodes, business rules, and concurrent filters. Query both systems with a fixed set of user questions and label retrieved memories for business relevance.

Measurement

Precision@k and recall of business-relevant facts over 50 questions, plus end-to-end answer accuracy with a fixed LLM.

Source video

Temporal Retraction Stress Test

movement-lab

A memory graph that stores provenance, timestamps, and fact validity will reflect the latest user correction significantly more often than an append-only vector store after the same contradictory conversation.

Proof of concept

Script dialogues in which a user states a fact, later corrects it, and then asks questions that would be affected by the stale fact. Store the same facts in (a) an append-only vector memory and (b) a temporal graph with versioned nodes and per-turn provenance. Query both after varying numbers of intervening turns.

Measurement

Percentage of responses reflecting the latest fact after 5, 10, and 20 intervening turns on a set of 100 corrected-fact prompts.

Source video

SchemaDepth: Ontology Granularity Study

new

A moderate domain schema with core entity types and business rules achieves a higher combined precision/coverage score than either no schema or an over-specified schema on long-horizon agent queries.

Proof of concept

Define three memory schemas for the financial-coach domain: minimal (no explicit entities), core (goals, debts, income sources), and fine-grained (adds interest rates, payment deadlines, employer details). Run the same graph extraction and retrieval pipeline under each schema, then answer fixed questions drawn from prior interactions plus questions about unforeseen facts outside the schema.

Measurement

Composite score of retrieval precision/recall and answer coverage for facts inside and outside the schema.

Source video

Graph Memory Cost-Accuracy Profiler

new

On the same conversation corpus, knowledge-graph memory can deliver answer-accuracy gains over vector-only memory without requiring prohibitively higher end-to-end latency or token cost.

Proof of concept

Instrument both a flat vector memory pipeline and a temporal-graph memory pipeline on identical transcripts. Log extraction time, query time, token usage, and answer accuracy for each retrieval path, using the same evaluation questions. Compare the cost required for each additional correct answer.

Measurement

Added latency (ms) and token cost per correct answer relative to the baseline vector pipeline.

Source video

Architectural implications

3

Memory infrastructure should include both a developer-maintained domain schema and a graph representation of extracted facts, rather than only a vector database appended to during conversation.

Before

Agents used vector stores as generic fact repositories without domain-specific structure or business constraints.

After

The memory layer stores typed entities, relationships, and temporal evolution derived from interaction history.

Consequence

Relevant memory becomes queryable by business meaning instead of by embedding similarity alone.

Source video

Retrieval for agent context should be a structured query step over graph entities and relations, not a single semantic similarity call.

Before

Context assembly was a top-k vector search over chunks or facts.

After

Context assembly applies concurrent filters by node type and business rule, then returns only the matching domain objects.

Consequence

More precise context selection can reduce irrelevant facts and related hallucinations in agent responses.

Source video

Entity schemas and memory structure are developer-defined code, similar to typed contracts or database schemas.

Before

Memory was implicitly defined by arbitrary facts or transcripts.

After

The application defines, in TypeScript/Pydantic/Zod, which entities the memory system should extract and store.

Consequence

Memory becomes more maintainable, reviewable, and aligned with product requirements.

Source video

Tradeoffs and failure modes

3

Generic vector memory versus domain-specific knowledge graph memory

Benefit

Domain-aware memory retrieves business-relevant facts and can improve agent accuracy by up to 18.5%.

Cost or risk

There is no universal memory schema; teams must invest in modeling their domain before the memory layer works well.

There is no one size fits all memory.
Open source video
Source video

Semantic recall breadth versus business relevance

Benefit

Semantic vector search can surface related language even when exact facts are absent.

Cost or risk

It also retrieves semantically similar but business-irrelevant facts, polluting the agent context.

Semantic similarity is not business relevance.
Open source video
Source video

Graph extraction pipeline overhead versus flat embedding storage

Benefit

Graph extraction gives structured, queryable state and higher benchmark accuracy.

Cost or risk

Extra extraction and graph querying may add latency and engineering complexity relative to simple vector insertion.

What is the performance overhead of knowledge graph extraction compared to flat vector embedding?
Open source video
Source video

Open questions

3

How should a temporal knowledge graph handle updates and contradictions when a user changes facts or expresses conflicting statements over time?

Why unresolved

The summary states that the graph evolves continuously from interactions but does not describe mechanisms for conflict resolution or temporal retraction.

Research direction

Evaluate temporal graph stores for consistency under repeated user corrections and stale fact removal.

Source video

What is the operational overhead of knowledge graph extraction versus flat vector embedding in real production systems?

Why unresolved

The claim of higher accuracy is not paired with latency, token cost, or infrastructure cost measurements.

Research direction

Benchmark graph extraction and concurrent graph querying against embedding pipelines for realistic conversation histories.

Source video

How do developers decide the correct level of schema detail for a domain-specific memory graph?

Why unresolved

The summary advocates for explicit entity schemas but gives no guidance for balancing schema specificity with coverage of unanticipated facts.

Research direction

Compare memory systems that use rich schemas against minimal schemas on long-horizon agent tasks.

Source video

Key claims

6
comparativeVerification needed

Zep achieves accuracy improvements of up to 18.5% over baseline methods on LongMemEval.

Evidence

Zep achieves accuracy improvements of up to 18.5% over baseline methods on LongMemEval.

Question

What are the exact baselines, evaluation sets, and experimental controls behind the 18.5% improvement?

Source video
causalVerification needed

Semantic similarity is not business relevance.

Evidence

Semantic similarity is not business relevance.

Question

Can retrieval benchmarks show cases where embedding similarity rank diverges from human or business relevance judgments?

Source video
causalVerification needed

Vector databases dump facts without causal or relational context.

Evidence

Vector databases dump facts without causal or relational context.

Question

Does augmenting vector stores with explicit relational metadata reduce the retrieval failures described in the media assistant example?

Source video
causalVerification needed

Poor memory relevance causes AI agents to hallucinate, give inaccurate responses, and fail at maintaining coherent multi-turn context.

Evidence

Poor memory relevance causes AI agents to hallucinate, give inaccurate responses, and fail at maintaining coherent multi-turn context.

Question

Can causal experiments isolate retrieval pollution as the trigger for hallucinated facts in agent responses?

Source video
comparativeVerification needed

Zep's temporal knowledge graph architecture achieves higher accuracy on benchmarks compared to full-context or standard vector methods.

Evidence

Zep's temporal knowledge graph architecture achieves higher accuracy on benchmarks compared to full-context or standard vector methods.

Question

Which benchmark tasks and baseline implementations were used, and is the comparison reproducible?

Source video
factualVerification needed

Developers can define custom entity types and schemas using TypeScript, Pydantic, or Zod.

Evidence

Developers can define custom entity types and schemas using TypeScript, Pydantic, or Zod.

Question

Does the Zep API actually expose schema definition through these languages in production?

Source video

Connections

5