How we solved Context Management in Agents — Sally-Ann Delucia

Overview

• Speaker: Sally-Ann Delucia
• Channel: AI Engineer
• Main topic: Context Management and Memory Strategies for AI Agents
• Purpose: To share practical engineering and product lessons on managing context windows, memory, and multi-agent delegation in production AI applications. Sally-Ann Delucia, Head of Product at Arize, shares hard-won lessons from building and scaling the AI agent Alyx over nearly a year. She explores the evolution from prompt engineering to context engineering, detailing why naive context strategies like truncation or LLM summarization fail, and outlines practical solutions including smart truncation, memory stores, sub-agents for offloading heavy tasks, and long-session evaluations.

Topic Map

Introduction and What is Alyx

• Explanation: Introduction of the speaker and Alyx, an AI harness built on top of Arize's observability platform with over 40 skills and advanced planning.
• Key claims:
  • Building successful AI agents is extremely difficult.
  • Alyx is an AI harness used to build AI applications.
• Examples:
  • Alyx features include prompt optimization, data generation, augmentation, and annotations.
• Terminology:
  • AI harness
  • Alyx
  • Arize
• Why it matters: Establishes the speaker's first-hand experience and context for the engineering challenges discussed.

The Problem: Context Engineering over Prompt Engineering

• Explanation: Citing Andrej Karpathy's concept of context engineering, the speaker explains that context is the new engineering problem because LLMs require the right context to perform optimally.
• Key claims:
  • The stack is changing from prompt engineering to context engineering.
  • Too much or too little context hurts performance and increases costs.
• Examples:
  • Agent traces in observability platforms multiply rapidly as user interaction grows.
• Terminology:
  • Context engineering
  • Prompt engineering
  • Token limit
• Why it matters: Frames context management as a critical product and UX problem, not just a technical token-limit challenge.

The Vicious Loop and Failed Strategies

• Explanation: Discussion of the vicious loop where agent spans grow, hit context limits, fail, and retry with the same bloated data, as well as why naive truncation and LLM summarization failed.
• Key claims:
  • Naive truncation causes agents to forget everything and break reasoning.
  • LLM summarization is too inconsistent, lacks control over importance, and is unreliable.
• Examples:
  • Taking the first 100 characters of a long context blob caused follow-up questions to look like new conversations.
• Terminology:
  • Naive truncation
  • LLM summarization
  • Context limit
• Why it matters: Demonstrates what does not work when trying to fit growing conversation history into model windows.

Escaping the Loop: Smart Truncation and Memory

• Explanation: Presenting the working solution used in Alyx combining smart truncation (keeping head and tail, removing duplicates) with an external memory store.
• Key claims:
  • Context decides what the model sees; memory decides what survives.
  • Smart truncation preserves the head and tail while retrieving middle content by ID from memory.
• Examples:
  • Keeping latest results, avoiding system prompt resets, and truncating the middle of long tool calls.
• Terminology:
  • Smart truncation
  • Memory store
  • Head and tail strategy
• Why it matters: Provides a robust, reproducible pattern for handling context growth in conversational agents.

Handling Long Sessions and Sub-Agents

• Explanation: Addressing the challenges of long user sessions and introducing sub-agents to offload data-intensive tasks like search.
• Key claims:
  • Users rarely restart chats, causing conversations and failures to appear late.
  • Not all context belongs in the same agent; sub-agents offload heavy tasks.
• Examples:
  • Search tasks involving multiple queries and intermediate reasoning are delegated to sub-agents, returning only results to the main conversation.
• Terminology:
  • Long-session evals
  • Sub-agents
  • Trace stack
• Why it matters: Shows how architectural partitioning (sub-agents) solves scaling issues in complex AI workflows.

What We're Still Figuring Out

• Explanation: Open challenges in agent context management including massive contexts hitting provider limits, true long-term memory, and principled context selection heuristics.
• Key claims:
  • Huge contexts still break provider limits when agents operate on agent data.
  • Real long-term memory across sessions and surfaces is still hard and unbuilt.
• Examples:
  • Users moving across different application surfaces expecting continuous memory.
• Terminology:
  • Long-term memory
  • Context budget
  • Context quality metrics
• Why it matters: Highlights the frontier of agent architecture and research areas for AI engineers.

Key Points

Context engineering replaces prompt engineering

• Explanation: Choosing strategically what the model sees matters more than just prompt wording or staying under token limits.
• Evidence: Karpathy's quote on context engineering and observed agent failure modes when context is poorly curated.
• Practical implication: Engineers must design sophisticated context retrieval and filtering mechanisms rather than dumping raw conversation histories.

Naive approaches like truncation and summarization fail

• Explanation: Dropping characters breaks reasoning, and LLM summarization is too inconsistent and unreliable.
• Evidence: First-hand development trials with Alyx where agents completely forgot context or generated erratic summaries.
• Practical implication: Avoid relying solely on basic programmatic cutoffs or asking LLMs to summarize context dynamically without structural control.

Smart truncation paired with external memory works

• Explanation: Keeping the head and tail of conversations while storing intermediate tool calls and messages in an ID-addressable memory store provides balance.
• Evidence: Successful production deployment in Alyx where agents can pull context by ID when needed.
• Practical implication: Implement hybrid architectures where short-term active context is pruned smartly and long-term context is stored externally.

Sub-agents offload heavy workflows

• Explanation: Delegating complex, data-heavy tasks (like search) to sub-agents keeps the main conversation context small and clean.
• Evidence: Reduction of context bloat and failure rates in Alyx after introducing specialized sub-agents.
• Practical implication: Use multi-agent hierarchies where sub-agents handle data-intensive operations and return only distilled summaries.

Frameworks, Models & Processes

Smart Truncation + Memory Strategy

• How it works: Combines selective message retention in the active prompt with an external memory store accessed via unique IDs.
• Components:
  • Head retention (initial context)
  • Tail retention (recent messages)
  • Middle truncation (removed from prompt, stored in memory)
  • ID-based retrieval mechanism
• When to use: When building conversational agents with long chat sessions and heavy tool-call history.

Examples & Case Studies

Applying naive character truncation (first 100 characters) to agent context.

• Illustrates: The failure mode of over-truncation.
• Lesson: The agent forgot everything, and follow-up questions were treated as entirely new conversations.

Delegating search tasks with multiple queries and intermediate reasoning to separate sub-agents.

• Illustrates: The effectiveness of architectural offloading.
• Lesson: Keeping heavy data out of the main conversation keeps the primary agent performant and error-free.

Actionable Takeaways

• Immediate:
  • Stop relying on naive truncation or raw LLM summarization for context windows.
  • Implement head-and-tail context retention with ID-based memory retrieval.
  • Offload data-intensive tasks to sub-agents to keep main conversation context lightweight.
• Strategic:
  • Context management is fundamentally a product and UX problem, not just an engineering one.
  • Build long-session evaluations (e.g., testing the 11th turn after loading 10) to catch context bugs early.
  • Recognize that agents do not fail because of prompts; they fail because of context.
• Questions to investigate:
  • How can we build true long-term memory across disjoint sessions and application surfaces?
  • What are principled context budgets and clear quantitative metrics for context quality?
  • How can we prevent provider limits from breaking multi-agent systems operating on extensive trace data?

Claims Worth Verifying

• Context engineering is more impactful than prompt engineering for agent success. (conceptual)
• Smart truncation combined with external memory eliminates context overflow loops in production agents. (architectural)

Notable Quotes

"The best context strategy is the one that lets your agent remember what matters and forget what doesn't." "Context management is a product + UX problem, not just an engineering one." "Context decides what the model sees. Memory decides what survives." "Agents don't fail because of prompts. They fail because of context."

Compressed Summary

• Context engineering replaces prompt engineering as the critical bottleneck in AI agents.
• Naive truncation and raw LLM summarization fail due to loss of reasoning and inconsistency.
• Smart truncation (head + tail + memory store) successfully manages active conversation bloat.
• Sub-agents isolate data-heavy workflows like search, keeping main agent context small.
• Long-session evaluations are essential for catching context degradation bugs before users do.
• Keywords: context engineering, smart truncation, memory store, sub-agents, agent architecture
• Core insight: AI agents fail due to poor context management rather than prompts, requiring structured head-tail truncation, external memory, and sub-agent delegation to succeed in production.