Context Engineering in 2026 — Louis-François Bouchard, Omar Solano & Samridhi Vaid, Towards AI

Overview

• Speaker: Louis-François Bouchard, Omar Solano, Samridhi Vaid
• Channel: AI Engineer
• Main topic: Context engineering for long agent sessions, including compaction, memory, retrieval, skills, and prompt caching.
• Purpose: To provide AI engineers with production-tested playbooks, architectural patterns, and empirical benchmark results for managing agent context windows efficiently in 2026. A comprehensive 2026 playbook on context engineering presented by Louis-François Bouchard, Omar Solano, and Samridhi Vaid from Towards AI. The presentation covers challenges in long agent sessions such as context rot, finite context windows, stateless models, and token cost escalation. It explores compaction techniques (trimming, sliding windows, tool-result clearing, summarization, delta summarization), offloading methods (RAG, GraphRAG, offloading to files/memory), and the dramatic impact of prompt caching. Through extensive evaluation on an open-source AI tutor case study across Gemini and DeepSeek models, the speakers share empirical results on cost, latency, recall accuracy, and best practices for production AI engineering.

Topic Map

The Problem with Long Agent Sessions

• Explanation: In long agent sessions, models often fail or do the exact opposite of what is desired. This is typically not because the model gets dumber, but because the context fills up and degrades in quality, leading to context rot.
• Key claims:
  • Long agent sessions suffer from context rot as the window fills up.
  • Model results get worse over time due to context pollution.
  • Finite context windows mean instructions, retrieved lessons, tool use, and code all compete for one attention budget.
• Examples:
  • An agent told 'NEVER push directly to main' eventually executing 'git push origin main' after 45 turns of logs and diffs.
• Terminology:
  • context rot
  • lost in the middle
  • stateless model
• Why it matters: Understanding context rot is foundational to preventing agents from failing during extended, multi-turn interactions.

Context Management vs. Memory

• Explanation: The finite context window and stateless nature of LLMs create two distinct problems to solve: context management within a single session and memory across multiple sessions.
• Key claims:
  • Models are stateless, meaning every call starts from zero between sessions.
  • Context management handles within-session token allocation, while memory handles cross-session persistence.
• Examples:
  • Reopening an AI tutor without background state causes the model to have no idea what occurred previously.
• Terminology:
  • context management
  • memory
  • stateless
• Why it matters: Differentiating single-session context from cross-session memory dictates how engineers design agent architectures.

Compaction Techniques: Trivial Tools to LM-Based

• Explanation: Compaction aims to keep the smallest possible context while retaining enough information to answer user questions. Methods range from non-LM rule-based pruning to language-model-driven summarization and reset.
• Key claims:
  • Compaction keeps what still matters and drops or relocates the rest.
  • Trivial tools like observation truncation, sliding windows, and tool-result clearing require no LLM call.
  • LM-based techniques include selective retention, summarization, delta summarization, and compaction with reset.
• Examples:
  • Truncating a 300-line stack trace to keep only head and tail before it enters chat history.
  • Claude Code's compaction reset approach where the entire history is collapsed into a fresh summary.
• Terminology:
  • compaction
  • observation truncation
  • sliding window
  • tool-result clearing
  • delta summarization
• Why it matters: Effective compaction reduces token costs and latency without sacrificing output quality.

Offloading and Retrieval Strategies

• Explanation: Moving tokens out of the active context window into files, memory, or RAG databases keeps prompts lean while maintaining access to a vast knowledge base.
• Key claims:
  • Offloading persists details to disk or memory and keeps a lightweight pointer in context.
  • Karpathy's LLM wiki approach uses a file-based knowledge base that agents maintain and re-read.
  • Hybrid search (dense embeddings plus BM25 keyword search followed by reranking) outperforms pure semantic search.
• Examples:
  • Using index.md as a map pointing to granular chunk files for the AI tutor corpus.
• Terminology:
  • offloading
  • RAG
  • GraphRAG
  • hybrid search
  • reranking
  • LLM wiki
• Why it matters: Offloading allows agents to leverage millions of tokens of reference material without bloating active context.

Prompt Caching and Economics

• Explanation: Prompt caching is a game-changer that makes cached prefix tokens up to 50x cheaper. However, summarization and rewriting prefixes can break the cache and force full-price recomputation.
• Key claims:
  • Prompt caching makes cached prefix tokens significantly cheaper (up to 50x on DeepSeek).
  • Rewriting or summarizing the prefix breaks the cache, forcing full-price recomputation.
  • Compaction must achieve greater than 50x compression to be worth breaking cache hits.
• Examples:
  • Comparing Gemini 3.5 Flash and DeepSeek V4 Flash caching behaviors and cost impact.
• Terminology:
  • prompt caching
  • cache hit
  • cache miss
  • TTFT
• Why it matters: Prompt caching inverts traditional compaction math; keeping everything and letting the cache handle it is often cheaper and higher quality.

Evaluation Harness and Empirical Results

• Explanation: An automated evaluation harness (run -> grade -> gate -> report) tested 11 presets across single-turn and multi-turn student tasks using real-world AI tutor interactions.
• Key claims:
  • Keeping everything in context won session memory recall tests at 92% vs 38% for compaction methods.
  • Compaction often hurts quality and increases tool calls because agents must re-retrieve discarded information.
  • DeepSeek V4 Flash with caching achieved significantly lower costs per turn while maintaining high recall.
• Examples:
  • Testing 5905 runs across the Gemini study evaluating single-turn and session tasks.
• Terminology:
  • eval harness
  • memory recall
  • hit rate
  • MRR
  • preset
• Why it matters: Empirical testing proves that conventional wisdom regarding aggressive context compaction can be counterproductive under modern prompt caching.

Key Points

Context rot degrades agent performance

• Explanation: As conversation history and tool outputs accumulate, models suffer from attention dilution and lose track of core instructions.
• Evidence: Agent logs showing failure to follow negative constraints ('NEVER push to main') after 45 turns.
• Practical implication: Engineers must actively manage context length or scope within agent loops.

Prompt caching inverts compaction economics

• Explanation: When cloud providers cache system prompts and conversation history, re-sending tokens is extremely cheap.
• Evidence: Cached tokens priced up to 50x cheaper on APIs like DeepSeek.
• Practical implication: Aggressive summarization that breaks cache hits can increase costs and latency instead of reducing them.

Keeping everything wins on memory recall

• Explanation: Across multi-turn evaluation tasks, retaining full conversation history achieved 92% memory recall compared to 38% for summarized compaction methods.
• Evidence: Evaluation benchmark charts comparing full history against compaction presets.
• Practical implication: Do not compact by default; weigh quality, cost, and latency constraints before modifying context.

Hybrid search is essential for local RAG scaling

• Explanation: Dense embeddings alone collapse at scale when facts are buried, whereas hybrid search combining dense and keyword matching maintains 100% recall.
• Evidence: Search benchmark showing dense-only recall dropping to 0% at 400k tokens while keyword/BM25 combined with dense held 100%.
• Practical implication: Implement hybrid search pipelines for large knowledge corpora.

Frameworks, Models & Processes

Context Engineering Architecture

• How it works: Deciding what the model sees on every single API call by managing system prompts, tool definitions, chat history, tool outputs, and retrieved knowledge chunks.
• Components:
  • System prompt
  • Tool definitions
  • Chat history
  • Tool outputs
  • Retrieved chunks & memory
• When to use: When building production LLM agents and multi-turn chatbots.

Evaluation Harness Pipeline

• How it works: An automated testing framework that runs agent tasks, grades outputs using code checks and LLM judges, gates results based on criteria, and generates telemetry reports.
• Components:
  • run_battery
  • grade
  • check_triggers
  • report
• When to use: When optimizing agent prompts, memory systems, and retrieval pipelines offline.

Examples & Case Studies

An agent ignores the rule 'NEVER push directly to main' after 45 turns of conversational noise.

• Illustrates: Context rot in long agent sessions where critical instructions are diluted.
• Lesson: Context windows need active curation or isolation to maintain instruction adherence.

Testing an AI tutor with 8.2M token corpus across 14 sources using hybrid search versus dense-only search.

• Illustrates: Scaling limitations of pure semantic search in large document corpora.
• Lesson: Combine dense retrieval with keyword search (BM25) and reranking for robust grounding.

Actionable Takeaways

• Immediate:
  • Implement observation truncation for large tool outputs like stack traces.
  • Explore prompt caching options provided by your model API to slash input costs.
  • Log all token usage, latency, and cache hit rates using telemetry tools like LangSmith or Opik.
• Strategic:
  • Avoid aggressive compaction by default; evaluate whether prompt caching makes keeping full history cheaper and more accurate.
  • Use hybrid search (dense + keyword + rerank) for grounding agents in large knowledge bases.
  • Build a rigorous evaluation harness before making architectural changes to agent context.
• Questions to investigate:
  • How does prompt caching pricing evolve across different frontier and local models?
  • What is the exact threshold where compaction savings outweigh cache break penalties?
  • How can local SLMs with 32k context windows effectively handle complex multi-turn student interactions?

Claims Worth Verifying

• Cached prefix tokens can be up to 50x cheaper on certain APIs like DeepSeek. (pricing and hardware performance)
• Retaining full conversation history achieved 92% memory recall versus 38% for summarized compaction in multi-turn tests. (benchmark result)
• Dense-only retrieval recall collapses to 0% at 400k tokens for buried facts, while keyword/BM25 hybrid maintains 100%. (technical benchmark)

Notable Quotes

"The model didn't get dumber. Its context did." "If I had to choose just one metric, I'd argue that the KV-cache hit rate is the single most important metric for a production-stage AI agent."

Compressed Summary

• Long agent sessions suffer from context rot and token cost explosion.
• Prompt caching inverts compaction math, making full-history retention cheaper and more accurate.
• Hybrid search (dense + keyword + rerank) is required for reliable document retrieval at scale.
• Rigorous offline eval harnesses are essential for testing agent context strategies.
• Keywords: context engineering, prompt caching, compaction, retrieval, memory
• Core insight: Under modern prompt caching, keeping full context history often outperforms summarization in both memory recall and cost efficiency.