Graham Neubig · Published 2026-09-08

CMU AI Agents 2026: 3. Long Context Modeling for Agents

privateReview: unreviewedConfidence: high
Context managementKV cacheattentioncompactioninferencecontext-managementkv-cacheprefix-cachinglong-contextpositional-encoding
Open on YouTube ↗

Summary

Overview

  • Speaker: Graham Neubig
  • Channel: Graham Neubig
  • Main topic: Context Management for Long-Context Agents
  • Purpose: To educate students and researchers on the computational, architectural, and data management challenges of maintaining and scaling long context in AI agents. Graham Neubig discusses context management for long-context AI agents at Carnegie Mellon University. The lecture covers how agent context grows quadratically due to linear history growth, long-context architectural challenges like capacity and efficiency, LLM inference serving stacks (prefill vs. decode, KV caching, serving metrics like TTFT and TPOT), hybrid and long-context architectures (sliding-window attention, linear attention, DeltaNet, Gated DeltaNet, Kimi Delta attention, sparse attention, KV compression), training strategies (short-to-long training, positional encodings, RoPE, position interpolation, YaRN), data mixtures for long context, context parallelism, prompt design for cache reuse, and context compaction policies.

Topic Map

Agent Context Growth and Quadratic Input Scaling

  • Explanation: Explains how agent histories grow linearly with conversation/tool calls, resulting in quadratic input token processing when incorporating fixed prefixes and cumulative histories.
  • Key claims:
    • Agent input token usage scales quadratically based on linear history growth.
    • A coding agent session can easily accumulate nearly 80,000 tokens on average across multiple tool calls and tool results.
  • Examples:
    • OpenHands coding agent sessions averaging 77,922 tokens per session across 1,500 sessions.
  • Terminology:
    • quadratic input
    • system prompt
    • tool calls
    • tool results
  • Why it matters: Demonstrates why naive context retention quickly leads to massive computational bottlenecks and high token costs.

LLM Inference Serving Stack and Metrics

  • Explanation: Details the inference serving pipeline, breaking down prefill and decode phases, KV cache usage, serving metrics like TTFT (Time to First Token) and TPOT (Time Per Output Token), and throughput/cost tradeoffs.
  • Key claims:
    • Prefill phase processes all prompt positions and writes to the KV cache, while decode generates one new token per step.
    • Longer input prompts significantly increase TTFT (Time to First Token).
  • Examples:
    • OpenRouter pricing and latency benchmarks across various LLM providers (Mistral, DeepSeek, Reka, etc.).
  • Terminology:
    • prefill
    • decode
    • KV cache
    • TTFT
    • TPOT
    • throughput
  • Why it matters: Understanding serving infrastructure is crucial for deploying cost-effective, low-latency agents at scale.

Long-Context Architectures: Attention variants and Hybrid Models

  • Explanation: Covers architectural modifications to standard O(n^2) global attention, including sliding-window attention, linear attention, DeltaNet, Gated DeltaNet, sparse attention, and KV compression (MQA, GQA, MLA).
  • Key claims:
    • Standard attention computes pairwise comparisons across the entire sequence, leading to O(n^2) complexity.
    • Linear attention replaces softmax to allow recurrent state updates in constant time O(1) per step.
    • Gated DeltaNet and Kimi Delta attention introduce per-channel forgetting and targeted memory corrections.
  • Examples:
    • Qwen3.8-Flash-Next, GLM-5.3-Flash, Kimi K3, and Nemotron Ultra architectures.
  • Terminology:
    • sliding-window attention
    • linear attention
    • DeltaNet
    • Gated DeltaNet
    • sparse attention
    • KV compression
    • MLA
  • Why it matters: Architectural innovations allow models to handle hundreds of thousands or millions of tokens without running out of memory or compute.

Training Long-Context Models and Positional Encodings

  • Explanation: Explores short-to-long pretraining curricula, positional encoding strategies (RoPE, NoPE), position interpolation, and YaRN for extending context lengths.
  • Key claims:
    • Training directly on 128k+ tokens is limited by available natural data, requiring short-to-long curricula.
    • Rotary Position Embeddings (RoPE) encode relative position via rotation and can be extended using position interpolation or YaRN.
  • Examples:
    • Pretraining on 4K-8K tokens, continuing training on 32K-128K packed documents, and long adaptation with synthetic curricula.
  • Terminology:
    • RoPE
    • NoPE
    • position interpolation
    • YaRN
    • packed sequences
  • Why it matters: Enables pre-trained models to effectively generalize to much longer context windows without full retraining from scratch.

Prompt Design for Cache Reuse and KV Caching Mechanics

  • Explanation: Explains how prompt caching works in serving engines, the massive cost and speed advantages of cached KV prefixes, and prompt engineering best practices to preserve cache hits.
  • Key claims:
    • Cached tokens cost roughly 1/10th the price of uncached input tokens on major API providers.
    • Any change to the system prefix or prompt structure invalidates the cache, destroying cost savings.
  • Examples:
    • PagedAttention (vLLM, SGLang) and RadixAttention for prefix sharing and block-table KV allocation.
  • Terminology:
    • KV cache
    • prefix caching
    • PagedAttention
    • RadixAttention
    • cache hit rate
  • Why it matters: Proper prompt design and caching infrastructure can reduce agent operational costs by up to 90%.

Context Compaction and State Estimation

  • Explanation: Examines methods for compressing long agent history into working states (anchors, checkpoints, recent tails) to prevent context window overflow.
  • Key claims:
    • Compacting history into a working state preserves task success while reducing token overhead.
    • Repeated compaction can lead to drift or forgetting critical early instructions if not managed carefully.
  • Examples:
    • Compacting CI failure logs and tool observations into structured summaries with preserved goals and evidence.
  • Terminology:
    • context compaction
    • anchors
    • checkpoint
    • recent tail
    • evidence store
  • Why it matters: Prevents long-running agents from hitting hardware limits or suffering from attention dilution in extremely long sessions.

Key Points

Agent input token complexity is quadratic

  • Explanation: As agent loops accumulate instructions, actions, tool calls, and observations, each step reprocesses previous history plus new inputs, leading to quadratic scaling in token processing.
  • Evidence: OpenHands session data showing average token usage reaching ~78k tokens per session.
  • Practical implication: Agents require efficient context management strategies like caching, compaction, or sparse attention to remain economically viable.

KV caching provides massive cost and latency savings

  • Explanation: Serving stacks cache the Key-Value states of prompt prefixes, allowing subsequent calls with the same prefix to skip the prefill computation phase.
  • Evidence: API pricing data showing cached input tokens priced at $0.007-$0.014 per million tokens compared to $0.22-$1.44 for regular inputs.
  • Practical implication: Developers must stabilize their system prompts and instructions to maximize cache hit rates.

Positional encodings break down at extreme lengths without adaptation

  • Explanation: Standard absolute or rotary positional embeddings struggle when evaluated on sequence lengths far exceeding their training distribution.
  • Evidence: Needle-in-a-haystack evaluations showing accuracy drops at the bottom of long documents for unadapted models.
  • Practical implication: Techniques like position interpolation, YaRN, or partial RoPE are necessary to safely extend context windows.

Frameworks, Models & Processes

The Long-Context System Stack

  • How it works: Manages agent context across five layers: Growth (agent loops ingesting history), Inference (prefill, decode, attention bottlenecks), Architecture (local, recurrent, sparse, and compressed mechanisms), Caching (stable prefixes avoiding redundant prefill), and Compaction (preserving intent and work state).
  • Components:
    • Growth
    • Inference
    • Architecture
    • Caching
    • Compaction
  • When to use: When designing or deploying production-grade LLM agents operating over extended multi-turn sessions.

Context Compaction Policy

  • How it works: Triggers compaction upon token threshold overflow or provider limits, selects regions to protect (beginning and recent tail) while compacting older regions into structured summaries, and replaces the raw history while retaining recovery links.
  • Components:
    • Trigger
    • Select
    • Replace
    • Recover
  • When to use: When agent execution spans hundreds of turns and exceeds model context limits.

Examples & Case Studies

An agent running a coding task accumulates 24k-line logs, tool outputs, and multiple iterations.

  • Illustrates: Rapid agent context bloat during debugging tasks.
  • Lesson: Raw logs and intermediate observations must be compacted into structured summaries (goal, constraints, evidence) to keep context manageable.

An agent session forgets an initial system instruction because compaction discarded early turns.

  • Illustrates: Compaction drift and anchor forgetting.
  • Lesson: Always protect foundational anchors and instructions at the beginning of the context window during compaction.

Actionable Takeaways

  • Immediate:
    • Stabilize system prompts and tool descriptions to maximize KV cache hit rates.
    • Avoid appending dynamic timestamps or changing state in cached prefix regions.
  • Strategic:
    • Combine sliding-window attention or linear attention models with cached prefill for optimal agent throughput.
    • Implement robust context compaction policies to prevent agents from going off the rails during long sessions.
  • Questions to investigate:
    • How do open-source KV caching layers compare to proprietary cloud provider cache eviction policies?
    • What is the exact trade-off between compaction frequency and agent task completion success rates?

Claims Worth Verifying

  • Cached input tokens are typically 1/10th the price of un-cached input tokens across major API providers. (empirical pricing claim)
  • Linear attention models update state in constant time O(1) per token step. (algorithmic complexity claim)

Notable Quotes

"Context Management for Long-Context Agents: How agents keep working as their history grows" (at 0:00) "The input is quadratic from the number of token accesses that you do based on the linear history growth." "Cached input tokens are one tenth of the price of uncached tokens."

Compressed Summary

  • Agent history growth is linear, but token processing complexity is quadratic without optimization.
  • Inference serving relies heavily on prefill/decode separation and KV caching (PagedAttention, RadixAttention).
  • Architectural innovations like sliding windows, linear attention, DeltaNet, and MLA reduce quadratic compute bottlenecks.
  • Prompt engineering for cache reuse requires keeping system prefixes completely static.
  • Context compaction condenses historical agent trajectories into working states while preserving anchors and recovery pointers.
  • Keywords: context management, kv cache, attention, compaction, inference
  • Core insight: Managing long-context agents requires a multi-layered system stack combining efficient attention architectures, KV caching, prompt stabilization, and strategic context compaction.

Core insights

6
Empirical Resultmedium noveltystrong evidence

Agent loops produce quadratic input-token growth: because each step reprocesses the accumulated history plus new instructions/observations, token processing scales with the square of history length rather than linearly. The effect is large in practice — OpenHands coding sessions averaged 77,922 tokens across 1,500 sessions.

Why it matters

Context management is not a polish step but a first-order driver of agent cost and latency; without caching, compaction, or sparse attention, long-running agents become economically unviable at realistic session lengths.

Generalization

Any iterative system that resends a growing state to a fixed-cost-per-token processor inherits quadratic work; the mitigation must live in the runtime (cache/compaction), not in the prompt text.

Agent input token usage scales quadratically based on linear history growth.
Open source video
OpenHands coding agent sessions averaging 77,922 tokens per session across 1,500 sessions.
Open source video
Agents require efficient context management strategies like caching, compaction, or sparse attention to remain economically viable.
Open source video
Practicehigh noveltystrong evidence

Prompt layout is a cache-layout constraint: KV prefix caching makes repeated prefixes skip prefill, with cached tokens priced roughly 1/10th of uncached input, but any mutation of the system prefix or prompt structure invalidates the cache entirely. Cost optimization therefore becomes a prompt-structure invariant (stable immutable prefix, volatile content appended last).

Why it matters

A one-token change near the front of a prompt can silently destroy the majority of an agent's cost savings, so prompt construction must be governed by cache topology rather than by authoring convenience.

Generalization

Wherever content-addressed or prefix-keyed caches sit in front of expensive compute, upstream content ordering becomes a correctness-adjacent performance contract enforced by the platform, not just a style choice.

Cached tokens cost roughly 1/10th the price of uncached input tokens on major API providers.
Open source video
Any change to the system prefix or prompt structure invalidates the cache, destroying cost savings.
Open source video
Proper prompt design and caching infrastructure can reduce agent operational costs by up to 90%.
Open source video
Failure Modemedium noveltymoderate evidence

Context compaction is effectively state estimation: long history is compressed into a working state (anchors, checkpoints, recent tail, evidence store) rather than truncated. It preserves task success while cutting token overhead, but repeated compaction introduces drift and loss of early instructions — a silent correctness failure rather than a capacity failure.

Why it matters

Compaction trades a visible resource limit for an invisible behavioral regression; without drift measurement, an agent can look healthy on token budgets while quietly violating instructions set in its first turns.

Generalization

Any lossy summarization of agent state needs an explicitly preserved, non-compressible subset (goals, evidence) plus a drift metric, otherwise reliability degrades monotonically with session length.

Compacting history into a working state preserves task success while reducing token overhead.
Open source video
Repeated compaction can lead to drift or forgetting critical early instructions if not managed carefully.
Open source video
Prevents long-running agents from hitting hardware limits or suffering from attention dilution in extremely long sessions.
Open source video
Mechanismmedium noveltymoderate evidence

Frontier long-context architectures are converging on hybrid and gated-recurrent designs rather than pure global attention: sliding-window, linear attention, sparse attention, KV compression (MQA/GQA/MLA), and DeltaNet-family models. The distinguishing mechanism in gated variants is per-channel forgetting plus targeted memory corrections inside the recurrent state.

Why it matters

Attention variant choice constrains what a model can recall at long range and how much state it must carry; gated per-channel forgetting means the state is actively edited, not merely accumulated, which changes recall characteristics for retrieval-heavy agent workloads.

Generalization

Constant-time recurrent state with learned erase/write gates is a general pattern for bounded-memory sequence processing, applicable beyond LLMs to any streaming state store.

Linear attention replaces softmax to allow recurrent state updates in constant time O(1) per step.
Open source video
Gated DeltaNet and Kimi Delta attention introduce per-channel forgetting and targeted memory corrections.
Open source video
Architectural innovations allow models to handle hundreds of thousands or millions of tokens without running out of memory or compute.
Open source video
Practicelow noveltymoderate evidence

Long-context capability is a training-curriculum problem, not only an architecture problem: natural 128k+ documents are scarce, so practice is short-to-long pretraining (4K-8K), continued training on 32K-128K packed documents, then synthetic long adaptation. RoPE-based models are extended via position interpolation or YaRN rather than retrained from scratch.

Why it matters

It explains why supported context lengths are a data-availability artifact, and why position-extension techniques are the cheap path to longer windows — useful when selecting or self-hosting models.

Generalization

Capability limits are frequently data-distribution limits; curriculum plus positional rescaling is a cheaper lever than architectural change when extending any positional/indexed model.

Training directly on 128k+ tokens is limited by available natural data, requiring short-to-long curricula.
Open source video
Rotary Position Embeddings (RoPE) encode relative position via rotation and can be extended using position interpolation or YaRN.
Open source video
Pretraining on 4K-8K tokens, continuing training on 32K-128K packed documents, and long adaptation with synthetic curricula.
Open source video
Mental Modelmedium noveltystrong evidence

Serving-phase asymmetry: prefill processes all prompt positions and writes the KV cache, while decode emits one token per step. Consequently prompt length drives TTFT, while decode drives TPOT — meaning agent latency for long contexts is dominated by prefill, which is exactly the phase caching and compaction eliminate.

Why it matters

It localizes the optimization target: for interactive agents, reducing uncached prompt length improves perceived responsiveness directly, and compaction versus caching can be reasoned about as two ways to shrink the same prefill term.

Generalization

Separating a bulk parallel setup phase from a serial incremental phase yields a reusable performance model (setup cost scales with input size, incremental cost with output size) for any inference or batch service.

Prefill phase processes all prompt positions and writes to the KV cache, while decode generates one new token per step.
Open source video
Longer input prompts significantly increase TTFT (Time to First Token).
Open source video

Deep dives

5

Compaction drift and instruction retention

Research question

Does instruction retention measurably decay as a function of number of compaction cycles, and can an evidence store or pinned anchors arrest the decay?

Why

Compaction trades a visible token budget for an invisible behavioral regression; without drift metrics, long-running agents can look healthy while violating instructions set in their first turns.

Repeated compaction can lead to drift or forgetting critical early instructions if not managed carefully.
Open source video
Compacting history into a working state preserves task success while reducing token overhead.
Open source video
Source video

Achievable prefix cache-hit rate under volatile tool results

Research question

What practical cache-hit rates are achievable for agents that must inject volatile tool results and observations every turn?

Why

If volatile observations routinely invalidate prefixes, the promised cached-token discount and cost reduction may be unrealizable for multi-tool agent traces, making prompt-layout discipline a first-order engineering constraint.

Any change to the system prefix or prompt structure invalidates the cache, destroying cost savings.
Open source video
Cached tokens cost roughly 1/10th the price of uncached input tokens on major API providers.
Open source video
Source video

Crossover between application-level compaction and engine-level prefix caching

Research question

At what session length and call frequency does application-level compaction outperform relying on prefix caching alone, and when does a combined policy dominate both?

Why

Compaction and caching both target the same prefill cost term but at different layers; choosing without a crossover model leads to either wasted tokens or cache-invalidation churn.

Prefill phase processes all prompt positions and writes to the KV cache, while decode generates one new token per step.
Open source video
Longer input prompts significantly increase TTFT (Time to First Token).
Open source video
Source video

Empirical shape of quadratic agent token growth

Research question

Does measured token consumption across real agent traces fit a quadratic in step count, or is the growth moderated by truncation and caching in practice?

Why

If the quadratic curve is dampened by early truncation or caching, priority shifts to measuring the actual exponent and where it bends; if not, runtime mitigation becomes mandatory rather than optional.

Agent input token usage scales quadratically based on linear history growth.
Open source video
OpenHands coding agent sessions averaging 77,922 tokens per session across 1,500 sessions.
Open source video
Source video

Recall quality of gated linear attention in agent transcripts

Research question

Do hybrid and gated linear-attention variants match softmax attention on exact long-range recall required by agent tool histories?

Why

If per-channel forgetting edits recurrent state, retrieval-heavy agent workloads may lose exact tool outputs, file paths, or constraints even when the overall token budget fits comfortably.

Linear attention replaces softmax to allow recurrent state updates in constant time O(1) per step.
Open source video
Gated DeltaNet and Kimi Delta attention introduce per-channel forgetting and targeted memory corrections.
Open source video
Source video

Article ideas

4

Prompt Layout Is a Cache Contract, Not a Style Choice

Agent teams should treat the prompt head as a versioned immutable resource; otherwise routine edits or dynamic content silently invalidate prefix caches and erase the dominant cost savings.

Angle

Show how a one-token change near the front can destroy most of an agent's cost reduction, and propose prefix-stability review, cache-hit observability, and regression tests as platform requirements.

Source video

Context Compaction Needs Drift Metrics, Not Just Token Budgets

Compaction converts a visible resource limit into an invisible correctness risk; teams should measure instruction retention per compaction cycle and preserve non-compressible anchors and evidence.

Angle

Treat compaction as state estimation with a drift metric, and evaluate policies by task success per token rather than token reduction alone.

Source video

The Prefill Is the Product: Why Agent Latency Lives Before the First Token

For interactive long-context agents, the dominant latency lever is the uncached prefill term; caching and compaction are two ways to shrink it, and prompt length is a product decision, not just an infra detail.

Angle

Explain prefill vs decode, TTFT vs TPOT, and localize optimization to prompt length, which agents control through caching and compaction.

Source video

Quadratic Agent Growth Is an Architectural Constraint, Not a Prompt Problem

Naive history retention makes token processing superlinear; the fix must live in runtime layers such as caching, compaction, and sparse attention, not in more clever prompt wording.

Angle

Connect realistic 77,922-token coding sessions to the three-layer mitigation stack at application, serving, and model-architecture levels.

Source video

Project ideas

4

Compaction Drift Harness

beyond-evals

Instruction retention measurably decays as a function of compaction cycle count, and an evidence store plus pinned anchors reduces decay without increasing token overhead beyond a controlled budget.

Proof of concept

Build a long-horizon agent benchmark that varies compaction cadence and structure (anchors, checkpoints, evidence store); run identical tasks with and without pinned early instructions, logging token spend, compaction events, and instruction-violation failures.

Measurement

Instruction-retention rate per compaction cycle, task success per token, drift from original goal and constraints, and token overhead; compare against a no-compaction baseline.

Source video

PrefixGuard Runtime

gatehouse

Enforcing a stable immutable prefix and append-only volatile content increases prefix cache hit rate and reduces cost per agent session compared with free-form prompt assembly.

Proof of concept

Instrument an agent runtime to log prefix-shared tokens versus uncached tokens per step; implement a prompt-layout linter/compiler that rejects prefix mutations, moves volatile content such as timestamps, retrieval, and tool results to the tail, and exposes cache-hit metrics.

Measurement

Cache-hit rate, cached versus uncached token ratio, cost per session, and TTFT; A/B against a free-form prompt baseline on a multi-tool agent trace.

Source video

Prefill Crossover Cost Model

movement-lab

There is a measurable crossover in session length and call frequency where compaction-only outperforms cache-only, and a combined policy dominates both under realistic agent traces.

Proof of concept

Build a cost and TTFT simulator for cache-only, compaction-only, and combined strategies as functions of session length, prefix stability, and call frequency; validate against a production agent trace with measured token counts and latencies.

Measurement

Predicted versus observed cost and TTFT, crossover-point accuracy, and best-policy selection across session-length bands.

Source video

Gated Recall Probe for Agent Transcripts

new

Gated linear-attention and hybrid models either match or exceed softmax attention on exact long-range recall embedded in realistic agent transcripts, or they exhibit predictable recall losses at specific context depths.

Proof of concept

Construct needle and retrieval tasks embedded in realistic tool-call transcripts of 20k-100k tokens, including tool outputs, file diffs, and issue text; evaluate hybrid versus global-attention models on exact recall of tool results, file paths, and constraints.

Measurement

Exact-match recall accuracy by context depth and distractor density, plus latency and memory footprint; compare architectures head-to-head.

Source video

Architectural implications

2

Prefix caching makes the head of the prompt a shared, immutable resource across calls; PagedAttention and RadixAttention exist specifically to share and block-allocate KV state across prefixes.

Before

Prompt construction is treated as free-form text authoring where dynamic content (timestamps, retrieved docs, tool results) can be interleaved anywhere.

After

Prompts are structured as a stable immutable prefix followed by append-only, volatile content, and the layout is treated as a versioned contract with the serving engine.

Consequence

Teams need prompt-layout review, cache-hit-rate observability, and regression tests that detect accidental prefix mutation, since such a mutation is a silent cost/latency regression.

Source video

Context cost reduction is split across two layers: engine-level KV prefix caching and application-level history compaction.

Before

A single 'context window management' concern owned by prompt code, typically implemented as truncating the oldest turns.

After

Two distinct responsibilities — a cache-aware serving/runtime layer and an application-level compaction-and-evidence layer with its own policy and failure modes.

Consequence

Compaction policy becomes a pluggable runtime component (anchors, checkpoints, recent tail, evidence store) rather than ad hoc string trimming, enabling evaluation of policies against task success.

Source video

Tradeoffs and failure modes

3

Context compaction policy

Benefit

Compacting history into a working state preserves task success while reducing token overhead, and prevents agents from hitting hardware limits or suffering attention dilution.

Cost or risk

Repeated compaction can cause drift or forgetting of critical early instructions, degrading behavior without any visible token-budget signal.

Repeated compaction can lead to drift or forgetting critical early instructions if not managed carefully.
Open source video
Source video

Prompt prefix caching

Benefit

Cached tokens cost roughly 1/10th the price of uncached input, and proper prompt design plus caching can cut agent operational costs by up to 90%.

Cost or risk

Any change to the system prefix or prompt structure invalidates the cache, so routine prompt edits or injected dynamic content near the front silently destroy the savings.

Any change to the system prefix or prompt structure invalidates the cache, destroying cost savings.
Open source video
Source video

Naive full-history retention

Benefit

Keeping the complete history maximizes information available to the agent and requires no summarization policy.

Cost or risk

Input token usage scales quadratically with linear history growth, so cost and prefill latency grow superlinearly as sessions lengthen.

Agent input token usage scales quadratically based on linear history growth.
Open source video
Source video

Open questions

4

What compaction policy maximizes task success per token, and how much drift does each policy introduce over long sessions?

Why unresolved

The summary asserts that compaction preserves task success while warning that repeated compaction drifts, but provides no comparative measurement of policies, thresholds, or drift metrics.

Research direction

Build a long-horizon agent benchmark that varies compaction cadence and structure (anchors, checkpoints, evidence store) and measure instruction retention and task success against tokens spent.

Source video

Do gated linear-attention variants (DeltaNet, Gated DeltaNet, Kimi Delta attention) match softmax attention on exact long-range recall required by agent tool histories?

Why unresolved

The summary describes the per-channel forgetting and memory-correction mechanisms but gives no retrieval or recall comparison against global attention.

Research direction

Head-to-head eval of hybrid versus global-attention models on needle/retrieval tasks embedded in realistic agent transcripts of tens of thousands of tokens.

Source video

What practical cache-hit rates are achievable for agents that must inject volatile tool results and observations every turn?

Why unresolved

The summary establishes the pricing ratio and the invalidation rule but does not quantify achievable hit rates for realistic multi-tool agent traces.

Research direction

Instrument an agent runtime to log prefix-shared tokens versus uncached tokens per step and measure achievable hit rate under different prompt layouts.

Source video

At what context length and call frequency does application-level compaction outperform relying on prefix caching alone?

Why unresolved

Both mechanisms are presented as cost reducers targeting the same prefill term, but no crossover point or combined-effect model is given.

Research direction

Model cost and TTFT as a function of session length for cache-only, compaction-only, and combined strategies, then validate against a production agent trace.

Source video

Key claims

8
causalVerification needed

Agent input token usage scales quadratically based on linear history growth.

Evidence

Agent input token usage scales quadratically based on linear history growth.

Question

Does measured token consumption across real agent traces fit a quadratic in step count, or is the growth moderated by truncation and caching in practice?

Source video
comparativeVerification needed

Cached tokens cost roughly one tenth the price of uncached input tokens on major API providers.

Evidence

Cached tokens cost roughly 1/10th the price of uncached input tokens on major API providers.

Question

Does the cached-to-uncached price ratio hold across current providers after recent pricing changes, and what are the cache TTL and storage-cost caveats?

Source video
causalVerification needed

Any change to the system prefix or prompt structure invalidates the cache and destroys the associated cost savings.

Evidence

Any change to the system prefix or prompt structure invalidates the cache, destroying cost savings.

Question

How much prefix modification is tolerated before cache invalidation under PagedAttention/RadixAttention and under provider-side prefix caching?

Source video
causalVerification needed

Repeated compaction can lead to drift or forgetting critical early instructions if not managed carefully.

Evidence

Repeated compaction can lead to drift or forgetting critical early instructions if not managed carefully.

Question

Does instruction retention measurably decay as a function of number of compaction cycles, and can an evidence store or pinned anchors arrest the decay?

Source video
causalVerification needed

Longer input prompts significantly increase time to first token.

Evidence

Longer input prompts significantly increase TTFT (Time to First Token).

Question

How does TTFT scale with prompt length across providers and hardware, and how much of it is removed by prefix cache hits?

Source video
factualVerification needed

Linear attention replaces softmax to allow recurrent state updates in constant time O(1) per step.

Evidence

Linear attention replaces softmax to allow recurrent state updates in constant time O(1) per step.

Question

What is the recall-quality cost of the fixed-size recurrent state versus softmax attention at agent-relevant context lengths?

Source video
causalVerification needed

Training directly on 128k+ tokens is limited by available natural data, requiring short-to-long training curricula.

Evidence

Training directly on 128k+ tokens is limited by available natural data, requiring short-to-long curricula.

Question

How much of the long-context capability gap is attributable to data scarcity versus optimization instability at extreme sequence lengths?

Source video
predictionVerification needed

Proper prompt design and caching infrastructure can reduce agent operational costs by up to 90%.

Evidence

Proper prompt design and caching infrastructure can reduce agent operational costs by up to 90%.

Question

Under what workload assumptions (prefix stability, session length, call frequency) is a 90% reduction actually realized in production?

Source video

Connections

5