AI Engineer · Published 2026-09-09

It’s Tokens All The Way Down: How RLMs are Different — Kevin Madura, AlixPartners

privateReview: unreviewedConfidence: high
RLMcontextreplDSPyagentcontext-engineeringcontext-rotCodeActrecursive-decompositionsub-lm
Open on YouTube ↗

Summary

Overview

  • Speaker: Kevin Madura
  • Channel: AI Engineer
  • Main topic: Recursive Language Models (RLMs)
  • Purpose: Introduce the concept, architecture, performance benchmarks, and practical real-world applications of Recursive Language Models. Kevin Madura from AlixPartners explains Recursive Language Models (RLMs), comparing them to traditional RAG, tool-calling, and agents. RLMs treat context as an object to manipulate symbolically within a REPL environment, enabling recursive decomposition and overcoming traditional context window decay.

Topic Map

What is an RLM?

  • Explanation: Definition and core mechanics of Recursive Language Models.
  • Key claims:
    • Treats context as an object to manipulate in a symbolic environment.
    • Delegates to sub-LMs as functions inside a REPL.
    • Leverages the bitter lesson: allows the model to decide how to work.
  • Examples:
    • Omar Khattab's DSPy-based RLM prototype for summarizing arbitrarily long documents.
  • Terminology:
    • RLM
    • Recursive Language Model
    • REPL
    • DSPy
  • Why it matters: Changes how language models interact with massive contexts by deferring processing recursively rather than stuffing tokens.

Context Engineering vs. Abstraction

  • Explanation: Comparing RAG, agents, tool-calling, CodeAct, and RLMs regarding context management.
  • Key claims:
    • RAG, agents, and tool-calling suffer from context rot as windows fill up.
    • RLMs keep context as a variable in the environment, avoiding window saturation.
  • Examples:
    • Dex Horthy's concept of the 'dumb zone' where performance degrades after 40% context usage.
  • Terminology:
    • context rot
    • dumb zone
    • smart zone
    • CodeAct
  • Why it matters: Focuses development on abstractions and pipelines rather than manual context window engineering.

Real World Use Cases & Performance

  • Explanation: Evaluating RLM performance on long reasoning tasks, dataframes, and codebase scanning.
  • Key claims:
    • RLMs lift accuracy significantly on long-horizon reasoning tasks compared to vanilla models.
    • Effective for complex dataframes and automated security auditing on large codebases.
  • Examples:
    • Analyzing 5k-row tables and OWASP vulnerable serverless app codebase scanning for $0.87.
  • Terminology:
    • LongCOT
    • BrowseComp
    • HALO
    • predict-rlm
  • Why it matters: Demonstrates practical production readiness and cost-effectiveness of RLMs.

Key Points

Context is part of the environment

  • Explanation: Instead of passing strings back and forth, the LLM interacts with variables in a REPL.
  • Evidence: Intermediate results live in script variables rather than clogging the context window.
  • Practical implication: Reduces token bloat and allows handling millions of tokens efficiently.

Deterministic shell vs. probabilistic core

  • Explanation: The orchestrator uses a deterministic shell to execute code while the probabilistic model decides what tasks to delegate.
  • Evidence: Diagram showing deterministic shell wrapping the probabilistic core.
  • Practical implication: Provides structure and reliability to complex multi-step reasoning.

Frameworks, Models & Processes

Recursive Language Model (RLM)

  • How it works: Treats context as an object in a REPL environment, allowing the model to write code, peek, grep, chunk, and recursively call sub-LMs.
  • Components:
    • Deterministic shell (REPL)
    • Probabilistic core (LLM)
    • Sub-LMs
    • Script variables
  • When to use: Very large or dense inputs/outputs, tasks amenable to decomposition, and long-horizon sessions.

Examples & Case Studies

Scanning an OWASP vulnerable serverless application codebase using an RLM agent.

  • Illustrates: Executing complex codebase security audits without manual context engineering.
  • Lesson: RLMs can process large codebases cheaply and effectively surface security vulnerabilities.

Actionable Takeaways

  • Immediate:
    • Use RLMs when context won't fit or work is compositional and latency can be spent for reliability.
    • Skip RLMs for low-latency tasks, small contexts, or weak base coders.
  • Strategic:
    • Focus on abstractions and pipelines rather than low-level context engineering.
    • Future models post-trained to be RLM-native will drastically accelerate complex AI workflows.
  • Questions to investigate:
    • How do RLM overhead costs compare across different base models in production?
    • What are the best practices for structuring sub-LM signatures in DSPy?

Claims Worth Verifying

  • RLM lifted accuracy from 2.6% to 45.4% on long-horizon reasoning benchmarks. (benchmark)

Notable Quotes

"It's Tokens All The Way Down: How RLMs are Different" (at 0:24)

Compressed Summary

  • RLMs treat context as an object manipulated symbolically inside a REPL environment.
  • They avoid context rot by deferring sub-tasks to recursive sub-LMs.
  • Outperforms vanilla models and standard RAG/tool-calling on long-context benchmarks.
  • Keywords: rlm, context, repl, dspy, agent
  • Core insight: Recursive Language Models solve context degradation by treating context as an environmental variable inside a REPL rather than stuffing tokens into a single prompt.

Core insights

6
Architecturehigh noveltystrong evidence

RLMs reframe context from a payload passed between calls into a live object that the model manipulates symbolically inside a REPL; intermediate results persist in script variables rather than occupying the context window.

Why it matters

If intermediate state lives in variables instead of prompt tokens, the effective working set is no longer bounded by the context window, which changes how large-input pipelines are designed and what can be attempted at all.

Generalization

Any agent harness can be restructured so that bulky intermediate artifacts live in an execution environment and only references/summaries cross the model boundary.

RLMs treat context as an object to manipulate symbolically within a REPL environment
Open source video
Intermediate results live in script variables rather than clogging the context window
Open source video
Instead of passing strings back and forth, the LLM interacts with variables in a REPL.
Open source video
Mechanismmedium noveltymoderate evidence

The RLM pattern separates a deterministic execution shell (REPL) from a probabilistic core (LLM); the shell guarantees execution semantics while the model only decides what to write and what to delegate.

Why it matters

This is a concrete reliability boundary: deterministic code handles control flow and data movement (observable, replayable, testable), while the model is confined to the decision of what sub-task to invoke.

Generalization

Agent reliability improves when the nondeterministic component is wrapped by a deterministic runtime that owns execution, state, and error handling.

The orchestrator uses a deterministic shell to execute code while the probabilistic model decides what tasks to delegate.
Open source video
Diagram showing deterministic shell wrapping the probabilistic core.
Open source video
Mechanismhigh noveltymoderate evidence

Recursion, not retrieval, is the proposed remedy for context degradation: sub-tasks are delegated to sub-LMs as functions, so the parent never needs to hold the full corpus.

Why it matters

Contrasts directly with RAG/tool-calling, which the summary claims still accumulate context rot as the window fills; the decomposition happens at runtime and is decided by the model rather than pre-chunked at index time.

Generalization

For compositional tasks over dense inputs, runtime recursive decomposition can substitute for retrieval-time chunking, at the price of latency and repeated model calls.

Delegates to sub-LMs as functions inside a REPL
Open source video
RLMs keep context as a variable in the environment, avoiding window saturation
Open source video
RAG, agents, and tool-calling suffer from context rot as windows fill up.
Open source video
Tradeoffmedium noveltymoderate evidence

The stated selection criterion is orthogonal to task complexity: use RLM when input does not fit or work is compositional and latency can be traded for reliability; skip when latency is critical, context is small, or the base model is a weak coder.

Why it matters

This makes base-model code-generation quality a hard prerequisite for the architecture, not an implementation detail — a weak coder cannot drive a REPL-based harness regardless of context size.

Generalization

Architectures that put code generation on the critical path inherit the base model's coding reliability as a system-level failure mode.

Skip RLMs for low-latency tasks, small contexts, or weak base coders.
Open source video
Empirical Resultmedium noveltyweak evidence

Reported benchmark movement on long-horizon reasoning is an order-of-magnitude jump — accuracy from 2.6% to 45.4% — paired with a concrete cost data point ($0.87 to scan an OWASP-vulnerable serverless codebase).

Why it matters

If reproducible, this is a step change rather than a marginal gain, and the low dollar cost suggests the extra recursive calls are economically tolerable for audit-class workloads.

Generalization

Recursive decomposition may be most valuable exactly where vanilla models collapse (long-horizon reasoning), not on average-case tasks.

RLM lifted accuracy from 2.6% to 45.4% on long-horizon reasoning benchmarks.
Open source video
Analyzing 5k-row tables and OWASP vulnerable serverless app codebase scanning for $0.87.
Open source video
Mental Modelmedium noveltymoderate evidence

The design explicitly invokes the bitter lesson: the model decides how to work rather than the engineer hand-structuring the context pipeline.

Why it matters

It reframes context engineering as a temporary scaffold that should be replaced by model-controlled decomposition, which steers investment toward harness primitives (REPL, sub-LM interfaces) over prompt-level tricks.

Generalization

When deciding where to encode strategy, prefer giving the model a general execution substrate over encoding a fixed pipeline, if the substrate is expressive enough.

Leverages the bitter lesson: allows the model to decide how to work.
Open source video
Focus on abstractions and pipelines rather than low-level context engineering.
Open source video

Deep dives

4

Cost and latency profile of recursive decomposition across base model tiers

Research question

How does token usage, wall-clock latency, and dollar cost per completed task scale with recursion depth and base model capability, and at what point does RLM overhead outweigh its accuracy gains versus a single-pass or RAG baseline?

Why

Production adoption hinges on a predictable cost model; without one, RLM remains a benchmark curiosity rather than a deployable architecture.

Analyzing 5k-row tables and OWASP vulnerable serverless app codebase scanning for $0.87.
Open source video
Effective for complex dataframes and automated security auditing on large codebases.
Open source video
Source video

Reproducibility and component ablation of the 2.6%→45.4% long-horizon accuracy lift

Research question

Is the reported order-of-magnitude accuracy gain on long-horizon reasoning reproducible across benchmarks and base models, and which RLM components (recursion depth, REPL context manipulation, sub-LM delegation) account for the improvement?

Why

If reproducible, this represents a step change in what models can solve; if not, the RLM proposal rests on a single unverified data point.

RLM lifted accuracy from 2.6% to 45.4% on long-horizon reasoning benchmarks.
Open source video
RLMs lift accuracy significantly on long-horizon reasoning tasks compared to vanilla models.
Open source video
Source video

Crossover point: when recursive decomposition beats in-window processing after latency and cost

Research question

At what input size, context utilization, and task composition does an RLM harness outperform RAG/tool-calling baselines after normalizing for latency and dollar cost?

Why

Selection criteria are currently qualitative ('skip for low-latency, small contexts'); a quantitative boundary would turn RLM from an art into an engineering decision.

Dex Horthy's concept of the 'dumb zone' where performance degrades after 40% context usage.
Open source video
RAG, agents, and tool-calling suffer from context rot as windows fill up.
Open source video
Source video

Sub-LM signature design in DSPy and its effect on recursion quality

Research question

How do sub-LM signature granularity, typing, and contract design in a DSPy-based RLM affect recursion depth, total call count, and final answer accuracy?

Why

The sub-LM interface is the new critical path; without design guidance, practitioners will rediscover suboptimal patterns by trial and error.

Omar Khattab's DSPy-based RLM prototype for summarizing arbitrarily long documents.
Open source video
Delegates to sub-LMs as functions inside a REPL.
Open source video
Source video

Article ideas

4

The Context Window Is Not a Container: Why Externalizing State Beats Prompt Engineering

Treating context as a manipulable object in a REPL rather than a token payload eliminates context rot and unlocks arbitrarily long inputs, but it shifts the system failure mode from prompt engineering to code-generation reliability—making base-model coding ability a hard architectural prerequisite.

Angle

Contrast the RLM architecture with RAG/tool-calling and show that the real cost of adoption is base-model coding ability, not token budget.

Source video

Recursive Decomposition Is Not Retrieval: A New Default for Dense, Compositional Tasks

For tasks where input does not fit or reasoning is long-horizon, runtime recursive decomposition outperforms retrieval-time chunking, but the tradeoff is latency and recursive call overhead that makes it unsuitable for interactive applications.

Angle

Argue against over-reliance on RAG for complex reasoning; provide a decision framework based on input size, latency tolerance, and base-model coding ability.

Source video

The Bitter Lesson Comes for Context Engineering

Hand-crafted context pipelines are a temporary scaffold; giving the model a general execution substrate (REPL, sub-LM calls) and letting it decide how to work will outperform fixed pipelines as base models improve.

Angle

Use the RLM architecture as evidence that model-controlled decomposition is the next abstraction layer, and predict the decline of manual context engineering.

Source video

What $0.87 Buys You: The Economics of Recursive LLM Agents

The reported $0.87 codebase scan suggests recursive decomposition is economically viable for audit-class workloads, but without per-task cost profiles across model tiers, the claim of production readiness is premature.

Angle

Analyze the single cost data point and demand a cost model; show how to instrument an RLM to produce one.

Source video

Project ideas

4

RLM Cost Profiler

new

RLM overhead cost (tokens, latency, dollars) scales superlinearly with recursion depth and inversely with base-model coding ability, making the architecture uneconomical below a certain capability tier.

Proof of concept

Instrument an RLM harness to log token usage, wall-clock time, and sub-call count per recursion level; run on a fixed long-horizon task across 3–5 base models of varying coding ability; fit a cost model.

Measurement

Dollars per completed task and p95 latency vs. base-model coding benchmark score and recursion depth.

Source video

RLM Crossover Benchmark

beyond-evals

There exists a measurable input-size and context-utilization threshold above which an RLM harness outperforms a RAG baseline on long-horizon reasoning, after normalizing for latency and dollar cost.

Proof of concept

Implement a minimal RLM harness (REPL + sub-LM calls) and a RAG baseline on a long-document QA task; sweep input sizes and context utilization levels; measure accuracy, latency, and cost per task.

Measurement

Accuracy delta and cost per correct answer as a function of input size and context utilization.

Source video

Sub-LM Signature Ablation

new

The granularity and typing of sub-LM signatures in a DSPy-based RLM significantly affects recursion depth and final accuracy; overly coarse signatures increase context per sub-call while overly fine ones explode call count.

Proof of concept

Build three versions of a document-summarization RLM with different sub-LM signature granularities; evaluate on summarization quality, recursion depth, and total calls.

Measurement

ROUGE/BERTScore vs. recursion depth and call count.

Source video

Deterministic Shell Observability

movement-lab

A deterministic REPL shell that traces executed code, serializes state, and supports replay reduces debugging time and failure recurrence in recursive agents compared to unstructured tool-calling.

Proof of concept

Add tracing and replay to an RLM harness; run a failure-prone task; measure time-to-diagnose and failure recurrence with and without the shell.

Measurement

Mean time to diagnosis (MTTD) and failure recurrence rate.

Source video

Architectural implications

2

Context handling responsibility moves from the application layer into the execution environment.

Before

Application code assembles strings, chunks documents, and manages what fits into the prompt window.

After

The model writes code against a context object living in a REPL; the application supplies runtime primitives (peek, grep, chunk, sub-LM call).

Consequence

Context-window engineering stops being the main design surface; harness APIs and sub-LM call interfaces become the new critical path, and token bloat is bounded by variable scope instead of prompt size.

Source video

Delegation decisions shift from developer-authored control flow to model-authored code.

Before

Engineers predefine the pipeline: retrieve, then summarize, then answer.

After

The model decides at runtime which sub-tasks to delegate to sub-LMs inside the REPL.

Consequence

Observability must capture executed code and sub-call trees, since the control flow is no longer statically known; debugging moves from prompt inspection to trace replay.

Source video

Tradeoffs and failure modes

4

Latency versus reliability

Benefit

Recursive decomposition preserves accuracy on long or dense inputs that would otherwise degrade.

Cost or risk

Multiple sequential sub-LM calls add latency, making the pattern unsuitable for latency-critical paths.

Skip RLMs for low-latency tasks, small contexts, or weak base coders.
Open source video
Source video

Dependence on base-model coding ability

Benefit

The REPL lets the model decompose, peek, and manipulate data flexibly.

Cost or risk

The architecture fails if the base model cannot write reliable code, since every orchestration step is expressed as code.

Skip RLMs for low-latency tasks, small contexts, or weak base coders.
Open source video
Source video

Context rot / the dumb zone

Benefit

Keeping context as an environment variable avoids saturating the window.

Cost or risk

Conventional RAG, agents, and tool-calling degrade as the window fills, with reported performance loss after roughly 40% usage.

RAG, agents, and tool-calling suffer from context rot as windows fill up.
Open source video
Source video

Recursion overhead cost

Benefit

Recursive sub-LM calls lift accuracy on long-horizon reasoning.

Cost or risk

Overhead cost across different base models is unquantified in the summary and unspecified in production terms.

How do RLM overhead costs compare across different base models in production?
Open source video
Source video

Open questions

4

How do RLM overhead costs compare across different base models in production?

Why unresolved

The summary reports a single $0.87 codebase-scan figure and a benchmark uplift, but no cost profile across model tiers or against RAG/tool-calling baselines.

Research direction

Run the same RLM workloads across multiple base models and measure tokens, wall-clock latency, and dollars per completed task versus a RAG baseline.

Source video

What are the best practices for structuring sub-LM signatures in DSPy?

Why unresolved

The summary names the DSPy-based prototype and the sub-LM-as-function pattern but gives no interface design guidance.

Research direction

Compare sub-LM signature granularity and typing strategies against decomposition quality, recursion depth, and total call count.

Source video

Is the 2.6% to 45.4% long-horizon accuracy lift reproducible and does it generalize beyond the reported benchmarks?

Why unresolved

The uplift is listed as a claim worth verifying, with no methodology, model, or benchmark details in the summary.

Research direction

Replicate on LongCOT, BrowseComp, and HALO with held-out tasks and ablate the recursion depth to isolate which component produces the gain.

Source video

At what point does recursive decomposition stop helping relative to the reported ~40% context-usage degradation threshold?

Why unresolved

The summary cites the 'dumb zone' concept but does not connect it quantitatively to when RLM switching becomes worthwhile.

Research direction

Sweep context utilization levels and workload sizes to find the crossover point where RLM beats in-window processing after accounting for latency.

Source video

Key claims

7
comparativeVerification needed

RLMs lifted accuracy from 2.6% to 45.4% on long-horizon reasoning benchmarks.

Evidence

RLM lifted accuracy from 2.6% to 45.4% on long-horizon reasoning benchmarks.

Question

Which benchmarks and base models produced the 2.6% to 45.4% accuracy shift, and under what evaluation protocol?

Source video
causalVerification needed

RAG, agents, and tool-calling suffer context rot as windows fill up, while RLMs keep context as an environment variable.

Evidence

RLMs keep context as a variable in the environment, avoiding window saturation.

Question

Does preserving context as an environment variable measurably prevent the degradation seen in RAG/tool-calling at equal task difficulty?

Source video
factualVerification needed

Performance degrades after roughly 40% context usage, described as the 'dumb zone'.

Evidence

Dex Horthy's concept of the 'dumb zone' where performance degrades after 40% context usage.

Question

Is the 40% context-usage threshold consistently reproducible across models and task types?

Source video
factualVerification needed

An RLM agent scanned an OWASP-vulnerable serverless application codebase for $0.87.

Evidence

Analyzing 5k-row tables and OWASP vulnerable serverless app codebase scanning for $0.87.

Question

What was the actual vulnerability detection rate and false-positive rate of that scan, and was model and token usage fully accounted for in the $0.87?

Source video
factualVerification not requested

The RLM architecture wraps a probabilistic core inside a deterministic shell.

Evidence

Diagram showing deterministic shell wrapping the probabilistic core.

Source video
opinionVerification needed

RLM overhead costs vary by base model in production in ways not yet characterized.

Evidence

How do RLM overhead costs compare across different base models in production?

Question

What is the measured cost and latency multiplier of RLM recursion per base model tier?

Source video
predictionVerification needed

Future models post-trained to be RLM-native will drastically accelerate complex AI workflows.

Evidence

Future models post-trained to be RLM-native will drastically accelerate complex AI workflows.

Question

Does RLM-native post-training measurably reduce recursion depth or call count versus prompting a general model into the same pattern?

Source video

Connections

5