Claude · Published 2026-05-23

Tool, skill, or subagent? Decomposing an agent that outgrew its prompt

privateReview: unreviewedConfidence: high
agentspromptsSkillstoolsevalsrefactoringagent-architectureprompt-decompositionprogressive-disclosurecode-executionsubagentscontext-management
Open on YouTube ↗

Summary

Overview

  • Speaker: William Steuk
  • Channel: Claude
  • Main topic: Decomposing and refactoring complex AI system prompts into modular tools, skills, and subagents
  • Purpose: To educate engineers and developers on how to refactor and optimize complex, prompt-heavy AI agents into clean, modular architectures using tools, skills, and subagents. William Steuk from Anthropic discusses the architectural evolution of AI agents that have outgrown their system prompts. Using an inventory management agent called 'StockPilot' as a case study, he explores how bolting on too many capabilities leads to regressions, degraded performance, and context bloat. He presents a systematic approach to refactoring — breaking down monolithic prompts into composable primitives: tools, skills, and subagents (specifically callable agents in Claude Managed Agents), and using evals ('hill climbing') to measure performance improvements.

Topic Map

Introduction and Problem Statement

  • Explanation: Introduction of William Steuk and the common anti-pattern of agent prompts growing to hundreds of lines due to continuous capability additions.
  • Key claims:
    • Agents that start out successful often degrade in performance as business requirements and capabilities are continually bolted onto the system prompt.
    • Monolithic system prompts lead to regressions in areas the agent was previously accelerating in.
  • Examples:
    • A system prompt growing to several hundred lines with dozens of tools and subagents.
  • Terminology:
    • Applied AI
    • system prompt
    • regression
    • StockPilot
  • Why it matters: Context bloat and architectural complexity cause AI agents to fail at specific tasks due to conflicting instructions and lack of modularity.

Introducing StockPilot and Evals

  • Explanation: Walkthrough of StockPilot, an inventory management agent for an outdoor-gear retailer, and its evaluation framework consisting of 12 eval tasks across 5 grader types.
  • Key claims:
    • StockPilot manages 250 SKUs, 3 warehouses, 12 suppliers, and 90 days of sales history.
    • Evals are divided into deterministic regression tasks (R-tasks) and non-deterministic LLM-as-a-judge failure mode tasks (F-tasks).
  • Examples:
    • R1 (stock lookup), R8 (promo-month forecast), F1 (daily low-stock sweep).
  • Terminology:
    • evals
    • graders
    • deterministic
    • non-deterministic
    • LLM-as-a-judge
  • Why it matters: Without robust evaluations, engineers cannot quantitatively measure whether architectural changes improve or degrade agent performance.

Anatomy of the Architectural Failure

  • Explanation: Detailed root cause analysis of why StockPilot failed specific evals (dropping from 83% to 62% success rate), including context window pollution, conflicting policy instructions, and missing tool support.
  • Key claims:
    • Model took an inefficient path because reasoning tasks were performed without proper dedicated tools.
    • Communication breakdowns between orchestrators and hardcoded subagents caused formatting failures.
    • Conflicting policies in different parts of a monolithic system prompt led to model confusion.
  • Examples:
    • R8 promo-month forecast hallucinating a promotion multiplier instead of using 3.1x.
    • F1 daily low-stock sweep taking a winding path through raw CSV data.
  • Terminology:
    • context pollution
    • orchestrator
    • subagent
    • root cause analysis
  • Why it matters: Understanding exact points of failure allows engineers to apply the correct architectural primitive rather than adding more text to the system prompt.

Refactoring Architecture: Tools, Skills, and Subagents

  • Explanation: Step-by-step refactoring using Claude Managed Agents (CMA), swapping legacy prompts for short prompts, loading organizational procedures on demand via skills, and implementing callable subagents.
  • Key claims:
    • Skills provide packaged, composable information that Claude pulls into context only when needed.
    • Code execution (Bash/Python) replaces rigid tool definitions for data analysis and file traversal.
    • Callable agents in CMA allow subagents to run in isolated sessions with full observability and metrics matching the orchestrator.
  • Examples:
    • Replacing 416-line system prompt with a 15-line prompt and 5 on-demand skills.
    • Replacing raw CSV dumping tool with a Python script that returns top-N JSON.
  • Terminology:
    • skills
    • progressive disclosure
    • code execution
    • Claude Managed Agents
    • callable agents
  • Why it matters: Modular architectures drastically reduce token consumption, execution latency, and cost while improving eval success rates.

Key Points

The Danger of Monolithic System Prompts

  • Explanation: Appending new business requirements and tool instructions directly into a system prompt leads to context bloat, token inefficiency, and model confusion.
  • Evidence: StockPilot system prompt grew to 400+ lines, causing eval performance to drop from 83% to 62%.
  • Practical implication: Keep system prompts short and focused only on core orchestration and foundational instructions.

Progressive Disclosure via Skills

  • Explanation: Instead of stuffing all procedures into the system prompt, package domain knowledge into modular 'skills' that the agent loads on demand.
  • Evidence: Swapping a 416-line legacy prompt for a 15-line prompt with 5 on-demand skills increased eval success from 71% to 92%.
  • Practical implication: Use skills for reference material, guidelines, and templates that are only needed for specific tasks.

Lean on Computer-Like Primitives

  • Explanation: Grant agents foundational tools like code execution (Bash/Python) and file system access rather than building rigid, custom single-purpose tools for every action.
  • Evidence: Replacing custom data-dump tools with Python scripts dropped token usage from 216k to 7.6k per task.
  • Practical implication: Trust the model's ability to write and execute code to inspect data rather than pre-processing everything into context.

Hill-Climbing with Evals

  • Explanation: Optimize agent architecture iteratively by running automated test suites, triaging failures, updating agent design, and measuring improvements.
  • Evidence: Running 'uv run evals --agent before' followed by architectural iterations and re-running evals.
  • Practical implication: Treat agent development like software engineering: measure performance regressions and improvements programmatically.

Frameworks, Models & Processes

Agent Decomposition Framework

  • How it works: Deconstructs an bloated agent prompt into foundational primitives by separating orchestration, specialized knowledge, and parallel execution.
  • Components:
    • Short System Prompt (orchestration and core behavior)
    • Skills (on-demand domain knowledge and procedures)
    • Tools (foundational capabilities like bash and file reading)
    • Callable Subagents (isolated, parallelizable sub-tasks with observability)
  • When to use: When an AI agent's prompt exceeds manageable lengths and exhibits regression in performance or excessive token costs.

Examples & Case Studies

StockPilot failed R8 promo-month forecast because the model hallucinated a promotion multiplier instead of using 3.1x due to conflicting policies in a 400-line prompt.

  • Illustrates: Context pollution and policy contradiction in monolithic prompts.
  • Lesson: Separate policies and forecasting instructions into targeted skills or subagents.

StockPilot's daily low-stock sweep (F1) failed because it dumped 395 rows of CSV data into context, causing context-over-context misses.

  • Illustrates: Inefficient data transfer between tools and the model.
  • Lesson: Replace rigid data-dump tools with code execution where the model writes Python scripts to query data in the sandbox.

Actionable Takeaways

  • Immediate:
    • Audit existing agent system prompts for length and bloat.
    • Extract static policies and procedures into modular skill files.
    • Implement automated evaluations to establish an empirical baseline.
  • Strategic:
    • Adopt progressive disclosure so agents only load context when required.
    • Favor code execution and file system primitives over hyper-specific custom tools.
    • Use managed agent platforms (like CMA) to offload orchestration harness, infrastructure, memory, and scaling concerns.
  • Questions to investigate:
    • When should a capability be implemented as a skill versus a callable subagent?
    • How do non-deterministic LLM-as-a-judge graders correlate with production reliability?

Claims Worth Verifying

  • Replacing rigid custom tools with Python code execution drops token usage significantly while maintaining task success. (technical benchmark)
  • Claude Managed Agents (CMA) provides native primitives for callable subagents with persistent threads and isolated sandboxes. (product architecture)

Notable Quotes

"Tool, skill, or subagent? Decomposing an agent that outgrew its prompt" (at 0:13) "Every choice was defensible when made. Bottlenecks in capability emerge from accumulation." (at 5:34) "Simple composable agents scale with model intelligence" (at 72:05)

Compressed Summary

  • Monolithic agent system prompts cause capability accumulation, context bloat, and regression.
  • Refactoring involves shortening system prompts and loading organizational procedures on-demand via skills.
  • Use foundational computer primitives like Python execution instead of rigid custom tools to reduce token usage.
  • Callable subagents allow parallel exploration and clean context isolation for complex sub-tasks.
  • Iterative 'hill-climbing' using automated evals validates architectural improvements.
  • Keywords: agents, prompts, skills, tools, evals, refactoring
  • Core insight: Decomposing bloated agent prompts into composable tools, on-demand skills, and isolated subagents drastically reduces token costs and improves reliability.

Core insights

7
Empirical Resultmedium noveltystrong evidence

Replacing a monolithic system prompt with a short orchestration prompt plus on-demand 'skills' (progressive disclosure) converts a large token-cost/attention penalty into a measurable accuracy gain: a 416-line prompt swapped for a 15-line prompt + 5 skills moved eval success from 71% to 92%, indicating the win comes from loading domain procedures only when the task requires them rather than from the prompt content itself.

Why it matters

It gives a concrete, measurable refactor pattern: treat domain procedures, guidelines and templates as retrievable modules rather than permanent system-prompt real estate, which simultaneously reduces context bloat and raises task success.

Generalization

Any agent whose instructions have grown by appending one procedure per feature can likely be decomposed into a thin orchestration prompt plus lazily-loaded capability documents, with accuracy and cost both improving.

Swapping a 416-line legacy prompt for a 15-line prompt with 5 on-demand skills increased eval success from 71% to 92%.
Open source video
Skills provide packaged, composable information that Claude pulls into context only when needed.
Open source video
Mechanismmedium noveltystrong evidence

Foundational computer-like primitives (code execution via Bash/Python, filesystem access) can replace many bespoke single-purpose tools, collapsing context consumption by roughly 28x — token usage fell from 216k to 7.6k per task when a raw data-dump tool was replaced with a Python script returning top-N JSON.

Why it matters

It reframes tool design: instead of encoding every analysis as a rigid tool schema plus pre-processed data in context, let the model write code against raw data and return only the reduced result, which directly attacks token cost, latency and pre-processing maintenance.

Generalization

Prefer a small set of general primitives over a large catalog of narrow tools; the model's ability to write code becomes the dispatch mechanism, and the context window only receives the computed answer.

Replacing custom data-dump tools with Python scripts dropped token usage from 216k to 7.6k per task.
Open source video
Trust the model's ability to write and execute code to inspect data rather than pre-processing everything into context.
Open source video
Architecturemedium noveltymoderate evidence

Subagents implemented as callable agents run in isolated sessions yet emit observability metrics that match the orchestrator's, meaning delegation does not create an observability gap — a prerequisite for treating subagents as an organizational boundary rather than a debugging hazard.

Why it matters

Observability parity between orchestrator and delegated agents determines whether a multi-agent decomposition is operable in production; if subagent telemetry is a second-class citizen, isolation becomes a liability.

Generalization

Any multi-agent topology should be judged partly on whether per-delegate telemetry is comparable to the parent's, otherwise failure attribution and cost accounting break down at the boundary.

Callable agents in CMA allow subagents to run in isolated sessions with full observability and metrics matching the orchestrator.
Open source video
Failure Modemedium noveltymoderate evidence

The failure signature of an overgrown prompt is not degraded tone but specific wrong answers: conflicting policy statements inside one prompt led to model confusion, and the agent hallucinated a promotion multiplier instead of using the supplied 3.1x value, while without a dedicated tool it took a winding path through raw CSV data during the daily low-stock sweep.

Why it matters

It ties a structural defect (policy collision and missing tool support in one monolithic context) to reproducible eval failures, which lets engineers diagnose architecture rather than rewording instructions.

Generalization

When an agent regresses on previously-passing tasks, look for mutually inconsistent instructions co-resident in context and for reasoning tasks that lack a dedicated tool, rather than adding clarifying prose.

Conflicting policies in different parts of a monolithic system prompt led to model confusion.
Open source video
R8 promo-month forecast hallucinating a promotion multiplier instead of using 3.1x.
Open source video
F1 daily low-stock sweep taking a winding path through raw CSV data.
Open source video
Failure Modemedium noveltymoderate evidence

Orchestrator-to-subagent interfaces are themselves a failure surface: communication breakdowns between orchestrators and hardcoded subagents caused formatting failures, implying the decomposition decision must account for the contract between caller and callee, not just capability placement.

Why it matters

It warns that splitting work into subagents can introduce new failure modes at the boundary, so the tool/skill/subagent choice is a coupling decision as much as a capability decision.

Generalization

Every delegation boundary needs an explicit, testable interface contract (schema, formatting, error semantics); otherwise decomposition trades context bloat for integration failures.

Communication breakdowns between orchestrators and hardcoded subagents caused formatting failures.
Open source video
Mental Modelmedium noveltymoderate evidence

The practical decision procedure implied by the case study is a routing rule over three primitives: short prompt for orchestration and core behavior, skills for on-demand domain knowledge and procedures, foundational tools (bash, file reading) for capability, and callable subagents for isolated parallelizable sub-tasks with observability — applied when the prompt exceeds manageable length or shows regression and excessive token cost.

Why it matters

It converts the 'tool, skill or subagent?' question from intuition into an allocation rule based on what the artifact is (behavior, knowledge, capability, parallel work) and what property is needed (token economy, isolation, parallelism, observability).

Generalization

Decomposition frameworks should be indexed by the kind of content being extracted and the runtime property required, not by feature count.

Deconstructs an bloated agent prompt into foundational primitives by separating orchestration, specialized knowledge, and parallel execution.
Open source video
When to use: When an AI agent's prompt exceeds manageable lengths and exhibits regression in performance or excessive token costs.
Open source video
Practicelow noveltystrong evidence

Evaluation architecture is split by determinism: 12 eval tasks across 5 grader types, divided into deterministic regression tasks (R-tasks) and non-deterministic LLM-as-a-judge failure-mode tasks (F-tasks), and used as a hill-climbing loop (run evals on the 'before' agent, change architecture, re-run) rather than a one-off benchmark.

Why it matters

It makes architecture changes falsifiable: regression tasks catch breakage while LLM-judge tasks probe qualitatively-shaped failure modes, and the loop discipline prevents prompt edits from being judged by vibes.

Generalization

Agent refactoring should be gated by a two-tier eval suite (deterministic regressions + judged failure modes) invoked before and after each structural change.

Evals are divided into deterministic regression tasks (R-tasks) and non-deterministic LLM-as-a-judge failure mode tasks (F-tasks).
Open source video
Running 'uv run evals --agent before' followed by architectural iterations and re-running evals.
Open source video

Deep dives

5

Skill selection as a retrieval failure surface

Research question

As a skill catalog scales beyond the 5-skill case study, how does the model's skill-selection precision decay, and does silent mis-retrieval eventually erase the gains of progressive disclosure?

Why

The documented 71%→92% gain depends on the model correctly deciding when to load a skill; if selection precision degrades with catalog size, progressive disclosure converts a visible context-bloat failure into an invisible retrieval failure that no prompt edit can fix.

Skills provide packaged, composable information that Claude pulls into context only when needed.
Open source video
Swapping a 416-line legacy prompt for a 15-line prompt with 5 on-demand skills increased eval success from 71% to 92%.
Open source video
Source video

Latency, sandbox cost and code-generation reliability under primitive-first tool design

Research question

Do the 28x token savings from replacing data-dump tools with generated Python survive as wall-clock latency, sandbox overhead and code-execution error/retry rates, and for which task shapes does rigid tool determinism still win?

Why

Token usage is the only reported metric; if generated code raises latency, retries or sandbox cost, the 'trust the model to write code' rule becomes a task-shape-dependent decision rather than a general replacement for narrow tools.

Replacing custom data-dump tools with Python scripts dropped token usage from 216k to 7.6k per task.
Open source video
Trust the model's ability to write and execute code to inspect data rather than pre-processing everything into context.
Open source video
Source video

Orchestrator-subagent interface contracts as a first-class decomposition artifact

Research question

Does specifying explicit, testable interface contracts (schema, formatting, error semantics) between orchestrator and callable subagents eliminate boundary formatting failures, or merely relocate them to a different layer?

Why

The pre-refactor system already failed at the boundary between hardcoded subagents and the orchestrator; without a named contract artifact, every decomposition trades context bloat for integration failures that evals may not be shaped to catch.

Communication breakdowns between orchestrators and hardcoded subagents caused formatting failures.
Open source video
Callable agents in CMA allow subagents to run in isolated sessions with full observability and metrics matching the orchestrator.
Open source video
Source video

Attributing refactor gains: progressive disclosure vs. removal of conflicting instructions

Research question

Of the measured 71%→92% improvement, how much is attributable to progressive disclosure itself versus the simultaneous deletion of contradictory policies and obsolete tools from one context window?

Why

The refactor bundled several changes at once; if most of the gain comes from removing conflicting instructions, the engineering advice shifts from 'build a skill system' to 'make prompt consistency auditable,' which is far cheaper.

Conflicting policies in different parts of a monolithic system prompt led to model confusion.
Open source video
StockPilot system prompt grew to 400+ lines, causing eval performance to drop from 83% to 62%.
Open source video
Source video

Two-tier eval weighting and Goodharting across R-tasks and F-tasks

Research question

Are deterministic regression tasks and LLM-as-a-judge failure-mode tasks complementary across refactor iterations, or can optimizing one tier hide regressions in the other as the hill-climbing loop proceeds?

Why

The hill-climbing loop uses the eval suite as its objective function; without per-tier score tracking and grader-disagreement instrumentation, the suite can be gamed by architecture changes that satisfy one tier and quietly break the other.

Evals are divided into deterministic regression tasks (R-tasks) and non-deterministic LLM-as-a-judge failure mode tasks (F-tasks).
Open source video
Running 'uv run evals --agent before' followed by architectural iterations and re-running evals.
Open source video
Source video

Article ideas

4

Your Overgrown System Prompt Is Not a Documentation Problem, It Is a Consistency Bug

Prompt growth does not degrade an agent by making it vaguer; it degrades it by co-locating mutually contradictory instructions in one context, so the failure signature is a specific wrong answer (a hallucinated promotion multiplier instead of the supplied 3.1x) and the correct response is structural, not more clarifying prose.

Angle

Diagnose regressions as architecture defects by looking for policy collisions and missing dedicated tools rather than rewording instructions.

Source video

The 28x Rule: Stop Building a Tool Catalog and Start Shipping Primitives

A small set of general primitives (code execution, filesystem access) subsumes many narrow tools because the model's generated code becomes the dispatch mechanism and only the reduced result enters the context window; the token savings are a symptom of a deeper shift from schema design to code-generation trust.

Angle

Reframe tool design around data reduction at the execution layer rather than pre-processing everything into context.

Source video

Progressive Disclosure Moves the Failure Mode, It Does Not Remove It

Skills convert a loud failure (context bloat and conflicting residency) into a quiet one (missed or wrong skill selection), so adopting progressive disclosure without selection-level evals is trading a diagnosable problem for an invisible one.

Angle

Treat skill loading as a retrieval problem requiring precision/miss metrics, not a prompt-formatting trick.

Source video

Every Delegation Boundary Needs a Contract: Decomposition Trades Context Bloat for Integration Failures

Choosing between tool, skill and subagent is a coupling decision as much as a capability decision, because the pre-refactor system's formatting failures came from the orchestrator-to-subagent seam rather than from any single component, so interfaces must be explicit and testable before decomposition pays off.

Angle

Index the tool/skill/subagent choice by the contract it creates, not only by the content it extracts.

Source video

Project ideas

4

Skill-Selector Stress Bench

beyond-evals

Skill-selection precision decays measurably as the on-demand catalog grows past roughly 20 skills, and beyond that point missed-load failures reduce eval success to at or below the monolithic-prompt baseline.

Proof of concept

Build a StockPilot-shaped agent with a thin orchestration prompt and a skill library scaled across N in {5, 10, 20, 50}, then run an identical eval suite at each N against a fixed monolithic-prompt baseline.

Measurement

Skill-selection precision, missed-load rate, incorrect-skill-load rate, and eval success delta versus the monolithic baseline at each N.

Source video

Primitive vs Schema Bake-off

movement-lab

For read-heavy data-analysis tasks, generated-code execution matches rigid single-purpose tools on task accuracy while cutting tokens by more than 10x, but loses on wall-clock latency and retry rate.

Proof of concept

Implement the same analysis tasks twice against identical raw data — once with rigid data-dump tools, once with Python/Bash primitives returning top-N JSON — and run matched trials under a sandboxed executor.

Measurement

Tokens per task, wall-clock latency, code-execution error and retry rate, sandbox overhead, and task accuracy per implementation.

Source video

Boundary Contract Harness

gatehouse

Declaring explicit typed contracts (schema, formatting, error semantics) at every orchestrator-to-subagent boundary reduces formatting failure rates to near zero, whereas hardcoded subagent coupling reproduces them regardless of model quality.

Proof of concept

Instrument an orchestrator with two identical subagent topologies — hardcoded coupling versus contract-validated callable agents — and replay the same delegation-heavy eval tasks through both.

Measurement

Boundary formatting failure rate per topology, plus false-rejection rate introduced by contract validation.

Source video

Refactor Attribution Ablation

new

A substantial fraction of the reported 71%→92% gain comes from removing contradictory instructions rather than from progressive disclosure, and the gain shrinks when the same skill system is applied to an already-consistent prompt.

Proof of concept

Run four arms on one eval suite: original long prompt, long prompt with conflicts surgically removed, 15-line prompt plus skills, and 15-line prompt plus skills built from the de-conflicted long prompt.

Measurement

Per-arm eval success delta, decomposed into the conflict-removal effect and the progressive-disclosure effect.

Source video

Architectural implications

3

Instructions that were permanently resident in the system prompt move to lazily-retrieved skill documents.

Before

A single multi-hundred-line prompt containing all policies, procedures and tool usage guidance.

After

A ~15-line orchestration prompt plus multiple on-demand skills pulled into context only when needed.

Consequence

Lower baseline context occupancy and higher task accuracy (71% to 92%), but correctness now depends on the model correctly deciding when to load a skill.

Source video

Pre-processing of data into context is replaced by code the agent writes and executes.

Before

Bespoke data-dump tool that inlines raw data (e.g., CSV) into the context window.

After

Python/Bash executed against raw files, returning only a reduced result such as top-N JSON.

Consequence

~28x token reduction per task (216k to 7.6k) and less tool-schema maintenance, at the cost of depending on code-generation reliability and an execution sandbox.

Source video

Monolithic delegation logic is replaced by subagents invoked as callable agents.

Before

Hardcoded subagents tightly coupled to the orchestrator with formatting mismatches.

After

Isolated sessions with full observability and metrics matching the orchestrator.

Consequence

Parallelizable, independently measurable sub-tasks, but requires an explicit interface contract to avoid boundary formatting failures.

Source video

Tradeoffs and failure modes

4

Skills vs. prompt residency

Benefit

Domain knowledge is loaded on demand, cutting context bloat and raising eval success from 71% to 92% with a 15-line prompt and 5 skills.

Cost or risk

Loading is conditional on the model recognizing the need, so a missed skill selection becomes a silent failure mode not present when instructions are always in context.

Skills provide packaged, composable information that Claude pulls into context only when needed.
Open source video
Source video

Code execution vs. rigid single-purpose tools

Benefit

Token usage dropped from 216k to 7.6k per task and data analysis no longer requires pre-processing everything into context.

Cost or risk

Shifts correctness onto the model's generated code and requires an execution environment, trading determinism and sandbox/safety surface for token economy.

Trust the model's ability to write and execute code to inspect data rather than pre-processing everything into context.
Open source video
Source video

Subagent decomposition vs. orchestrator control

Benefit

Subagents run in isolated sessions in parallel with full observability and metrics matching the orchestrator.

Cost or risk

New boundary failures appear; hardcoded orchestrator-to-subagent communication already produced formatting failures in the pre-refactor system.

Communication breakdowns between orchestrators and hardcoded subagents caused formatting failures.
Open source video
Source video

Appending capabilities to a system prompt

Benefit

Fastest path to add a business requirement, and the agent starts out successful.

Cost or risk

Prompt growth to 400+ lines produced conflicting policies, context pollution and a measured regression from 83% to 62% success.

StockPilot system prompt grew to 400+ lines, causing eval performance to drop from 83% to 62%.
Open source video
Source video

Open questions

5

What is the decision rule that assigns a given capability to a tool, a skill, or a subagent, rather than the presented four-primitive framework?

Why unresolved

The summary presents components and a 'when to use' trigger for bloated prompts, but no discriminating criteria between the three decomposition targets.

Research direction

Build a taxonomy of capability types (deterministic data access, procedural knowledge, parallel independent work) and empirically test mis-assignments against eval suites.

Source video

Does progressive disclosure degrade as the skill library grows, and how does the model reliably select the right skill?

Why unresolved

The evidence covers only 5 on-demand skills in one case study; retrieval/selection failure at larger skill counts is not addressed.

Research direction

Scale the skill catalog in a benchmark and measure selection precision, missed-load failures, and accuracy decay versus a monolithic prompt baseline.

Source video

Do the token savings from code execution (216k to 7.6k) survive as latency, sandbox cost and safety overhead, and how does code-execution reliability compare to rigid tool determinism?

Why unresolved

Only token usage is reported; latency, cost of execution, and failure/retry rates for generated code are absent.

Research direction

Run matched tasks comparing rigid tools, code execution, and hybrid designs on tokens, wall-clock latency, error rate, and required sandboxing.

Source video

Are the reported eval gains attributable to decomposition per se, or to the specific Claude Managed Agents runtime and callable-agent semantics?

Why unresolved

All results come from one case study on a specific platform (CMA) with a proprietary agent stack.

Research direction

Replicate the StockPilot refactor pattern on another orchestration framework and compare deltas on the same eval task shapes.

Source video

How should the eval suite itself be weighted — are R-tasks and F-tasks complementary or can optimization for one hide regressions in the other?

Why unresolved

The summary describes the R/F split and the hill-climbing loop but reports no per-category results or disagreement analysis between graders.

Research direction

Instrument grader disagreement and track R-task and F-task scores separately across refactor iterations to detect Goodharting.

Source video

Key claims

8
causalVerification needed

A system prompt that grew past 400 lines with dozens of tools and subagents caused StockPilot's eval performance to fall from 83% to 62%.

Evidence

StockPilot system prompt grew to 400+ lines, causing eval performance to drop from 83% to 62%.

Question

Is the 83% to 62% drop reproducible and separable from model-version or data changes over the same period?

Source video
comparativeVerification needed

Replacing a 416-line legacy prompt with a 15-line prompt plus 5 on-demand skills raised eval success from 71% to 92%.

Evidence

Swapping a 416-line legacy prompt for a 15-line prompt with 5 on-demand skills increased eval success from 71% to 92%.

Question

Is the gain driven by progressive disclosure or by the simultaneous removal of conflicting instructions and obsolete tools?

Source video
factualVerification needed

Replacing custom data-dump tools with Python scripts reduced token usage from 216k to 7.6k per task.

Evidence

Replacing custom data-dump tools with Python scripts dropped token usage from 216k to 7.6k per task.

Question

Does the reduction hold across task types, and what latency and code-execution failure rate accompanies it?

Source video
factualVerification needed

Callable agents in CMA let subagents run in isolated sessions with full observability and metrics matching the orchestrator.

Evidence

Callable agents in CMA allow subagents to run in isolated sessions with full observability and metrics matching the orchestrator.

Question

Are the subagent metrics truly equivalent in granularity and cost accounting, or only in aggregate surface?

Source video
factualVerification not requested

The evaluation suite is deliberately split into deterministic regression tasks and non-deterministic LLM-as-a-judge failure-mode tasks.

Evidence

Evals are divided into deterministic regression tasks (R-tasks) and non-deterministic LLM-as-a-judge failure mode tasks (F-tasks).

Source video
causalVerification needed

Conflicting policies placed in different parts of a monolithic system prompt led to model confusion and hallucination (a fabricated promotion multiplier instead of 3.1x).

Evidence

Conflicting policies in different parts of a monolithic system prompt led to model confusion.

Question

Can prompt conflict be isolated as the cause versus context length or instruction position effects?

Source video
causalVerification needed

Communication breakdowns between orchestrators and hardcoded subagents caused formatting failures.

Evidence

Communication breakdowns between orchestrators and hardcoded subagents caused formatting failures.

Question

Do callable agents with explicit contracts eliminate these formatting failures or merely relocate them?

Source video
opinionVerification not requested

The recommended use condition is an agent prompt that has exceeded manageable length and shows regression or excessive token cost.

Evidence

When to use: When an AI agent's prompt exceeds manageable lengths and exhibits regression in performance or excessive token costs.

Source video

Connections

5