AI Engineer · Published 2026-05-18

Anthropic Workshop: Build Agents That Run for Hours — Ash Prabaker & Andrew Wilson

privateReview: unreviewedConfidence: high
AI agentsharness designevaluator loopContext managementanthropic claudeai-agentsagent-harnessgenerator-evaluatoradversarial-evaluationcontext-managementlong-horizon-tasksfilesystem-coordinationmulti-agent-architecture
Open on YouTube ↗

Summary

Overview

  • Speaker: Ash Prabaker and Andrew Wilson
  • Channel: AI Engineer
  • Main topic: Building and maintaining long-running AI agents using Claude models and custom agent harnesses
  • Purpose: To educate AI engineers on the architectural patterns, harnesses, and model evolutions required to successfully run coding and development agents autonomously for extended periods (5 to 30+ hours). Ash Prabaker and Andrew Wilson from Anthropic's Applied AI team discuss the evolution of Claude models and agent harnesses over a year, focusing on how to build agents that can run for hours without losing coherence, planning, or quality. They examine the three reasons agents lose the plot—context degradation, planning failures, and poor self-verification—and present practical fixes including model capability improvements, harness design patterns like the generator-evaluator loop (inspired by GANs), sprint contracts, and file system sharing.

Topic Map

Introduction and Problem Framing

  • Explanation: Introduces the speakers from Anthropic and frames the core challenge: building agents that can run for hours on complex tasks without losing context or intent.
  • Key claims:
    • Agents running for 5 to 6+ hours require careful harness design, not just one-shot model prompting.
    • Demos of one-shot browser execution often hide the underlying complexity of the agent harness.
  • Examples:
    • One-shot browser demos vs. multi-hour agent harnesses.
  • Terminology:
    • agent harness
    • applied AI
    • context management
  • Why it matters: Setting expectations for production agent deployments beyond short prompt-response cycles.

Three Reasons Agents Lose the Plot

  • Explanation: Analyzes why long-running agents fail, categorizing failures into context limits, planning deficiencies, and poor self-verification.
  • Key claims:
    • Context windows are finite and suffer from coherence degradation as sessions deepen.
    • Models struggle with long-term planning, often trying to one-shot tasks or stopping prematurely.
    • Models are poor judges of their own code output, exhibiting sycophancy and superficial testing.
  • Examples:
    • Curl testing instead of click testing in browser apps.
    • Stubs features and moving on without checking backend implementation.
  • Terminology:
    • context rot
    • context anxiety
    • reward hacking
    • sycophancy
  • Why it matters: Identifies the core failure modes engineers must engineer around when building autonomous agents.

Evolution of Claude Models and Harnesses

  • Explanation: Walks through a timeline from Sonnet 3.5 to Opus 4.6, demonstrating how models and harnesses co-evolved to extend autonomous runtime from minutes to days.
  • Key claims:
    • Model task completion horizons increased from 1 hour (Sonnet 3.7) to 12+ hours (Opus 4.6).
    • Harnesses have co-evolved with models, starting with complex multi-prompt loops and shedding unnecessary scaffolding as models became more capable.
  • Examples:
    • Artifacts introduced in Sonnet 3.5 providing the first closed feedback loop.
    • Claude Code and agent SDK releases featuring checkpoints, memory tools, and agent teams.
  • Terminology:
    • METR Horizon
    • Claude Code
    • MCP servers
    • agent SDK
    • agent teams
  • Why it matters: Shows the trajectory of LLM agent capability and how hardware/model upgrades shift the burden from harness complexity to model intelligence.

Harness Design: Separating Builder from Judge

  • Explanation: Presents the generator-evaluator loop inspired by GANs, where a separate evaluator agent (using Playwright and rubrics) tests and critiques the generator's work.
  • Key claims:
    • Tuning a standalone evaluator to be skeptical and harsh is tractable; making a generator self-critical is not.
    • Subjective quality like design and originality can be graded using detailed rubrics applied by an adversarial evaluator.
  • Examples:
    • Frontend web development loop where Playwright navigates, scores, and critiques pages before passing feedback to the generator.
  • Terminology:
    • generator-evaluator loop
    • adversarial evaluator
    • Playwright
    • subjective quality gradable
  • Why it matters: Solves the self-evaluation flaw in LLMs by introducing an adversarial feedback loop.

Scaling to Full Stack with Three Agents

  • Explanation: Introduces a three-agent architecture—Planner, Generator, and Evaluator—communicating exclusively via the filesystem.
  • Key claims:
    • A Planner agent breaks a high-level 1-line prompt into a high-level spec and sprint workflow.
    • Generator and Evaluator negotiate sprint contracts via files before writing any code.
    • Zero shared context; all coordination goes through files on disk to prevent context rot.
  • Examples:
    • Retro game maker and browser music production app built over 4 to 6 hours using the 3-agent harness.
  • Terminology:
    • Planner agent
    • Generator agent
    • Evaluator agent
    • sprint contracts
    • filesystem state
  • Why it matters: Provides a robust, production-tested blueprint for multi-hour autonomous software engineering agents.

Key Points

Self-evaluation is a trap

  • Explanation: LLMs cannot reliably evaluate their own work objectively due to sycophancy and bias toward finishing tasks.
  • Evidence: Out of the box, Claude identifies legitimate issues but talks itself into approving flawed work anyway.
  • Practical implication: Always separate the builder from the judge by using an independent adversarial evaluator agent.

Compaction doesn't cure coherence drift

  • Explanation: Summarizing or compacting long session contexts leads to loss of detail and drifting off task.
  • Evidence: Long-running sessions degrade in coherence as context windows fill up.
  • Practical implication: Use structured handoffs, fresh context windows per session, and filesystem artifacts for state persistence.

Make subjective quality gradable

  • Explanation: Quality metrics like design taste and originality can be evaluated deterministically if broken down into explicit rubrics.
  • Evidence: Using a 4-criteria rubric (design, originality, craft, functionality) paired with Playwright navigation successfully prevented purple-gradient slop.
  • Practical implication: Write detailed rubrics for your evaluator agent rather than relying on vague prompts.

Read the traces

  • Explanation: Debugging long-running agents requires analyzing step-by-step logs rather than guessing at prompt failures.
  • Evidence: The primary debugging loop is reading evaluator logs, finding where judgment diverged, and updating prompts.
  • Practical implication: Build tooling to inspect agent transcripts and trace execution history.

Harnesses are on a diet as models improve

  • Explanation: As base models become more capable and context-aware, complex scaffolding can be removed.
  • Evidence: Moving from Opus 4.5 to 4.6 dropped context resets between sessions, sprint decomposition, and per-sprint evaluator cadence in favor of continuous builds.
  • Practical implication: Continuously re-evaluate your harness architecture as newer, smarter models are released.

Frameworks, Models & Processes

Generator-Evaluator Loop

  • How it works: A generator model builds an artifact, and an independent evaluator model tests and grades it against explicit criteria, feeding actionable critique back into the loop.
  • Components:
    • Generator (builds the thing)
    • Evaluator (grades the thing using tools like Playwright and rubrics)
    • Feedback channel (critique and refinement)
  • When to use: When building multi-hour autonomous coding or creative agents where self-verification is critical.

Three-Agent Full Stack Architecture

  • How it works: Splits tasks across three distinct agents—Planner, Generator, and Evaluator—coordinating via local files on disk with zero shared context.
  • Components:
    • Planner (breaks 1-line prompt into spec and sprint milestones)
    • Generator (executes one sprint at a time)
    • Evaluator (tests against contract via Playwright and rubric)
    • Filesystem (shared state via spec.md, app/git, findings.md)
  • When to use: For building complex, full-stack software applications autonomously over several hours.

Examples & Case Studies

Building a retro game maker and a browser Digital Audio Workstation (DAW) using solo agents vs. full multi-agent harnesses.

  • Illustrates: Solo agents failed at play mode and sprite interaction after 20 minutes, whereas the 3-agent harness successfully built fully featured, tested apps over 6 hours.
  • Lesson: Complex multi-hour tasks require structured planning, sprint contracts, and adversarial evaluation.

Actionable Takeaways

  • Immediate:
    • Use separate adversarial evaluator agents instead of letting models grade their own work.
    • Rely on filesystem state (spec.md, progress.md) instead of infinite context windows for long tasks.
    • Break subjective quality into explicit rubrics for automated evaluation tools like Playwright.
  • Strategic:
    • Harness design co-evolves with base model capabilities; be ready to strip out scaffolding as models improve.
    • Granular contract criteria (e.g., 27 specific criteria per sprint) prevent vague agent outputs.
    • Treat prompt engineering for agents like writing rigorous software specifications.
  • Questions to investigate:
    • How to scale multi-agent team communication without excessive API costs?
    • What are the optimal compaction strategies for long-running sessions in non-coding domains?
    • How to train models natively to be self-critical without requiring heavy external harnesses?

Claims Worth Verifying

  • Tuning a standalone evaluator to be skeptical is tractable, but making a generator self-critical is not. (empirical observation)
  • Models exhibit context anxiety and rush to finish tasks as they approach the end of their context window. (behavioral observation)

Notable Quotes

"The space of interesting harness combinations doesn't shrink as models improve. It moves." (at 27:22) "Out of the box, Claude is a poor QA agent. It would identify legitimate issues, then talk itself into deciding they weren't a big deal and approve the work anyway." (at 53:59) "Then Opus 4.6 shipped — and half of what we just showed you became dead weight." (at 56:55)

Compressed Summary

  • Self-evaluation in LLM agents fails due to sycophancy; adversarial evaluators are required.
  • Context rot requires structured filesystem handoffs and fresh sessions rather than pure compaction.
  • A three-agent architecture (Planner, Generator, Evaluator) successfully powers multi-hour autonomous software development.
  • As base models improve with native context awareness and reasoning, harness scaffolding shrinks and simplifies.
  • Keywords: ai agents, harness design, evaluator loop, context management, anthropic claude
  • Core insight: Building reliable long-running AI agents requires pairing capable base models with adversarial evaluator loops and structured filesystem state rather than relying on self-evaluation or infinite context windows.

Core insights

6
Architecturehigh noveltystrong evidence

Separating the builder from the judge is an architectural requirement, not a prompting trick: an independent adversarial evaluator agent can be tuned to be skeptical and harsh, whereas a generator cannot reliably be made self-critical. Self-evaluation fails because the model identifies real issues and then talks itself into approving the work anyway.

Why it matters

It relocates verification responsibility out of the generation context into a separately tunable agent, which changes how you allocate context, prompts, and compute in any agentic build pipeline.

Generalization

Any system where an LLM both produces and grades its own output (code review, document drafting, data labeling) should split the roles, because the failure is in role collision rather than in model capability.

Tuning a standalone evaluator to be skeptical and harsh is tractable; making a generator self-critical is not.
Open source video
Out of the box, Claude identifies legitimate issues but talks itself into approving flawed work anyway.
Open source video
Always separate the builder from the judge by using an independent adversarial evaluator agent.
Open source video
Architecturehigh noveltystrong evidence

The three-agent harness (Planner, Generator, Evaluator) coordinates with zero shared context, passing all state as files on disk, and requires Generator and Evaluator to negotiate 'sprint contracts' via files before any code is written.

Why it matters

It replaces the shared context window as the coordination substrate with an explicit, inspectable filesystem protocol, which is what allows multi-hour runs without context rot and makes inter-agent agreements auditable artifacts.

Generalization

Agent-to-agent coordination can be mediated by durable external artifacts (contracts, specs, state files) rather than shared in-context memory; the artifact becomes the interface contract and the observability surface.

A Planner agent breaks a high-level 1-line prompt into a high-level spec and sprint workflow.
Open source video
Generator and Evaluator negotiate sprint contracts via files before writing any code.
Open source video
Zero shared context; all coordination goes through files on disk to prevent context rot.
Open source video
Tradeoffhigh noveltymoderate evidence

Harness complexity is a depreciating asset: as models improved from Opus 4.5 to 4.6, previously necessary scaffolding (context resets between sessions, sprint decomposition, per-sprint evaluator cadence) was deleted in favor of continuous builds.

Why it matters

Contradicts the default instinct to accumulate orchestration layers; a harness that is optimal for one model generation becomes dead weight and even a source of failure for the next.

Generalization

Scaffolding written to compensate for a model weakness should be treated as versioned against model capability, with periodic deliberate removal experiments rather than permanent accretion.

Moving from Opus 4.5 to 4.6 dropped context resets between sessions, sprint decomposition, and per-sprint evaluator cadence in favor of continuous builds.
Open source video
Harnesses have co-evolved with models, starting with complex multi-prompt loops and shedding unnecessary scaffolding as models became more capable.
Open source video
Continuously re-evaluate your harness architecture as newer, sma
Open source video
Failure Modehigh noveltymoderate evidence

Compaction is not a fix for long-horizon coherence loss. Summarizing long sessions loses detail and causes task drift; the corrective pattern is structured handoffs, a fresh context window per session, and filesystem artifacts carrying state forward.

Why it matters

It invalidates the common assumption that context summarization is a sufficient long-session strategy, and points to context-window lifecycle management as a first-class design decision.

Generalization

Any long-running LLM workflow should treat context windows as disposable and externalize state, rather than trying to preserve fidelity by compression.

Summarizing or compacting long session contexts leads to loss of detail and drifting off task.
Open source video
Long-running sessions degrade in coherence as context windows fill up.
Open source video
Use structured handoffs, fresh context windows per session, and filesystem artifacts for state persistence.
Open source video
Mechanismhigh noveltymoderate evidence

Subjective qualities such as design taste and originality become machine-gradable when decomposed into explicit rubrics and paired with an agent that actually navigates the artifact (Playwright) rather than inspecting it abstractly.

Why it matters

It converts a supposedly un-automatable evaluation dimension into a deterministic-ish grading loop, enabling closed-loop iteration on aesthetic output without human review per iteration.

Generalization

Taste-like criteria (readability, API ergonomics, UX polish) can be operationalized as rubric checklists plus a tool-using judge that exercises the artifact, instead of being excluded from automated evaluation.

Subjective quality like design and originality can be graded using detailed rubrics applied by an adversarial evaluator.
Open source video
Using a 4-criteria rubric (design, originality, craft, functionality) paired with Playwright navigation successfully prevented purple-gradient slop.
Open source video
Write detailed rubrics for your evaluator agent rather than relying on vague prompts.
Open source video
Practicemedium noveltymoderate evidence

The primary debugging loop for long-running agents is reading evaluator traces: inspecting step-by-step transcripts to find where judgment diverged, then updating prompts, rather than guessing at prompt failures.

Why it matters

It makes trace infrastructure a prerequisite for operating agents, and reframes harness development as an iterative log-analysis workflow rather than prompt engineering by intuition.

Generalization

For any multi-step LLM system, the debugging unit is the transcript and the divergence point, not the final output; invest in trace inspection tooling before scaling runtime.

The primary debugging loop is reading evaluator logs, finding where judgment diverged, and updating prompts.
Open source video
Build tooling to inspect agent transcripts and trace execution history.
Open source video

Deep dives

5

Cost accounting for the adversarial evaluator

Research question

What does the separate evaluator agent cost in tokens, wall-clock time, and latency relative to the horizon extension it enables, and where is the evaluator-cadence frontier?

Why

The harness pattern is presented as a cost-bearing architectural addition, but without a cost/quality curve engineers cannot decide whether a second agent, sprint contracts, and rubric passes are worth the extra inference and coordination burden on real tasks.

Agents running for 5 to 6+ hours require careful harness design, not just one-shot model prompting.
Open source video
Models are poor judges of their own code output, exhibiting sycophancy and superficial testing.
Open source video
Source video

Harness-layer removal heuristics across model generations

Research question

What measurable signal indicates that a harness layer has become dead weight after a model upgrade rather than remaining load-bearing?

Why

Without a removal criterion, orchestration scaffolding accretes permanently and can silently degrade performance, turning an advantage tuned for one model generation into a liability on the next.

Harnesses have co-evolved with models, starting with complex multi-prompt loops and shedding unnecessary scaffolding as models became more capable.
Open source video
Model task completion horizons increased from 1 hour (Sonnet 3.7) to 12+ hours (Opus 4.6).
Open source video
Source video

Filesystem-mediated coordination past three roles

Research question

Does zero-shared-context, file-based coordination between Planner, Generator, and Evaluator still hold as agent count and task breadth increase beyond three roles?

Why

The architecture is demonstrated at exactly three agents on software construction; if artifact contention, contract negotiation overhead, or state divergence failures appear at higher agent counts, the protocol needs a versioning and locking model.

Context windows are finite and suffer from coherence degradation as sessions deepen.
Open source video
present practical fixes including model capability improvements, harness design patterns like the generator-evaluator loop (inspired by GANs), sprint contracts, and file system sharing.
Open source video
Source video

Domain observability dependence of the evaluator loop

Research question

How much of the generator-evaluator loop's discriminative power depends on a tool that provides ground-truth interaction (like browser navigation), and does it degrade in domains without an exercisable surface?

Why

All demonstrated examples have an interactive surface an evaluator can navigate; if rubric-only grading collapses in weakly observable domains such as database migrations or distributed configuration, the pattern's applicability is narrower than presented.

Models struggle with long-term planning, often trying to one-shot tasks or stopping prematurely.
Open source video
Curl testing instead of click testing in browser apps.
Open source video
Source video

Cross-domain rubric transfer for subjective quality

Research question

Are rubrics for subjective qualities reusable across artifact classes, or must each domain be hand-crafted, and how much evaluator judgment degrades on transfer?

Why

Treating taste as machine-gradable is only operationally useful if rubric authoring cost amortizes; per-domain hand-crafting makes the approach a bespoke service rather than a reusable harness capability.

Models are poor judges of their own code output, exhibiting sycophancy and superficial testing.
Open source video
Stubs features and moving on without checking backend implementation.
Open source video
Source video

Article ideas

4

Verification Is Not a Prompt: Why Self-Review Fails and Split Agents Don't

The generator's failure to reject its own flawed work is a role-collision defect, not a capability deficit, so the correct fix is architectural separation of builder and judge rather than better self-critique prompting.

Angle

Contrast the empirical failure pattern of self-review with the separately tunable adversarial evaluator, and trace the engineering consequences for context allocation, compute budgeting, and audit trails.

Source video

Your Agent Harness Is a Depreciating Asset

Scaffolding written to compensate for model weaknesses should be versioned against model capability and deliberately removed, because a harness tuned for one model generation becomes active overhead on the next.

Angle

Argue against the default instinct of perpetual orchestration accretion, using the co-evolution from Sonnet 3.5 through Opus 4.6 as the case and proposing removal experiments as a standing engineering practice.

Source video

Compaction Is a Lie: Context Windows Should Be Disposable

Summarization-based continuity is not a sufficient long-horizon strategy because it loses detail and causes task drift; the reliable pattern externalizes state into filesystem artifacts and treats each context window as disposable.

Angle

Reframe context-window lifecycle management as a first-class design decision rather than an optimization, and identify where structured handoffs beat compression.

Source video

Grading Taste: Turning Subjective Quality into a Deterministic-Enough Loop

Design taste and originality become machine-gradable when decomposed into explicit rubrics and paired with a judge that actually exercises the artifact, not one that inspects it abstractly.

Angle

Show how a supposedly un-automatable evaluation dimension can be converted into a closed iteration loop, and interrogate how much of the win comes from the rubric versus from tool-using navigation.

Source video

Project ideas

4

Harness Ablation Rig

new

Harness components can be classified as load-bearing or dead weight by ablating them across model generations, and the resulting task-completion horizon and coherence deltas form a repeatable removal heuristic.

Proof of concept

Instrument the three-agent harness so each layer (context resets, sprint decomposition, per-sprint evaluator cadence) can be toggled independently, then run the same task suite across two model checkpoints and record horizon, coherence, and token cost.

Measurement

Task-completion horizon, drift rate against a spec, and total token cost per completed task for each ablation condition, compared across model versions.

Source video

Evaluator Cost Frontier Sweeper

beyond-evals

There exists an evaluator-cadence setting that dominates alternatives on quality per unit of wall-clock time, and continuous evaluation is not always optimal as cadence increases.

Proof of concept

Instrument the generator-evaluator loop to report per-agent token and time splits, then sweep evaluator cadence from per-sprint to continuous while holding the task fixed.

Measurement

Quality score per task against wall-clock hours and tokens consumed, plotted to identify the Pareto frontier across cadence settings.

Source video

Cross-Domain Rubric Transfer Benchmark

beyond-evals

Rubrics written for one artifact class transfer to another with measurable but bounded degradation in evaluator discriminative power, rather than failing outright.

Proof of concept

Take a frontend-oriented rubric, apply it to a non-frontend artifact class produced by the same harness, and compare evaluator scores against human rankings.

Measurement

Rank correlation between rubric-based evaluator scores and human judgments per domain, and the score delta between native and transferred rubrics.

Source video

File-First Coordination Scaling Probe

gatehouse

Zero-shared-context filesystem coordination fails past a bounded agent count through artifact contention and contract negotiation overhead before it fails through state divergence.

Proof of concept

Extend the file-based protocol from three roles to five and above on the same task class, logging every file read, write, and contract negotiation step.

Measurement

Contract negotiation overhead per sprint, write-contention collisions, and state divergence incidents as a function of agent count.

Source video

Architectural implications

4

Verification is moved out of the generating context into a dedicated adversarial evaluator agent with its own tooling and rubrics.

Before

A single agent generates output and self-reviews it in the same context window.

After

A separate evaluator agent navigates the artifact (e.g., Playwright), scores against an explicit rubric, and returns structured critique to the generator.

Consequence

Verification becomes independently tunable and auditable, but introduces additional inference cost and a new tuning burden (making the evaluator appropriately harsh).

Source video

Coordination substrate shifts from the shared context window to concrete files on disk.

Before

Multiple roles share one accumulating context and implicit conversational state.

After

Planner, Generator, and Evaluator run with zero shared context and exchange specs, sprint contracts, and state via files.

Consequence

Context rot is avoided and every inter-agent agreement becomes a durable artifact, but the harness now needs file schemas, contract negotiation steps, and tooling to inspect the resulting state.

Source video

Harness layers are treated as removable rather than permanent.

Before

Orchestration scaffolding accumulates over time to compensate for model weaknesses.

After

Scaffolding is deleted when a model upgrade absorbs the capability (e.g., context resets and sprint decomposition dropped between Opus 4.5 and 4.6).

Consequence

Harness architecture must be re-validated on every model upgrade, adding a recurring migration cost but preventing stale complexity from degrading performance.

Source video

Task decomposition is delegated to a Planner agent instead of being fixed in the harness.

Before

The harness hard-codes sprint structure and planning steps in its control flow.

After

A Planner converts a one-line prompt into a high-level spec and sprint workflow that downstream agents follow.

Consequence

Planning becomes prompt- and model-mediated (and therefore improvable with model upgrades) rather than an invariant of the orchestration code.

Source video

Tradeoffs and failure modes

5

Self-evaluation

Benefit

Avoids the cost and complexity of a second agent and keeps the feedback loop fast.

Cost or risk

The generator acknowledges legitimate problems but approves flawed work anyway, shipping defects it already identified.

Out of the box, Claude identifies legitimate issues but talks itself into approving flawed work anyway.
Open source video
Source video

Context compaction for long sessions

Benefit

Keeps sessions within the context window and reduces token cost per step.

Cost or risk

Detail is lost and the agent drifts off task; coherence degrades as the window fills.

Summarizing or compacting long session contexts leads to loss of detail and drifting off task.
Open source video
Source video

Harness scaffolding depth

Benefit

Compensates for model weaknesses and enables longer autonomous runs on earlier model generations.

Cost or risk

Becomes unnecessary overhead once models improve, requiring deliberate removal and re-validation on model upgrades.

Moving from Opus 4.5 to 4.6 dropped context resets between sessions, sprint decomposition, and per-sprint evaluator cadence in favor of continuous builds.
Open source video
Source video

Superficial self-testing of generated software

Benefit

Fast iteration and apparent task completion without heavy test infrastructure.

Cost or risk

Verification exercises a proxy rather than the real interface, and stubbed features are accepted without backend checks, producing false confidence.

Curl testing instead of click testing in browser apps.
Open source video
Source video

Trace-driven debugging

Benefit

Locates the exact step where the evaluator's judgment diverged, enabling targeted prompt fixes.

Cost or risk

Requires building and maintaining transcript inspection tooling before the harness can be debugged at scale.

The primary debugging loop is reading evaluator logs, finding where judgment diverged, and updating prompts.
Open source video
Source video

Open questions

5

How much of the generator-evaluator loop's success depends on a tool that provides ground-truth interaction (Playwright driving a browser), and does it degrade in domains without such an exercisable surface?

Why unresolved

The demonstrated examples are frontend web pages, a retro game maker, and a browser music production app, all of which have an interaction surface an evaluator can navigate.

Research direction

Build the same loop for a domain with weak observability (e.g., database migrations, distributed system config) and measure whether rubric-only grading retains its discriminative power.

Source video

What signal tells an engineer that a harness layer is now dead weight after a model upgrade, rather than still load-bearing?

Why unresolved

The summary offers only one comparative data point (Opus 4.5 to 4.6) and frames the behavior as a general trend without a removal criterion.

Research direction

Ablate harness components across model generations and track task-completion horizon and coherence to derive a repeatable removal heuristic.

Source video

What does the adversarial evaluator cost in tokens, latency, and wall-clock time relative to the horizon extension it buys?

Why unresolved

The summary reports runtime gains (4 to 6 hour builds, 12+ hour horizons) but no cost accounting for the extra agent, sprint contracts, or rubric passes.

Research direction

Instrument the three-agent harness to report per-agent token and time split, then sweep evaluator cadence to find the cost/quality frontier.

Source video

Are rubrics for subjective quality reusable across tasks, or must they be hand-crafted per domain?

Why unresolved

One specific four-criteria rubric is described (design, originality, craft, functionality) and is tied to frontend output, with no evidence of transfer elsewhere.

Research direction

Test cross-domain rubric transfer and measure how much evaluator judgment degrades when a rubric from one artifact class is applied to another.

Source video

Does filesystem-mediated coordination with zero shared context still hold as the agent count and task breadth grow beyond three roles?

Why unresolved

The architecture is described and demonstrated at exactly three agents on software construction tasks; no scaling limits or contention behaviors are reported.

Research direction

Extend the file-based protocol to more roles and measure artifact contention, contract negotiation overhead, and state divergence failures.

Source video

Key claims

8
comparativeVerification needed

Model task completion horizons increased from roughly 1 hour on Sonnet 3.7 to 12+ hours on Opus 4.6.

Evidence

Model task completion horizons increased from 1 hour (Sonnet 3.7) to 12+ hours (Opus 4.6).

Question

On what task suite and with what harness was the 1-hour versus 12+ hour horizon measured, and does the horizon hold for real production tasks rather than benchmarks?

Source video
causalVerification needed

A standalone evaluator can be tuned to be skeptical and harsh, but a generator cannot be made reliably self-critical.

Evidence

Tuning a standalone evaluator to be skeptical and harsh is tractable; making a generator self-critical is not.

Question

Is the generator's failure caused by sycophancy, by a task-completion bias, or by role collision, and can any prompt or decoding intervention recover reliable self-critique?

Source video
causalVerification needed

Agents that run for 5 to 6+ hours require deliberate harness design rather than one-shot prompting.

Evidence

Agents running for 5 to 6+ hours require careful harness design, not just one-shot model prompting.

Question

At what runtime threshold does harness complexity start to dominate model capability, and is the threshold task-dependent?

Source video
causalVerification needed

Subjective quality attributes such as design and originality can be graded by an adversarial evaluator using detailed rubrics.

Evidence

Subjective quality like design and originality can be graded using detailed rubrics applied by an adversarial evaluator.

Question

How closely do rubric-based evaluator scores correlate with human judgments of the same artifacts?

Source video
factualVerification needed

A four-criteria rubric combined with Playwright navigation prevented low-quality default frontend output ('purple-gradient slop').

Evidence

Using a 4-criteria rubric (design, originality, craft, functionality) paired with Playwright navigation successfully prevented purple-gradient slop.

Question

Was this measured against a control condition, and how much of the effect came from the rubric versus from tool-based navigation of the page?

Source video
causalVerification needed

All coordination between the Planner, Generator, and Evaluator flows through files on disk with no shared context, which prevents context rot.

Evidence

Zero shared context; all coordination goes through files on disk to prevent context rot.

Question

Does filesystem-only coordination actually reduce context rot compared to a shared-context control, or does it mainly make failures inspectable?

Source video
causalVerification needed

Summarizing or compacting long session contexts causes loss of detail and task drift.

Evidence

Summarizing or compacting long session contexts leads to loss of detail and drifting off task.

Question

Which compaction strategies (hierarchical summaries, retrieval-based recall, structured handoffs) minimize drift, and at what session length does the effect appear?

Source video
factualVerification needed

Harness complexity was reduced after a model upgrade, dropping context resets between sessions, sprint decomposition, and per-sprint evaluator cadence in favor of continuous builds.

Evidence

Moving from Opus 4.5 to 4.6 dropped context resets between sessions, sprint decomposition, and per-sprint evaluator cadence in favor of continuous builds.

Question

Did output quality hold constant when those layers were removed, and is the removal attributable to the model upgrade rather than to concurrent prompt or tooling changes?

Source video

Connections

5