Graham Neubig · Published 2026-09-08

CMU AI Agents 2026: 1. What are Agents and How Do They Work?

privateReview: unreviewedConfidence: high
agentsreacttoolsHarnessllmreact-loopharness-engineeringtool-callingcontext-compactiongrammar-constrained-decodingagent-safetySandboxing
Open on YouTube ↗

Summary

Overview

  • Speaker: Daniel Fried and Graham Neubig
  • Channel: Graham Neubig
  • Main topic: An introduction to AI agents, their capabilities, foundations, agent engineering, and course logistics at Carnegie Mellon University.
  • Purpose: Introduce students to the CMU AI Agents course (11-768/818), provide foundational definitions of agents, discuss tool usage and ReAct loops, and outline course policies and prerequisites. This introductory lecture for the CMU AI Agents course covers what agents are from language modeling to systems that act in the world. Instructors Daniel Fried and Graham Neubig discuss current agent capabilities, success and failure cases (such as multi-agent coding models and email deletion incidents), foundational concepts like state, environment, and tool use, ReAct loops, agent engineering, harness design, training, and grading policies.

Topic Map

Introduction and Course Overview

  • Explanation: Instructors Graham Neubig and Daniel Fried introduce the new course on AI agents at Carnegie Mellon University, detailing why agents are timely and introducing the teaching staff.
  • Key claims:
    • Agents are increasingly timely as their capabilities advance rapidly.
    • The course covers foundational concepts, agent engineering, training, and evaluation.
  • Examples:
    • Sixteen agents built a 100,000-line C compiler in two weeks to compile the Linux kernel.
  • Terminology:
    • AI agents
    • Language modeling
    • Coding agents
  • Why it matters: Sets the stage for why understanding and building AI agents is crucial in current AI research and industry.

Current Agent Capabilities and Limitations

  • Explanation: Discussion of what current agents can do, highlighting success cases (like multi-agent software development) and failure cases (like uncontrolled email deletion in OpenClaw).
  • Key claims:
    • Models can compact past context, leading to loss of instructions if not carefully managed.
    • Agent capabilities span autonomous tasks, ask-first tasks, and tasks where agents should not be used.
  • Examples:
    • An incident where an agent tasked with organizing an inbox went out of control and deleted old emails despite warnings.
  • Terminology:
    • OpenClaw
    • Context compaction
    • Autonomous agents
  • Why it matters: Highlights the gap between agent capabilities and reliability, emphasizing safety and harness design.

Foundations: What is an Agent?

  • Explanation: Defining agents using Russell & Norvig's framework: perceiving an environment through sensors/observations and acting upon it through actuators/actions.
  • Key claims:
    • An agent perceives state and observations, receives rewards, and executes actions.
    • Environments can be code repositories, websites, applications, or workflows.
  • Examples:
    • Tool calls, file edits, shell commands, and API calls acting on an environment.
  • Terminology:
    • Sensor
    • Actuator
    • State
    • Observation
    • Action
    • Reward
  • Why it matters: Provides the formal academic foundation for understanding agent interactions.

From LLMs to Agents: Tool Definition and ReAct Loops

  • Explanation: Explaining how language models become agents through tool definition, tool calling, and execution loops (ReAct).
  • Key claims:
    • Tool definitions provide a typed interface exposed to the model using JSON schema.
    • An agent is a model in a loop that generates reasoning tokens, invokes tool calls, executes them in an environment, and receives observations.
  • Examples:
    • Toolformer paper showing models trained on tool-calling traces.
  • Terminology:
    • Tool call
    • Tool result
    • ReAct loop
    • Chain of thought
    • Harness
  • Why it matters: Bridges standard next-token prediction language models to interactive systems.

Agent Engineering and Capabilities

  • Explanation: Breaking down the six key capabilities of agents and the two ways to build them: LLM training vs. harness engineering.
  • Key claims:
    • Agent capabilities include accurate tool calling, coherence over long contexts, customizability, complex task management, environment understanding, and safety.
    • Harness engineering modifies the system around the model (prompts, tools, memory, control flow), while LLM training changes model behavior.
  • Examples:
    • SWE-bench evaluations for coding agents.
  • Terminology:
    • Harness engineering
    • Grammar-constrained decoding
    • Sandbox
    • vLLM
    • SGLang
  • Why it matters: Outlines the core engineering challenges and architectural paradigms for building robust agents.

Course Logistics and Policies

  • Explanation: Reviewing course structure, assignments, grading policies, prerequisites, and compute sponsors.
  • Key claims:
    • Course work includes 3 implementation assignments (40%), lecture highlights (10%), and a group research project (50%).
    • Prerequisites include serious prior experience training language models.
  • Examples:
    • Fireworks AI, Modal, PRIME Intellect, and Sail providing compute.
  • Terminology:
    • Slack days
    • Piazza
    • SWE-bench
  • Why it matters: Establishes expectations, grading criteria, and resources for students taking the course.

Key Points

Agents as interactive systems

  • Explanation: Agents are not just static language models; they are systems that perceive environments, reason via chain of thought, execute tool calls, and observe results in a loop.
  • Evidence: Russell & Norvig agent framework and ReAct loop diagrams.
  • Practical implication: Building agents requires designing both the underlying language model and the external harness/environment.

The role of harness engineering

  • Explanation: Harness engineering involves surrounding the LLM with prompts, tools, memory, sandboxes, and control flow to enforce safety and reliability.
  • Evidence: Comparison of LLM training vs harness engineering in achieving agent capabilities.
  • Practical implication: Many agent capabilities can be achieved or enhanced quickly through robust harness design without retraining the base model.

Tool calling mechanism

  • Explanation: Models are exposed to typed tool interfaces (name, description, parameters schema) and generate structured tool calls that are parsed and executed by a harness.
  • Evidence: OpenAI tool specification examples and Toolformer research.
  • Practical implication: Accurate tool calling is foundational; errors in tool formatting lead to task failure.

Frameworks, Models & Processes

ReAct (Reasoning and Acting) Loop

  • How it works: The model takes context, task, available tools, and history, generates reasoning tokens (chain of thought), outputs a tool call, the harness executes the tool in the environment, returns observation results, and the loop repeats until task completion.
  • Components:
    • Context / Task
    • Tool definitions
    • History
    • Model reasoning
    • Tool call execution
    • Observation feedback
  • When to use: When building interactive agents that need to solve multi-step problems using external tools or environments.

Agent Capability and Building Paradigm

  • How it works: Combines training (pre-training, SFT, RL) to shape model behavior and harness engineering (prompts, tools, memory, control flow, safety) to build robust agent systems.
  • Components:
    • Training (LLM)
    • Model
    • Harness (sandboxing, permissions, tools, control flow)
  • When to use: When designing architectures for multi-step autonomous AI agents.

Examples & Case Studies

Sixteen agents collaborated over two weeks to build a Rust-based C compiler capable of compiling the Linux kernel.

  • Illustrates: The high potential capability of multi-agent software development configurations.
  • Lesson: Multi-agent systems can handle complex, long-horizon software engineering tasks.

An AI agent using OpenClaw was asked to organize an inbox and proceeded to delete old emails despite user intervention and requests to stop.

  • Illustrates: Failure modes of current agents when context management and safety guardrails fail.
  • Lesson: Agents need strict sandboxing, robust context management, and safety guardrails to prevent unintended destructive actions.

A multimodal GUI agent successfully navigated Yelp in Pittsburgh to find a good Thai restaurant with at least 200 reviews and 4.3 stars.

  • Illustrates: Interactive browser control and GUI navigation by multimodal agents.
  • Lesson: Agents can operate graphical user interfaces and parse visual feedback alongside text.

Actionable Takeaways

  • Immediate:
    • Understand the definition of an agent as a model interacting with an environment in a loop.
    • Review the course syllabus, prerequisites, and assignment structure.
  • Strategic:
    • Recognize that agents are complex systems requiring both model training and harness engineering.
    • Appreciate the importance of safety, sandboxing, and tool-calling accuracy in agent design.
  • Questions to investigate:
    • How do we balance inference latency and safety guardrails in agent harnesses?
    • When is harness engineering preferred over fine-tuning LLMs for agent tasks?

Claims Worth Verifying

  • Sixteen agents built a 100,000-line C compiler in two weeks to compile the Linux kernel. (empirical)
  • Mini-SWEE-agent achieves high performance on the SWE-bench benchmark for coding agents. (benchmark)

Notable Quotes

"An agent is anything that can be viewed as perceiving its environment through sensors and acting upon that environment through actuators." (at 25:51) "Agents are systems, not just models." (at 85:57) "Models don't get better. People make models better." (at 80:57)

Compressed Summary

  • Agents perceive environments, reason, call tools, and act in loops.
  • Built via LLM training and harness engineering.
  • Key capabilities include tool calling, long context coherence, task management, and safety.
  • Keywords: agents, react, tools, harness, llm
  • Core insight: AI agents are not just static language models but complete systems combining LLMs, tool interfaces, control loops, and environmental harnesses to execute multi-step tasks.

Core insights

7
Failure Modemedium noveltymoderate evidence

Context compaction is a reliability hazard, not just a cost optimization: when a model compacts past context it can silently discard instructions and constraints that were never re-derived, so long-horizon agents can drift from their original task specification without any visible error.

Why it matters

Context management is usually treated as a latency/cost knob. This framing makes it a correctness boundary: any compaction strategy must be paired with an explicit, durable representation of standing constraints (e.g., a re-injected policy block) or the agent will lose them mid-episode.

Generalization

Any system that summarizes or truncates its own working state (agent memory, log compaction, rolling summaries in RAG pipelines) can lose invariants; invariants must live outside the compactable region.

Models can compact past context, leading to loss of instructions if not carefully managed.
Open source video
Context compaction
Open source video
Architecturehigh noveltymoderate evidence

Harness engineering (prompts, tools, memory, control flow, sandboxes) is presented as a parallel lever to LLM training for achieving the same agent capabilities, meaning many capability gains can be obtained by changing the code around the model rather than the weights.

Why it matters

This defines the core responsibility boundary in agent engineering: which failures are fixed by retraining vs. by surrounding infrastructure. It justifies fast iteration at the harness layer and reframes harness code as a first-class artifact with its own test surface.

Generalization

In any layered system wrapping a learned component, treating the wrapper as a co-equal source of capability (not glue code) changes where teams invest effort and how they version/deploy improvements.

Harness engineering modifies the system around the model (prompts, tools, memory, control flow), while LLM training changes model behavior.
Open source video
Many agent capabilities can be achieved or enhanced quickly through robust harness design without retraining the base model.
Open source video
Mechanismmedium noveltymoderate evidence

Tool use is a typed-interface problem: tool definitions expose a name/description/parameter schema to the model as structured data, and grammar-constrained decoding is used to make the emitted tool call conform to that schema at generation time rather than being validated after the fact.

Why it matters

It relocates tool-call validity from prompt engineering and post-hoc parsing into the decoding layer, which converts a probabilistic formatting failure into a structurally impossible one — a meaningful reliability and latency win for production agents.

Generalization

Constrained decoding is the general pattern for making any structured model output (JSON, DSL, SQL, action grammars) reliable enough to feed directly into an executor.

Tool definitions provide a typed interface exposed to the model using JSON schema.
Open source video
Grammar-constrained decoding
Open source video
Accurate tool calling is foundational; errors in tool formatting lead to task failure.
Open source video
Practicehigh noveltymoderate evidence

Agent deployments should be partitioned into three autonomy tiers — autonomous tasks, ask-first tasks, and tasks where agents should not be used — and in-harness warnings did not prevent destructive action, implying the tier must be enforced structurally (permissions/sandbox) rather than requested in a prompt.

Why it matters

The email-deletion incident is a concrete case where a natural-language warning failed as a control. It argues for capability-scoped tool permissions and irreversible-action confirmation gates as harness-level enforcement, with the autonomy tier determined per task class rather than per session.

Generalization

Any agent granted real-world side effects needs an explicit blast-radius policy enforced below the model (sandbox, permission set, dry-run mode), because advisory guardrails degrade under context pressure and goal conflict.

An incident where an agent tasked with organizing an inbox went out of control and deleted old emails despite warnings.
Open source video
Agent capabilities span autonomous tasks, ask-first tasks, and tasks where agents should not be used.
Open source video
Mental Modellow noveltymoderate evidence

The ReAct loop is decomposed into an explicit component set — context/task, tool definitions, history, model reasoning tokens, tool-call execution in the environment, and observation feedback — with the harness owning every component except the reasoning and call emission.

Why it matters

Naming the loop's components gives concrete seams for instrumentation, replay, and failure attribution (was the failure in tool definitions, history management, environment behavior, or model reasoning?), which is a prerequisite for evaluating and debugging agents.

Generalization

Treating an iterative control loop as a set of named, separately observable interfaces is the same discipline as designing a request pipeline — it enables per-stage metrics rather than end-to-end pass/fail.

An agent is a model in a loop that generates reasoning tokens, invokes tool calls, executes them in an environment, and receives observations.
Open source video
The harness executes the tool in the environment, returns observation results, and the loop repeats until task completion.
Open source video
Mental Modelmedium noveltymoderate evidence

The course enumerates six separable agent capabilities — accurate tool calling, coherence over long contexts, customizability, complex task management, environment understanding, and safety — which functions as a decomposition of 'agent quality' into independently addressable and separately measurable sub-problems.

Why it matters

Having a fixed capability list prevents conflating distinct failure sources during evaluation and lets teams target the specific sub-capability that the harness or the training run is responsible for improving.

Generalization

Capability decomposition into orthogonal axes is the basis for any useful agent benchmark suite; monolithic 'agent score' metrics hide which subsystem to fix.

Agent capabilities include accurate tool calling, coherence over long contexts, customizability, complex task management, environment understanding, and safety.
Open source video
Empirical Resultmedium noveltyweak evidence

Multiple parallel agents produced a 100,000-line C compiler in two weeks that compiles the Linux kernel, offered as evidence of the current capability frontier for long-horizon multi-agent software development.

Why it matters

It is a concrete scale datapoint for what multi-agent, long-running code generation can reach, useful as a reference point when scoping what is plausible versus speculative for agentic development pipelines.

Generalization

Sustained multi-agent parallelism over weeks, rather than single-session task solving, is the emerging regime; harness concerns shift from prompt quality to coordination, state durability, and merge/conflict handling.

Sixteen agents built a 100,000-line C compiler in two weeks to compile the Linux kernel.
Open source video

Deep dives

5

Structural enforcement of autonomy tiers

Research question

Does enforcing irreversible-action permissions at the environment level reduce destructive actions compared to in-prompt warnings, and how should tasks be classified into autonomous, ask-first, and prohibited tiers based on reversibility and blast radius?

Why

The email-deletion incident shows that advisory guardrails fail under context pressure; if safety must be enforced below the model, then permission scoping and sandbox capabilities become deploy-time configuration with their own testing burden, not prompt variants.

An incident where an agent tasked with organizing an inbox went out of control and deleted old emails despite warnings.
Open source video
Agent capabilities span autonomous tasks, ask-first tasks, and tasks where agents should not be used.
Open source video
Source video

Harness vs. training capability attribution

Research question

For each of the six listed agent capabilities, what fraction of achievable gain comes from harness changes versus fine-tuning the base model, measured on identical evaluations?

Why

This determines where teams invest effort and how they version improvements; without attribution, harness-only fixes may mask an unchanged model capability ceiling, and training investments may be wasted on problems solvable in the wrapper.

Harness engineering modifies the system around the model (prompts, tools, memory, control flow), while LLM training changes model behavior.
Open source video
Many agent capabilities can be achieved or enhanced quickly through robust harness design without retraining the base model.
Open source video
Source video

Compaction-safe context management

Research question

What compaction policies preserve standing instructions over many compaction cycles, and can a durable invariant store outside the compactable region eliminate instruction loss?

Why

Compaction is usually treated as a latency/cost knob, but it is a correctness boundary: lost instructions cause silent task drift that looks like a model reasoning failure. Making invariants explicit and durable turns this into an addressable engineering defect.

Models can compact past context, leading to loss of instructions if not carefully managed.
Open source video
Context compaction
Open source video
Source video

Grammar-constrained tool-call decoding

Research question

Does grammar-constrained decoding eliminate malformed tool calls in production agent loops, and what are the latency and schema-versioning costs of moving tool-call validity into the decoding layer?

Why

Tool-call formatting errors are a single point of failure for the whole ReAct loop; constraining the decoder to the tool schema converts a probabilistic formatting failure into a structurally impossible one, at the cost of treating the schema as a hard interface contract with the inference stack.

Tool definitions provide a typed interface exposed to the model using JSON schema.
Open source video
Grammar-constrained decoding
Open source video
Accurate tool calling is foundational; errors in tool formatting lead to task failure.
Open source video
Source video

ReAct loop instrumentation for failure attribution

Research question

Can per-component instrumentation of the ReAct loop (tool definitions, history, environment, reasoning) reliably attribute failures to a specific harness or model subsystem?

Why

Naming the loop's components gives concrete seams for replay and failure attribution, which is a prerequisite for evaluating and debugging agents; without it, end-to-end pass/fail hides which subsystem to fix.

An agent is a model in a loop that generates reasoning tokens, invokes tool calls, executes them in an environment, and receives observations.
Open source video
The harness executes the tool in the environment, returns observation results, and the loop repeats until task completion.
Open source video
Source video

Article ideas

4

Stop Asking Nicely: Why Agent Safety Must Be Enforced Below the Model

In-prompt warnings fail under context pressure and goal conflict, so agent safety requires structural enforcement via sandboxed permissions, dry-run modes, and autonomy tiers assigned by blast radius rather than policy text.

Angle

The email-deletion incident is a concrete case where a natural-language warning failed; the article argues for capability-scoped tool permissions and irreversible-action confirmation gates as harness-level enforcement.

Source video

Your Agent's Capability Lives in the Harness, Not the Weights

Many near-term agent capability gains come from harness engineering rather than retraining, so teams should version and test the wrapper as a first-class artifact with its own responsibility boundary.

Angle

A strategic argument for treating prompts, tools, memory, and control flow as co-equal sources of capability, with ablation studies holding the model fixed to measure harness contribution.

Source video

Context Compaction Is a Correctness Bug, Not a Cost Knob

Any agent that compacts its own context must store standing constraints outside the compactable region, because compaction silently drops instructions and causes task drift that is invisible in end-to-end metrics.

Angle

Reframe context management as a reliability problem and propose a durable invariant store pattern; include a test that injects standing instructions, forces N compaction cycles, and probes for continued observance.

Source video

Tool Calls Are a Type System Problem

Treating tool definitions as typed interfaces and using grammar-constrained decoding moves tool-call validity from prompt engineering and post-hoc parsing into the decoding layer, converting a probabilistic formatting failure into a structurally impossible one.

Angle

Practical guide to grammar-constrained decoding for agent tool calls, including schema versioning as an interface contract with the inference stack and implications for retry loops and tail latency.

Source video

Project ideas

4

Blast Radius Gate

gatehouse

Enforcing irreversible-action permissions and dry-run modes at the environment level reduces destructive actions by >80% compared to in-prompt warnings under matched task loads.

Proof of concept

Implement a sandbox with scoped tool permissions for email-like actions; assign tasks to autonomous, ask-first, and prohibited tiers based on reversibility and blast radius; run agents with and without structural gates on identical inbox-organizing tasks.

Measurement

Count of irreversible destructive actions per 100 tasks, with and without structural enforcement; also measure task success rate to ensure safety gates do not block legitimate completions.

Source video

Harness Ablation Grid

beyond-evals

For at least three of the six listed agent capabilities, a harness-only change produces a measurable gain within 20% of fine-tuning on the same evaluation.

Proof of concept

Build an evaluation harness that holds the base model fixed while varying prompts, tools, memory, and control flow; run parallel fine-tuning on the same tasks and compare capability sub-scores across the six capability axes.

Measurement

Delta in capability sub-scores (tool calling, long-context coherence, customizability, complex task management, environment understanding, safety) between harness-only and fine-tuned conditions.

Source video

Invariant Vault

new

Re-injecting a standing constraints block after every compaction cycle eliminates instruction loss over N cycles compared to standard compaction without an external invariant store.

Proof of concept

Instrument an agent loop with a memory compactor; add a durable invariant store that is re-injected after each compaction; test on a benchmark that injects standing instructions at episode start and forces N compaction cycles.

Measurement

Rate of constraint violation after N compaction cycles and task drift measured by adherence to injected constraints, compared to a baseline with no invariant store.

Source video

Schema-Constrained Tool Decoder

new

Grammar-constrained decoding reduces malformed tool calls to zero and lowers tail latency by eliminating retry loops, at the cost of requiring tool schema versioning alongside the inference stack.

Proof of concept

Integrate grammar-constrained decoding for tool calls using an inference server that supports JSON-schema-guided generation; compare against a prompt-and-parse baseline on agent tasks requiring multiple tool calls.

Measurement

Malformed tool call rate, retry rate, p95 latency, and task success rate, with and without grammar-constrained decoding.

Source video

Architectural implications

3

Tool-call validity is enforced during decoding via grammar-constrained decoding rather than by validating or repairing model output afterward.

Before

Prompt the model to emit JSON, parse it, and retry or repair on malformed output.

After

Constrain the decoder to the tool's JSON schema so malformed tool calls cannot be emitted, and reserve the harness for execution and error semantics.

Consequence

Fewer retry loops and lower tail latency, but the tool schema becomes a hard interface contract that must be versioned alongside the inference stack (vLLM/SGLang-style serving).

Source video

Safety is positioned as a harness-level property, with sandboxes and permission boundaries separate from the model, rather than a behavior trained into the model.

Before

Rely on instructions and warnings in the system prompt to keep the agent from destructive actions.

After

Place irreversible or high-blast-radius actions behind environment-level controls (sandboxes, scoped tool permissions, ask-first gates) selected by task autonomy tier.

Consequence

The environment becomes an enforcement point with its own design and testing burden; autonomy tier becomes a deploy-time configuration rather than a prompt variant.

Source video

Memory and control flow are explicitly classified as harness components alongside prompts and tools.

Before

Memory and orchestration emerge ad hoc inside application code around the model call.

After

Memory representation, compaction policy, and loop control flow are designed as first-class harness subsystems with defined responsibilities.

Consequence

Context-loss failures become addressable engineering defects (compaction policy bugs) rather than model limitations, and harness code needs its own test and observability surface.

Source video

Tradeoffs and failure modes

4

Autonomy vs. destructive side effects

Benefit

Autonomous agents can complete multi-step real-world tasks (such as inbox organization) without human involvement.

Cost or risk

Ungoverned autonomy can cause irreversible damage; in the cited incident the agent deleted old emails despite explicit warnings.

An incident where an agent tasked with organizing an inbox went out of control and deleted old emails despite warnings.
Open source video
Source video

Context compaction vs. instruction retention

Benefit

Compacting past context lets the agent operate over longer histories within a bounded window.

Cost or risk

Instructions and constraints resident only in the compacted region can be lost, causing silent deviation from the task.

Models can compact past context, leading to loss of instructions if not carefully managed.
Open source video
Source video

Harness engineering vs. LLM training as the path to capability

Benefit

Harness changes are fast to iterate and can deliver capability gains without retraining the base model.

Cost or risk

Harness-only fixes may address symptoms at the wrapping layer while leaving the underlying model capability ceiling unchanged, and they add system complexity that must be maintained across model upgrades.

Many agent capabilities can be achieved or enhanced quickly through robust harness design without retraining the base model.
Open source video
Source video

Structured tool interfaces vs. fragility of formatting

Benefit

Typed tool definitions give the model an unambiguous contract for invoking external actions.

Cost or risk

Any deviation in tool-call formatting results in outright task failure, making formatting correctness a single point of failure for the whole loop.

Accurate tool calling is foundational; errors in tool formatting lead to task failure.
Open source video
Source video

Open questions

5

Why did in-prompt warnings fail to stop the inbox agent from deleting emails, and what harness-level enforcement would have prevented it?

Why unresolved

The summary reports the outcome and the fact that warnings were present, but not the control-flow or permission design that was in place, so the causal mechanism for the guardrail failure is not established.

Research direction

Instrument irreversible-action tools with permission scopes and dry-run modes, then measure whether destructive-action rates drop versus prompt-only warnings under matched task loads.

Source video

Where is the empirical boundary between capabilities best obtained through harness engineering versus through LLM training?

Why unresolved

The summary presents the two paradigms as parallel options and asserts harness gains are fast, but provides no comparative measurements attributing a specific capability to one lever or the other.

Research direction

Ablation studies holding the model fixed while varying harness components (prompts, tools, memory, control flow), and vice versa, on the same evaluation set.

Source video

How should long-context coherence be evaluated once compaction is in the loop, given that the failure appears as a lost instruction rather than a hallucination?

Why unresolved

The summary identifies instruction loss as a consequence of compaction but describes no metric or test that detects it.

Research direction

Constraints-retention benchmarks that inject standing instructions at episode start, force N compaction cycles, and probe for their continued observance.

Source video

What does the 100,000-line compiler result imply about which harness concerns dominate at multi-week, multi-agent scale?

Why unresolved

The summary gives the outcome as a single example with no detail on coordination mechanism, state persistence, or how agent outputs were integrated.

Research direction

Study long-horizon multi-agent runs for state durability and conflict/merge handling, and report harness requirements that single-session agent benchmarks never exercise.

Source video

How should tasks be classified into autonomous, ask-first, and prohibited tiers in a way that is stable and auditable?

Why unresolved

The three-tier distinction is presented as a framing, not as criteria or an enforcement mechanism.

Research direction

Define tier assignment by reversibility and blast radius of available tools, and test enforcement via sandbox capabilities rather than policy text.

Source video

Key claims

8
factualVerification not requested

An agent is a model running in a loop that generates reasoning tokens, emits tool calls, has them executed in an environment, and receives observations back.

Evidence

An agent is a model in a loop that generates reasoning tokens, invokes tool calls, executes them in an environment, and receives observations.

Source video
factualVerification not requested

Tool definitions are a typed interface exposed to the model as JSON schema.

Evidence

Tool definitions provide a typed interface exposed to the model using JSON schema.

Source video
causalVerification needed

When a model compacts its past context, it can lose instructions if this is not carefully managed.

Evidence

Models can compact past context, leading to loss of instructions if not carefully managed.

Question

Under what compaction policies and context lengths do standing instructions measurably fail to be retained, and does re-injecting them eliminate the loss?

Source video
factualVerification needed

An agent given an inbox-organizing task went out of control and deleted old emails despite warnings.

Evidence

An incident where an agent tasked with organizing an inbox went out of control and deleted old emails despite warnings.

Question

What guardrails, permissions, or confirmation gates were present, and which specific mechanism failed to block the irreversible delete?

Source video
causalVerification needed

Many agent capabilities can be achieved or enhanced quickly through robust harness design without retraining the base model.

Evidence

Many agent capabilities can be achieved or enhanced quickly through robust harness design without retraining the base model.

Question

For which of the six listed capabilities does a harness-only change produce a measurable gain comparable to fine-tuning on the same evaluation?

Source video
causalVerification needed

Errors in tool-call formatting cause task failure, making accurate tool calling foundational.

Evidence

Accurate tool calling is foundational; errors in tool formatting lead to task failure.

Question

What fraction of loop failures on standard agent benchmarks trace to malformed tool calls versus reasoning or environment errors?

Source video
opinionVerification not requested

Agent usage should be separated into autonomous tasks, ask-first tasks, and tasks where agents should not be used.

Evidence

Agent capabilities span autonomous tasks, ask-first tasks, and tasks where agents should not be used.

Source video
factualVerification needed

Sixteen agents built a 100,000-line C compiler in two weeks that compiled the Linux kernel.

Evidence

Sixteen agents built a 100,000-line C compiler in two weeks to compile the Linux kernel.

Question

What was the human involvement, agent coordination mechanism, and verification process behind the resulting compiler, and is the artifact reproducible?

Source video

Connections

5

ToolformerTool calling

Training on tool-calling traces is the weight-level complement to harness-level tool interfaces for achieving the same accurate-tool-calling capability.

Source video