AI Engineer · Published 2026-04-17

Harness Engineering: How to Build Software When Humans Steer, Agents Execute — Ryan Lopopolo, OpenAI

privateReview: unreviewedConfidence: high
harness engineeringai agentscode generationsoftware architecturedeveloper workflowNon-functional requirementsguardrailscode review automationRepository modularitycontext window constraintsmulti-agent orchestration
Open on YouTube ↗

Summary

Overview

  • Speaker: Ryan Lopopolo
  • Channel: AI Engineer
  • Main topic: Harness Engineering for Software Development with AI Agents
  • Purpose: To educate AI engineers on how to build, structure, and operationalize codebases and team processes to maximize the productivity and reliability of AI coding agents. Ryan Lopopolo from OpenAI discusses harness engineering—building software systems where humans steer and AI coding agents execute. He explains how the role of software engineers has shifted from implementation to systems design and delegation, treating code as free and infinite, while human time, attention, and model context windows remain scarce resources.

Topic Map

The Shift in Software Engineering

  • Explanation: Software engineering has changed because code is now free and implementation is no longer the scarce resource.
  • Key claims:
    • Models are good enough to do full software engineering tasks.
    • Code is free and infinitely parallelizable.
    • Human time, attention, and model context windows are the only scarce resources.
  • Examples:
    • Banishing teams from touching editors and forcing them to work through models.
  • Terminology:
    • Harness Engineering
    • Token Billionaire
    • Agents Execute
  • Why it matters: Engineers must shift from writing code to systems design, prompt engineering, and unblocking teams of agents.

Scarce Resources in the Age of AI

  • Explanation: Analyzing what remains scarce when code generation is infinite: human time, attention, and model context windows.
  • Key claims:
    • Human time is the primary bottleneck in software delivery.
    • We must shift synchronous human time into high-leverage activities.
  • Examples:
    • Stack-ranking tasks where human intervention is required versus where agents can run in parallel.
  • Terminology:
    • Human Time
    • Model Attention
    • Model Context Window
  • Why it matters: Understanding scarce resources dictates where human engineers should spend their limited focus.

Harnessing Agents with Non-Functional Requirements

  • Explanation: How to specify non-functional requirements and guardrails so agents consistently produce acceptable code without slop.
  • Key claims:
    • Agents have seen trillions of lines of code and can make any choice; humans must specify non-functional requirements.
    • Writing linter rules, docs, and tests acts as guardrails for agents.
  • Examples:
    • Writing tests that limit files to 350 lines or enforcing retry/timeout patterns on network code.
  • Terminology:
    • Non-Functional Requirements
    • Guardrails
    • Auto-Compaction
    • Lint Failure
  • Why it matters: Clear guardrails prevent agents from introducing architectural drift or poor code quality into the repository.

Scaling Multiple Agents and Code Review

  • Explanation: Managing code review bottlenecks when agents generate high volumes of pull requests.
  • Key claims:
    • Human code review becomes a major bottleneck when agents produce high pull request volume.
    • Garbage collection days and automated review agents help close the loop.
  • Examples:
    • Setting aside Fridays as garbage collection days to eliminate merge conflicts and review slop.
  • Terminology:
    • Garbage Collection Day
    • Merge Conflicts
    • Review Agents
    • Symphony
  • Why it matters: Scaling agent output requires automated review and feedback loops to maintain codebase health.

Key Points

Code is free, implementation is not the bottleneck

  • Explanation: With powerful LLMs like GPT-5.2, producing code is virtually free and infinitely parallelizable.
  • Evidence: Engineers can fire off 15 agents in parallel to solve problems.
  • Practical implication: Engineering teams should focus on systems design and prompt architecture rather than manual coding.

Human time and attention are scarce

  • Explanation: Human oversight is the limiting factor in software velocity.
  • Evidence: Engineers spend too much time on low-signal code reviews instead of high-leverage guidance.
  • Practical implication: Automate routine checks and push feedback loops into automated tests and linter agents.

Repository structure dictates agent success

  • Explanation: Codebases must be structured so agents can reason locally and transfer context effectively.
  • Evidence: Monolithic codebases cause agents to fail due to lack of modular boundaries.
  • Practical implication: Organize code into clean subdirectories with clear interfaces and localized dependencies.

Frameworks, Models & Processes

Harness Engineering Framework

  • How it works: Humans steer agent execution by providing structured prompts, tests, linters, and architectural guardrails.
  • Components:
    • Human Steering
    • Agent Execution
    • Automated Guardrails
    • Continuous Feedback Loops
  • When to use: When developing software with AI coding agents at scale.

Examples & Case Studies

Banning teams from touching editors to force them to work entirely through models.

  • Illustrates: Leaning into the full capability of AI agents as software engineers.
  • Lesson: Forcing agent-first workflows exposes friction points in harness design.

Actionable Takeaways

  • Immediate:
    • Write explicit linter rules and tests to serve as guardrails for agents.
    • Use agent SDKs and prompts to automate repetitive refactoring tasks.
  • Strategic:
    • Shift engineering focus from implementation to systems architecture and agent orchestration.
    • Design repository structures for local observability and agent context efficiency.
  • Questions to investigate:
    • How do we handle human code review bottlenecks when agents generate massive PR volume?
    • What is the optimal balance between human steering and autonomous agent execution?

Claims Worth Verifying

  • Implementation is no longer the scarce resource of software engineering. (philosophical claim)
  • Every engineer has access to thousands of engineers worth of capacity 24/7. (hyperbolic projection)

Notable Quotes

"I am a token billionaire, and I believe that in order for us to get into our AGI future, we want everybody to be token billionaires." (at 0:46) "The way we build software has changed." (at 2:18) "Code is free." (at 3:35)

Compressed Summary

  • Code is free and infinitely parallelizable.
  • Human time and attention are the primary scarce resources.
  • Harness engineering requires robust linters, tests, and guardrails to guide agents.
  • Automated code review and feedback loops prevent merge bottlenecks.
  • Keywords: harness engineering, ai agents, code generation, software architecture, developer workflow
  • Core insight: Harness engineering replaces manual implementation with human steering and agent execution, transforming software engineering into systems design and orchestration.

Core insights

5
Architecturehigh noveltymoderate evidence

Non-functional requirements should be encoded as executable artifacts—linter rules, docs, and tests—that constrain agent output, because an agent exposed to trillions of code lines can pick any implementation without such guardrails.

Why it matters

Quality enforcement shifts from post-hoc PR feedback to pre-execution gate-keeping; engineering effort goes into writing constraints and standards rather than hand-correcting generated code.

Generalization

For any high-volume generator (agents, bulk codemods, synthetic data), maintain a repository-local set of executable 'constitution' tests that accepted outputs must satisfy.

Writing linter rules, docs, and tests acts as guardrails for agents.
Open source video
Writing tests that limit files to 350 lines or enforcing retry/timeout patterns on network code.
Open source video
Agents have seen trillions of lines of code and can make any choice; humans must specify non-functional requirements.
Open source video
Architecturehigh noveltymoderate evidence

Repository structure is a context-engineering decision: with agent execution, modularity and localized dependencies are prerequisites for agent success, because monolithic codebases blow past model context limits and cause agents to fail.

Why it matters

Architectural boundaries in agent-driven codebases define how much of the system an agent must load; poor boundaries make otherwise competent agents unreliable at scale.

Generalization

Design module interfaces and package boundaries as if they were context-window budgets; every dependency an agent needs should be reachable in few hops.

Monolithic codebases cause agents to fail due to lack of modular boundaries.
Open source video
Organize code into clean subdirectories with clear interfaces and localized dependencies.
Open source video
Practicehigh noveltymoderate evidence

Parallel agents shift the bottleneck to human code review; scaling requires automated review agents plus scheduled 'garbage collection days' to absorb merge conflicts and review slop before they compound.

Why it matters

Without a closed-loop cleanup cadence, firing many parallel agents creates adversarial merge-conflict pressure that negates their throughput advantage.

Generalization

Any cheap, continuous contribution pipeline needs automated triage/gating and periodic reconciliation to keep artifact accumulation from overwhelming humans.

Human code review becomes a major bottleneck when agents produce high pull request volume.
Open source video
Garbage collection days and automated review agents help close the loop.
Open source video
Setting aside Fridays as garbage collection days to eliminate merge conflicts and review slop.
Open source video
Mental Modelmedium noveltymoderate evidence

The binding constraints under abundance of code are human time, attention, and model context windows; therefore tasks should be stack-ranked by whether they require human intervention, with agents handling the parallelizable remainder.

Why it matters

Project planning and delegation should be driven by scarcity of human judgment and context, not by implementation capacity; this determines which tasks are synchronous and which are delegated to parallel agents.

Generalization

Adopt a resource allocation model where bottleneck resources are identified first (human attention, context), then schedule work to minimize consumption of them.

Human time is the primary bottleneck in software delivery.
Open source video
Stack-ranking tasks where human intervention is required versus where agents can run in parallel.
Open source video
We must shift synchronous human time into high-leverage activities.
Open source video
Predictionmedium noveltyweak evidence

The engineer's role in an agent-saturated codebase becomes harness design—specifying constraints, interfaces, and agent workflows—rather than implementation; at the extreme, teams forbid direct editor use to expose harness frictions.

Why it matters

If adopted, it changes hiring, career progression, and daily workflow: leverage comes from the quality of the harness and delegation decisions, not typing speed.

Generalization

When unit production is commoditized, coordination, standards, and feedback-loop design become the core engineering competence.

The role of software engineers has shifted from implementation to systems design and delegation.
Open source video
Banishing teams from touching editors and forcing them to work through models.
Open source video

Deep dives

4

Executable Non-Functional Requirements: The Guardrail Contract

Research question

Which classes of non-functional requirements can be reliably expressed as linter rules, docs, and tests that an agent must satisfy before merge, and which classes still require semantic human review?

Why

Quality enforcement shifts from post-hoc PR feedback to pre-execution gate-keeping; knowing the expressiveness boundary prevents false confidence in self-checking code.

Writing linter rules, docs, and tests acts as guardrails for agents.
Open source video
Writing tests that limit files to 350 lines or enforcing retry/timeout patterns on network code.
Open source video
Agents have seen trillions of lines of code and can make any choice; humans must specify non-functional requirements.
Open source video
Source video

Context-Window Architecture: Measuring Modularity's Effect on Agent Task Success

Research question

Does limiting an agent's needed context to a small, locally dependent module meaningfully increase task success relative to the same task in a monolithic codebase?

Why

If modularity is a reliability lever, architecture reviews should budget module sizes and dependency hops for agents' finite context windows.

Monolithic codebases cause agents to fail due to lack of modular boundaries.
Open source video
Organize code into clean subdirectories with clear interfaces and localized dependencies.
Open source video
Source video

Modeling Agent Parallelism, Review Throughput, and Garbage-Collection Cadence

Research question

At what number of concurrent agents does human code review become the binding bottleneck, and how do automated review agents and scheduled garbage-collection days change the optimal parallelism and recovery cadence?

Why

Teams adopting many parallel agents need quantified policies to avoid PR pileups and merge-conflict tax that erase throughput gains.

Human code review becomes a major bottleneck when agents produce high pull request volume.
Open source video
Garbage collection days and automated review agents help close the loop.
Open source video
Source video

Scarcity-Driven Task Scheduling in Human-Agent Teams

Research question

Can a task stack-ranking based on required human intervention and model context consumption predict better delivery outcomes than conventional priority queues?

Why

When code generation is abundant and human attention/context are scarce, planning should first classify tasks by whether they must be synchronous and then allocate remaining work to parallel agents.

Human time is the primary bottleneck in software delivery.
Open source video
Stack-ranking tasks where human intervention is required versus where agents can run in parallel.
Open source video
We must shift synchronous human time into high-leverage activities.
Open source video
Source video

Article ideas

4

Linters Are the New Contracts: How Executable Guardrails Replace Post-Hoc Code Review

Teams that encode non-functional requirements as linter rules/tests before delegating to agents produce code that converges to standards automatically, so code review should become exception handling instead of style policing.

Angle

Argues for shifting engineering effort from line-level review to authoring repository constraints.

Source video

Your Monolith Is Burning Context: Rearchitecting Repositories for Agent Work

Modularity is no longer just maintainability for humans; modules are context-window budgets, and repositories that fail to localize dependencies will make even strong agents fail routine tasks.

Angle

Technical architecture argument using context constraints as a first-class metric.

Source video

Garbage Collection Days: The Hidden Cadence That Makes Parallel Agents Profitable

Parallel agents shift work from code production to conflict resolution; teams that do not schedule cleanup days see throughput collapse under review and merge debt.

Angle

Operational/process argument for making cleanup rituals explicit in delivery cadence.

Source video

From Coder to Harness Engineer: What Working Through Agents Teaches About Leverage

The scarce skills in agentic development are systems design, constraint specification, and delegation; engineers who measure themselves in lines typed will lose to those who measure harness quality.

Angle

Career and role evolution, using editor bans as provocative evidence of agent-first workflow.

Source video

Project ideas

3

GuardRailBench: Quantifying Executable NFR Guardrails

beyond-evals

Encoding non-functional requirements as lint/test gates before agent execution increases first-pass merge-ready pull request rate by at least 25 percentage points versus an ungated agent baseline on matched code tasks.

Proof of concept

Using one codebase and a fixed task set, run N independent agents in two modes: (A) default repo, (B) same repo plus linter rules and tests asserting file-size limits, retry/timeout behavior, and layering constraints. Blind human reviewers judge merge-readiness without knowing mode.

Measurement

First-pass merge-ready rate, mean revision cycles/PR, violations of encoded NFRs per PR, time-to-merge.

Source video

Context-Slice Refactor Experiment

new

Refactoring a moderate monolith into context-sized modules with local dependency graphs improves agent task success by at least 20% against the unmodified monolith under identical context-window and agent settings.

Proof of concept

Select an existing monolith and representative feature tasks; ask the same agent with the same tools and context window to complete tasks before and after refactoring into subpackages with clear interfaces and localized dependencies.

Measurement

Task success rate, mean final context utilization, number of agent iterations, compile/test pass rate.

Source video

Agent PR Flow Simulator

gatehouse

A discrete-event model of parallel agents and serial human reviewers predicts that end-to-end throughput saturates after only 2–3 concurrent agents, and adding automated review plus garbage-collection slots raises the productive parallelism threshold by at least 3x.

Proof of concept

Build a small simulator parameterized by measured agent PR rate, human review service time, conflict probability, and GC cadence; validate predictions against a team pilot running batches of 1–10 agents.

Measurement

Throughput at saturation, queued PR backlog, conflict count, merge latency; simulated vs pilot outcomes.

Source video

Architectural implications

4

Codebase structure is a first-class determinant of agent task success.

Before

Repos were modularized primarily for human readability and team ownership.

After

Repos should be modularized so each task needs only a context-sized slice of code with clear interfaces.

Consequence

Architecture reviews must include agent-context cost: dependency proximity and module size become reliability levers.

Source video

Quality constraints that humans once enforced during review can be shifted into linters and tests for agents.

Before

Non-functional requirements were transmitted via PR review comments and human memory.

After

Non-functional requirements are written as executable repo rules and test assertions before agent work begins.

Consequence

Agent output converges to standards automatically; reviewers focus on semantic exceptions rather than rote standards.

Source video

Agent throughput creates a PR pipeline that exceeds human review capacity.

Before

Team pulled each PR into human review queue synchronously; queue length limited velocity.

After

Automated review agents gate PRs and a scheduled garbage-collection day reconciles merge conflicts.

Consequence

Human review becomes a sampled exception process rather than a serial gate on every change.

Source video

High-leverage human time is spent designing harnesses and interventions, not implementing routine tasks.

Before

Sprint planning assigns implementation work to human developers sequentially.

After

Humans decide which narrow tasks need intervention and launch parallel agents for everything else.

Consequence

Team structure and rituals turn into orchestration (context supply, constraint setting, exception handling), not execution.

Source video

Tradeoffs and failure modes

4

Giving agents freedom without non-functional guardrails

Benefit

Agents can choose any code shape quickly because they have pattern knowledge across trillions of lines.

Cost or risk

Output drifts architecturally and becomes 'slop'—inconsistent, low-quality files that fail established conventions.

Agents have seen trillions of lines of code and can make any choice; humans must specify non-functional requirements.
Open source video
Source video

Scaling agent parallelism vs. human review throughput

Benefit

Parallel agents multiply implementation throughput (e.g., 15 agents at once).

Cost or risk

Human code review becomes the serial bottleneck; conflicts and review slop accumulate and derail velocity.

Human code review becomes a major bottleneck when agents produce high pull request volume.
Open source video
Source video

Forcing an agent-first workflow by banning editor use

Benefit

Exposes friction points in the harness so teams fix them explicitly instead of bypassing agents.

Cost or risk

Losing the safety valve of direct manual editing during the transition; teams must tolerate reduced short-term flexibility.

Forcing agent-first workflows exposes friction points in harness design.
Open source video
Source video

Running parallel agents vs. accumulating merge conflicts

Benefit

Constant, parallel code generation keeps momentum and throughput high.

Cost or risk

Merge conflicts compound, demanding dedicated cleanup days that consume capacity and dilute focus.

Setting aside Fridays as garbage collection days to eliminate merge conflicts and review slop.
Open source video
Source video

Open questions

4

How do we handle human code review bottlenecks when agents generate massive PR volume?

Why unresolved

The talk only names high-level remedies (review agents, garbage collection days) without specifying accuracy, cost, or operating parameters.

Research direction

Measure automated review agents against human reviewers (precision/recall, latency, missed issues) and design audit sampling controls.

Source video

What is the optimal balance between human steering and autonomous agent execution?

Why unresolved

The right split likely varies by codebase, task, and risk; no framework in the talk states how to find the optimum.

Research direction

A/B test different amounts of human intervention/steering against agent success rates, merge conflict counts, and on-call incidents.

Source video

When is it actually productive to ban manual code editing and route all work through models?

Why unresolved

The example is an intentional forcing function; its costs outside of harness-friction discovery are not quantified.

Research direction

Compare forced-agent teams vs. hybrid (human edit + agent) teams on code quality, throughput, and developer satisfaction.

Source video

What kinds of non-functional requirements can be reliably expressed as automated tests/lint rules for agents?

Why unresolved

Examples (350-line file limit, retry/timeout patterns) are narrow; larger architectural invariants may not be machine-checkable today.

Research direction

Catalog which NFR categories (size, coupling, security, observability) can be guardrailed and which require semantic review.

Source video

Key claims

6
opinionVerification not requested

Code is free and infinitely parallelizable.

Evidence

Code is free.

Source video
predictionVerification needed

Models are good enough to do full software engineering tasks.

Evidence

Models are good enough to do full software engineering tasks.

Question

What evaluation exercises full end-to-end software engineering tasks and measures completion reliability?

Source video
causalVerification needed

Monolithic codebases cause agents to fail due to lack of modular boundaries.

Evidence

Monolithic codebases cause agents to fail due to lack of modular boundaries.

Question

Is there controlled evidence linking repository modularity (or context length) to agent task success rate?

Source video
causalVerification needed

Human code review becomes a major bottleneck when agents produce high pull request volume.

Evidence

Human code review becomes a major bottleneck when agents produce high pull request volume.

Question

At what PR volume per reviewer does agent-generated output exceed human review capacity in practice?

Source video
predictionVerification needed

Implementation is no longer a scarce resource in software engineering.

Evidence

Implementation is no longer the scarce resource of software engineering.

Question

How does one measure 'implementation scarcity' given that generation is cheap but correctness, security, and integration still require judgment?

Source video
predictionVerification needed

Every engineer has access to thousands of engineers worth of capacity 24/7.

Evidence

Every engineer has access to thousands of engineers worth of capacity 24/7.

Question

What metric, e.g., tokens/PRs generated per day, supports or refutes this equivalency?

Source video

Connections

5