AI Engineer · Published 2026-10-06

Harness Engineering: How to Build a Software Factory — Dru Knox, Tessl

privateReview: unreviewedConfidence: high
Harness engineeringSoftware Factoryagentic developmentcontrol planeimprovement loopsharness-engineeringagentic-sdlccontrol-planeimprovement-loopsautonomy-metricsskills-registryexecution-environmentssoftware-factory
Open on YouTube ↗

Summary

Overview

  • Speaker: Dru Knox
  • Channel: AI Engineer
  • Main topic: Harness engineering and the architecture of agentic software factories
  • Purpose: To introduce harness engineering as an essential discipline for developers transitioning to agentic software factories and explain its core loops, stack components, and practical implementation. Dru Knox, Head of Product & Design at Tessl, presents harness engineering as the new core discipline for agentic software developers. He explains how software factories shift human effort from building individual features to building and improving the loops, control planes, and agent IT layers that automate software development end-to-end. The talk covers the three loops of harness engineering (inner loop, outer loop, and meta loop), the key metrics of autonomy, automation, and quality, and how Tessl's platform facilitates building agentic software factories.

Topic Map

Introduction to Software Factories

  • Explanation: Defining a software factory as an agentic system where the final product is built entirely by agents, while humans shape and improve automation, autonomy, and quality.
  • Key claims:
    • Software factories are agentic systems where the end product is built entirely by agents.
    • Humans only shape the work and improve automation, autonomy, and quality.
  • Examples:
    • Internal tool builders creating agentic systems that produce software products.
  • Terminology:
    • Software factory
    • Agentic system
    • Automation
    • Autonomy
    • Quality
  • Why it matters: Sets the vision for moving beyond interactive coding sessions to fully automated software production.

Harness Engineering and the Three Loops

  • Explanation: Explaining harness engineering as the direct creation, monitoring, and improvement of a project's loops, broken into inner, outer, and meta loops.
  • Key claims:
    • Harness engineering is the new core discipline of agentic development.
    • There are three core loops: inner loop (drives automation), outer loop (drives autonomy), and meta loop (drives quality).
  • Examples:
    • Inner loop: agent edits, runs, and checks code before a PR.
    • Outer loop: agent runs tests, linting, and architecture reviews after a PR is opened.
    • Meta loop: observability over output that feeds improvements back into inner and outer loops.
  • Terminology:
    • Harness engineering
    • Loop engineering
    • Inner loop
    • Outer loop
    • Meta loop
  • Why it matters: Provides a structural framework for how developers should engineer agent systems rather than hand-building individual features.

Components of the Agentic Stack

  • Explanation: Breaking down the three layers of a software factory: Control Plane, Agent IT, and Improvement Loops.
  • Key claims:
    • The agentic stack consists of three layers: Control Plane, Agent IT, and Improvement Loops.
    • Control plane handles issue trackers, PR reviews, and skills registries.
    • Agent IT manages company brains, internal services, production logs, and execution environments.
  • Examples:
    • Linear and GitHub integrations for issue tracking and PR review.
    • Weekly repo maintenance scans and automated playbooks.
  • Terminology:
    • Control plane
    • Agent IT
    • Improvement loops
    • Skills registry
    • Execution environment
  • Why it matters: Gives developers a concrete architecture to build towards when constructing a software factory.

Key Points

The three key metrics of software factories

  • Explanation: Automation measures how much human correction agents need; Autonomy measures oversight needed; Quality measures end-product excellence.
  • Evidence: Presented as the core pillars to measure when building towards a software factory.
  • Practical implication: Teams must improve autonomy first, then automation, while keeping quality constant and ultimately raising it.

Why harness engineering is hard

  • Explanation: Involves knowledge gaps due to fast-moving best practices, unplanned work when agents fail, competition with shipping features, and poor legibility of local agent logs.
  • Evidence: Derived from the psychological and operational challenges faced by teams adopting agentic workflows.
  • Practical implication: Teams need upfront strategies to surface local agent workflows and knowledge onto public, accessible surfaces.

Tessl's platform and agentic SDLC

  • Explanation: Tessl offers an agentic platform to help set up control planes, skills registries, maintenance agents, and automated improvement loops.
  • Evidence: Demonstrated via Tessl Agent and terminal workflows running tessl commands.
  • Practical implication: Allows engineering teams to adopt agentic workflows incrementally rather than building an entire factory from scratch.

Frameworks, Models & Processes

The Three Loops of Harness Engineering

  • How it works: Two loops run on code while a third improves the loops themselves.
  • Components:
    • Inner loop (drives automation)
    • Outer loop (drives autonomy)
    • Meta loop (drives quality)
  • When to use: When designing the architecture of an agentic software development pipeline.

The Agentic Stack (Three Layers)

  • How it works: Three hierarchical layers required to make a software factory run successfully.
  • Components:
    • Layer 01: Control plane (issue tracker, PR review, skills/plugin registry)
    • Layer 02: Agent IT (company brain, internal services, production logs, execution environments)
    • Layer 03: Improvement loops (repo maintenance, development playbooks, task automation, output quality optimization)
  • When to use: When organizing infrastructure and tooling for agentic development.

Examples & Case Studies

Setting up repository maintenance and issue tracking integration with Tessl agent in the terminal.

  • Illustrates: How agentic workflows can automate flaky test detection and repo cleanup.
  • Lesson: Automating maintenance tasks reduces unplanned engineering toil.

Actionable Takeaways

  • Immediate:
    • Track manual takeovers, human PR comments, auto-initiated PRs, and quality signals.
    • Start building control planes with issue tracker and GitHub integrations.
  • Strategic:
    • Shift focus from hand-building individual features to building and improving the loops that build them.
    • Move agent workflows from local machines to shared, legible execution environments.
  • Questions to investigate:
    • How can we minimize manual takeovers in our agentic pipelines?
    • What core playbooks can be codified into agent skills?

Claims Worth Verifying

  • A software factory is an agentic system where the final product is built entirely by agents. (Definition)
  • Harness engineering is the direct creation, monitoring, and improvement of a project's loops. (Definition)

Notable Quotes

"A software factory is an agentic system where the final product is built entirely by agents." "Stop hand-building features and start building the machine that builds them."

Compressed Summary

  • Software factories shift developer focus to building agentic systems that produce software.
  • Harness engineering encompasses the inner, outer, and meta loops of agentic development.
  • The agentic stack requires three layers: control plane, agent IT, and improvement loops.
  • Metrics to track include automation, autonomy, and quality.
  • Tessl provides tools to implement control planes and automated improvement loops incrementally.
  • Keywords: harness engineering, software factory, agentic development, control plane, improvement loops
  • Core insight: Harness engineering is the new core discipline for agentic developers, focusing on building and optimizing the loops and control planes that power autonomous software factories.

Core insights

6
Architecturehigh noveltymoderate evidence

Harness engineering is defined as a recursive architecture: two loops operate on code (inner loop drives automation before a PR; outer loop drives autonomy after a PR opens) while a third meta loop observes output and feeds improvements back into the other two loops. The harness itself is the artifact under continuous improvement, not just the code it produces.

Why it matters

It reframes the engineering target from hand-built features to self-modifying pipeline loops, which changes what you instrument, version, and own; the loops become first-class products with their own quality metric.

Generalization

Any agentic pipeline can be decomposed into code-transforming loops plus a separate loop that improves those loops, giving a concrete place to attach observability and feedback.

There are three core loops: inner loop (drives automation), outer loop (drives autonomy), and meta loop (drives quality).
Open source video
Two loops run on code while a third improves the loops themselves.
Open source video
Meta loop: observability over output that feeds improvements back into inner and outer loops.
Open source video
Tradeoffhigh noveltymoderate evidence

The three factory metrics map to distinct loop layers (automation=inner, autonomy=outer, quality=meta), and the speaker prescribes a sequencing constraint: raise autonomy first, then automation, while holding quality constant and only later raising it.

Why it matters

It argues against naive parallel optimization of all three metrics and implies an ordering discipline — you cannot safely expand automation before the oversight/autonomy layer is trustworthy.

Generalization

For staged rollout of autonomous systems, expand the scope of machine action (automation) only after the oversight and containment layer (autonomy) is established, keeping a fixed quality floor.

Teams must improve autonomy first, then automation, while keeping quality constant and ultimately raising it.
Open source video
Automation measures how much human correction agents need; Autonomy measures oversight needed; Quality measures end-product excellence.
Open source video
Architecturemedium noveltymoderate evidence

The agentic stack is layered so that control-plane concerns (issue tracker, PR review, skills/plugin registry) are separated from runtime/infrastructure concerns (company brain, internal services, production logs, execution environments), with improvement loops as a third layer above both.

Why it matters

It gives explicit placement for responsibilities — coordination and policy at the control plane, execution and context at the agent IT layer — which affects where you put integrations, secrets, and observability rather than lumping them into one agent.

Generalization

Separating a coordination/control plane from an execution/context layer is a reusable boundary for multi-agent systems, letting each layer scale and be replaced independently.

The agentic stack consists of three layers: Control Plane, Agent IT, and Improvement Loops.
Open source video
Control plane handles issue trackers, PR reviews, and skills registries.
Open source video
Agent IT manages company brains, internal services, production logs, and execution environments.
Open source video
Failure Modemedium noveltymoderate evidence

Adoption of harness engineering is blocked by legibility: local agent logs are poorly observable and knowledge stays on individual machines, so the recommended remedy is moving agent workflows from local machines to shared, legible execution environments and surfacing knowledge onto public accessible surfaces.

Why it matters

Without shared legible execution surfaces the meta loop has no reliable input, so the improvement loop silently fails; this makes observability infrastructure a precondition rather than an add-on.

Generalization

Self-improving agent systems require their execution telemetry to be centralized and shared, because per-developer local logs cannot feed collective improvement.

poor legibility of local agent logs
Open source video
Teams need upfront strategies to surface local agent workflows and knowledge onto public, accessible surfaces.
Open source video
Move agent workflows from local machines to shared, legible execution environments.
Open source video
Practicemedium noveltymoderate evidence

Progress is instrumented with concrete proxies for the autonomy/automation metrics: manual takeovers, human PR comments, auto-initiated PRs, and quality signals are the tracked events.

Why it matters

It converts an abstract maturity goal into measurable event counts, making it feasible to build dashboards and regressions around agentic workflows rather than relying on subjective judgment.

Generalization

Autonomy and automation can be measured indirectly via intervention and self-initiation counters (human interruptions, machine-opened changes) rather than requiring a bespoke metric.

Track manual takeovers, human PR comments, auto-initiated PRs, and quality signals.
Open source video
Mechanismmedium noveltyweak evidence

Codifying core playbooks into reusable agent skills (via a skills/plugin registry) is framed as an open area to investigate, implying the registry of codified playbooks is a bottleneck for scaling automation.

Why it matters

If playbooks are the reusable unit of automation, then the skills registry becomes a shared dependency whose design (discovery, versioning, reuse) determines how fast new workflows can be automated.

Generalization

Turn recurring operational procedures into registry-distributed, versioned capabilities so agents can reuse them instead of re-deriving them per task.

What core playbooks can be codified into agent skills?
Open source video
Control plane handles issue trackers, PR reviews, and skills registries.
Open source video

Deep dives

5

Separating autonomy and automation metrics

Research question

Can intervention-rate (manual takeovers, human PR comments) and self-initiation-rate (auto-initiated PRs) be measured independently enough to enforce autonomy-before-automation sequencing without confounding with quality?

Why

Without separable metrics, teams cannot validate the prescribed rollout order, and automation may expand before the oversight layer is trustworthy, creating unplanned work.

Track manual takeovers, human PR comments, auto-initiated PRs, and quality signals.
Open source video
Teams must improve autonomy first, then automation, while keeping quality constant and ultimately raising it.
Open source video
Source video

Meta loop convergence and dampening

Research question

What stabilization, filtering, and rollback mechanisms prevent the meta loop from oscillating or overfitting to noisy quality signals?

Why

If the meta loop chases noise, loop changes may not persist and the improvement layer becomes a source of variance rather than compounding gains.

Meta loop: observability over output that feeds improvements back into inner and outer loops.
Open source video
The talk covers the three loops of harness engineering (inner loop, outer loop, and meta loop), the key metrics of autonomy, automation, and quality
Open source video
Source video

Shared legible execution as precondition for meta loop

Research question

Does moving agent execution from local machines to shared, legible environments causally improve meta-loop improvement rates, and what minimal telemetry makes local knowledge observable?

Why

If local logs remain illegible, the meta loop has no reliable input, so the improvement layer silently fails regardless of how well inner and outer loops are built.

Teams need upfront strategies to surface local agent workflows and knowledge onto public, accessible surfaces.
Open source video
Move agent workflows from local machines to shared, legible execution environments.
Open source video
Source video

Minimum viable control plane

Research question

Which control-plane components (issue tracker, PR review, skills registry) are necessary for a small team to obtain meta-loop benefits without adopting the full agent IT layer?

Why

A minimal path lowers adoption cost and tests whether the control plane or the execution layer is the true bottleneck for loop improvement.

Control plane handles issue trackers, PR reviews, and skills registries.
Open source video
The agentic stack consists of three layers: Control Plane, Agent IT, and Improvement Loops.
Open source video
Source video

Playbook codification criteria for skills registries

Research question

What criteria determine which recurring playbooks should be codified into versioned agent skills, and how does registry discovery/versioning design affect reuse and human correction?

Why

If the registry is the distribution bottleneck, unclear codification criteria slow automation scaling and produce unusable skill sprawl.

What core playbooks can be codified into agent skills?
Open source video
Control plane handles issue trackers, PR reviews, and skills registries.
Open source video
Source video

Article ideas

4

Autonomy Before Automation: The Sequencing Discipline Agentic Software Factories Need

Expanding automation before the oversight layer is trustworthy increases unplanned work and erodes quality; teams should raise autonomy first, hold quality constant, then automate.

Angle

Argue against parallel optimization of automation, autonomy, and quality, using the three-loop model to show where each metric actually lives.

Source video

Your Agent Logs Are a Liability: Shared Legible Execution as the Precondition for Self-Improving Factories

Local agent logs make the meta loop blind; moving execution to shared, legible environments is not an ops nicety but a precondition for collective improvement.

Angle

Treat observability infrastructure as a first-class control-plane concern, not a post-hoc add-on.

Source video

The Skills Registry Is the Real Control Plane Bottleneck

Scaling agentic automation depends less on model capability than on designing discovery, versioning, and reuse for codified playbooks; the registry determines how fast new workflows can be automated.

Angle

Reframe agent skills as versioned operational capabilities rather than prompts.

Source video

Measure Interventions, Not Vibes: Proxies for Autonomy and Automation in Agentic SDLC

Manual takeovers, human PR comments, and auto-initiated PRs are sufficient proxies to drive staged autonomy/automation rollout; without them, maturity claims are subjective.

Angle

Show how to build dashboards and regression alerts around agentic workflows using intervention counters.

Source video

Project ideas

4

Autonomy-Automation Split Dashboard

beyond-evals

A dashboard built on manual takeover counts, human PR comments, and auto-initiated PRs can distinguish autonomy from automation and predict quality regressions before they occur.

Proof of concept

Instrument an agentic repository with event counters for manual takeovers, human PR comments, auto-initiated PRs, and quality signals; run a staged rollout raising autonomy first and then automation; display time-series metrics.

Measurement

Correlation between intervention-rate drop and auto-initiation increase; quality regressions (test failures, rework) before/after the prescribed sequencing.

Source video

Meta Loop Damper

movement-lab

Adding dampening, signal filtering, and a change budget to the meta loop reduces oscillation and overfitting compared with an unconstrained feedback loop.

Proof of concept

Build a small agentic pipeline or simulation with inner, outer, and meta loops; compare an unconstrained meta loop against a damped version that filters noisy quality signals and limits loop changes per cycle.

Measurement

Persistence of loop changes across iterations, oscillation count, quality variance, and net improvement over baseline.

Source video

Minimal Control Plane

gatehouse

A reduced control plane with only issue tracker and PR review integrations yields measurable legibility and improvement-loop benefits for a small team without adopting the full agent IT layer.

Proof of concept

Stand up a minimal control plane for a small agentic repository; compare a four-week run against a local-only workflow with no shared execution/observability surfaces.

Measurement

Time-to-legible agent logs, number of meta-loop-driven improvements, human correction rate, and manual takeovers.

Source video

Skills Registry Reuse Experiment

new

Codifying a subset of recurring playbooks as versioned agent skills reduces human correction per task compared with ad hoc prompting.

Proof of concept

Catalog recurring repository operations; implement three to five as registry-distributed skills; run an A/B test on agent tasks with and without access to the registry skills.

Measurement

Human PR comments and manual takeovers per task, time-to-PR, reuse counts, and quality signals.

Source video

Architectural implications

2

Agent coordination responsibilities (issue tracking, PR review, skills registry) are pulled out of the human IDE workflow into an explicit control plane.

Before

Developers drive issue tracking and PR review interactively inside their editor and local git workflow.

After

A control plane layer owns issue tracker and GitHub integrations, PR review, and a skills/plugin registry.

Consequence

Coordination logic becomes centralized and automatable, but the control plane becomes a critical integration surface whose availability and permissions gate all agentic work.

Source video

Agent execution moves from individual local machines to shared execution environments.

Before

Agent runs and logs live on each developer's local machine, largely illegible to others.

After

Execution environments are managed as part of the agent IT layer and shared across the team.

Consequence

Enables the meta loop to observe real output and improves legibility, at the cost of building and operating shared execution infrastructure.

Source video

Tradeoffs and failure modes

3

Adopting harness engineering vs shipping features

Benefit

Building and improving loops compounds automation and reduces long-term toil.

Cost or risk

Competition with shipping features means investing in the factory competes directly for engineering time against product delivery.

competition with shipping features
Open source video
Source video

Agent failure handling

Benefit

Agents can operate autonomously across loops.

Cost or risk

When agents fail they create unplanned work, adding unpredictable toil on top of planned engineering.

unplanned work when agents fail
Open source video
Source video

Local vs shared agent workflows

Benefit

Local execution is simple and fast for individuals.

Cost or risk

Local agent logs are poorly legible, blocking collective observability and the meta loop.

poor legibility of local agent logs
Open source video
Source video

Open questions

4

How do you measure autonomy and automation separately enough to drive the sequencing rule (autonomy before automation) without conflating them with quality?

Why unresolved

The summary lists proxies (manual takeovers, human PR comments, auto-initiated PRs, quality signals) but gives no method to attribute them to autonomy vs automation.

Research direction

Define intervention-rate and self-initiation-rate metrics and validate whether they move independently across staged rollouts.

Source video

What makes a meta loop actually converge rather than chase noisy quality signals?

Why unresolved

The summary states the meta loop feeds improvements back into inner and outer loops but does not describe stabilization, filtering, or overfitting risks.

Research direction

Prototype feedback loops with dampening and signal filtering; test whether loop changes persist or oscillate.

Source video

Which playbooks are worth codifying into agent skills, and where should the boundary be?

Why unresolved

Explicitly posed as a question to investigate, with no criteria given.

Research direction

Catalog recurring operations, codify a subset as skills, and measure whether reuse reduces human correction.

Source video

What is the minimum control plane needed for a small team to get the meta loop's benefits?

Why unresolved

The talk describes Tessl's platform approach as incremental adoption but does not specify a minimal viable set of control-plane components.

Research direction

Build reduced control planes (issue tracker + PR integration only) and compare legibility/improvement outcomes.

Source video

Key claims

7
opinionVerification needed

A software factory is an agentic system where the final product is built entirely by agents.

Evidence

A software factory is an agentic system where the final product is built entirely by agents.

Question

Can a production software system be maintained end-to-end by agents with humans only improving automation, autonomy, and quality?

Source video
opinionVerification needed

Harness engineering is the direct creation, monitoring, and improvement of a project's loops.

Evidence

Harness engineering is the direct creation, monitoring, and improvement of a project's loops.

Question

Is loop engineering a distinct discipline from conventional CI/CD and DevOps automation?

Source video
causalVerification needed

The inner loop drives automation, the outer loop drives autonomy, and the meta loop drives quality.

Evidence

There are three core loops: inner loop (drives automation), outer loop (drives autonomy), and meta loop (drives quality).

Question

Do these three loops map causally to the three metrics, or is the mapping a conceptual convenience?

Source video
comparativeVerification needed

Teams should improve autonomy first, then automation, while keeping quality constant.

Evidence

Teams must improve autonomy first, then automation, while keeping quality constant and ultimately raising it.

Question

Does raising automation before autonomy actually degrade outcomes in agentic pipelines?

Source video
opinionVerification needed

The agentic stack consists of three layers: Control Plane, Agent IT, and Improvement Loops.

Evidence

The agentic stack consists of three layers: Control Plane, Agent IT, and Improvement Loops.

Question

Is this three-layer decomposition sufficient across teams of different sizes and stacks?

Source video
causalVerification needed

Local agent logs have poor legibility, and agent workflows and knowledge should be surfaced on public, accessible surfaces.

Evidence

Teams need upfront strategies to surface local agent workflows and knowledge onto public, accessible surfaces.

Question

Does moving execution to shared environments measurably improve loop improvement rates?

Source video
causalVerification needed

Automating maintenance tasks reduces unplanned engineering toil.

Evidence

Automating maintenance tasks reduces unplanned engineering toil.

Question

How much unplanned toil do agent-driven repo maintenance scans eliminate versus introduce?

Source video

Connections

4