Sequoia Capital · Published 2026-08-13

When to Build Your Own Agent Harness | Harrison Chase, LangChain

privateReview: unreviewedConfidence: high
agent harnessevalsObservabilitytracesdata flywheelagent-harnesscontext-engineeringdata-flywheelllm-judgesmemory
Open on YouTube ↗

Summary

Overview

  • Speaker: Harrison Chase
  • Channel: Sequoia Capital
  • Main topic: Agent harnesses, evals, observability, and continuous improvement loops for LLM agents
  • Purpose: To provide technical guidance on building, evaluating, and improving agent harnesses and continuous learning systems for AI applications. Harrison Chase, co-founder and CEO of LangChain, discusses the core anatomy of LLM agents—breaking them down into models, context, and harnesses. He covers when to build custom agent harnesses versus using off-the-shelf options, the importance of evals and observability (benchmarking with tools like Harbor), and the creation of a continuous learning data flywheel using tools like LangSmith Engine.

Topic Map

Parts of an Agent

  • Explanation: An agent consists of three core components: a model, context, and a harness that orchestrates them.
  • Key claims:
    • To own your intelligence, you must own all three parts: model, context, and harness.
    • Model ownership involves open weights and the ability to switch models.
    • Context ownership includes episodic, semantic, and procedural memory.
  • Examples:
    • Switching models to avoid lock-in and use the best model available.
    • Using RAG and chat history as semantic and episodic context.
  • Terminology:
    • harness
    • model
    • context
    • open weights
    • RAG
  • Why it matters: Understanding the components of an agent is essential for owning and customizing AI intelligence effectively.

How to Own Your Harness

  • Explanation: A harness gets the model the right context at the right time for a given task, orchestrating fixed and dynamic context.
  • Key claims:
    • Agents need environments, data connections, long-run context management, parallelization, human-in-the-loop, and self-improvement.
    • The base agent loop is an LLM calling tools in a loop until a task completes.
    • Customizing harnesses can be done via middleware (hooks and plugins) or custom cognitive architectures.
  • Examples:
    • LangChain as a base minimal harness vs. DeepAgents as a customizable harness.
    • Adding summarization, sandbox environments, and sub-agents as middleware.
  • Terminology:
    • middleware
    • hooks
    • plugins
    • sandboxes
    • sub-agents
    • context offloading
  • Why it matters: Harness engineering allows developers to adapt general models into specialized, reliable agents for domain-specific tasks.

Evals and Observability

  • Explanation: Every mission-critical agent needs benchmarks to define what 'good' looks like and catch regressions.
  • Key claims:
    • Private evals define quality inside an organization.
    • Harbor is becoming an industry-standard open source eval runner.
    • Observability into LLM inputs, outputs, and message trajectories is critical for debugging agent failures.
  • Examples:
    • Frontier-bench and Terminal-bench as coding agent benchmarks.
    • LangSmith trace views showing tool calls, latencies, and token costs.
  • Terminology:
    • evals
    • observability
    • Harbor
    • frontier-bench
    • trajectories
    • traces
  • Why it matters: Without proper evals and observability, debugging agents and preventing regressions is nearly impossible.

Data Flywheel and Continuous Improvement

  • Explanation: Building a continuous learning loop requires collecting traces, mining data, curating evals, and running experiments.
  • Key claims:
    • A continuous learning loop acts as a hill-climbing machine to compound AI investment value.
    • Feedback is crucial for the flywheel, coming from UX design and synthetic feedback (online evals).
    • LangSmith Engine automates trace data analysis to create issue boards and suggest prompt or harness fixes.
  • Examples:
    • Using fine-tuned SLMs as judges to reduce evaluation costs.
    • LangSmith Engine automatically grouping agent failure traces and proposing prompt modifications.
  • Terminology:
    • data flywheel
    • synthetic feedback
    • online evals
    • SLM judges
    • LangSmith Engine
    • issue board
  • Why it matters: Automating the feedback and iteration loop enables agents to improve continuously over time with minimal manual overhead.

Key Points

Anatomy of an Agent

  • Explanation: Agents are composed of three parts: a model, context, and a harness.
  • Evidence: Diagrams showing harness wrapping model and context.
  • Practical implication: Developers should control all three layers to avoid vendor lock-in and maximize agent performance.

When to Build vs. Use Off-the-Shelf Harnesses

  • Explanation: Off-the-shelf harnesses work well for in-distribution tasks, while custom harnesses are needed for out-of-distribution tasks.
  • Evidence: Discussion of Codex/Claude Code vs. specialized domain harnesses like legal AI.
  • Practical implication: Start with general off-the-shelf harnesses and customize or build proprietary ones as domain specificity increases.

The Importance of Observability

  • Explanation: Most agent failures stem from poor context fed to the LLM rather than model incompetence.
  • Evidence: Analysis of LLM call errors and trace trajectories.
  • Practical implication: Implement detailed trace logging and trajectory inspection to effectively debug agent behaviors.

Frameworks, Models & Processes

Recipe for Continuously Improving Agents

  • How it works: A four-step iterative process to upgrade agent performance using production data.
  • Components:
    • Build a v1 agent and start running it
    • Collect lots of traces
    • Curate the trace data via data mining
    • Run experiments on your data
  • When to use: When deploying production LLM agents that need to improve over time based on real usage.

Examples & Case Studies

Comparing Codex, Claude Code, and DeepAgents on coding benchmarks using Harbor.

  • Illustrates: How different agent harnesses and models perform across standardized tasks.
  • Lesson: Benchmarking allows developers to quantitatively evaluate trade-offs between different models and harness architectures.

Actionable Takeaways

  • Immediate:
    • Define private evals and benchmarks for your critical agents.
    • Ensure robust tracing and observability are set up from day one.
    • Start with simple general harnesses before building custom cognitive architectures.
  • Strategic:
    • Own your model, context, and harness to retain organizational memory and institutional knowledge.
    • Build a data flywheel using trace mining and automated feedback loops.
  • Questions to investigate:
    • When does an out-of-distribution task justify building a fully custom cognitive architecture?
    • How can synthetic evaluation costs be minimized at scale?

Claims Worth Verifying

  • Harbor is becoming an industry-standard open source eval runner. (industry trend)
  • Most agent errors are caused by poor context rather than poor model capability. (technical assertion)

Notable Quotes

"To own your intelligence, you have to own all three parts." "Create your private evals, because evals define what good looks like inside the organization."

Compressed Summary

  • Agents consist of models, context, and harnesses.
  • Harness engineering involves middleware and custom cognitive architectures.
  • Evals and benchmarking via tools like Harbor catch regressions.
  • Observability into trace trajectories is critical for debugging agent failures.
  • LangSmith Engine powers a data flywheel for continuous agent improvement.
  • Keywords: agent harness, evals, observability, traces, data flywheel
  • Core insight: Owning your agent intelligence requires controlling models, context, and harnesses while powering continuous improvement through robust evals, observability, and data flywheels.

Core insights

7
Mental Modelmedium noveltystrong evidence

The harness — not the model — is the engineering surface: it is defined as the component that gets the model the right context at the right time, orchestrating fixed versus dynamic context. This reframes agent quality as a context-assembly and orchestration problem, with the model as a swappable dependency.

Why it matters

It tells engineers where to invest effort and where to enforce abstraction boundaries: context assembly, orchestration, and loop control rather than model selection.

Generalization

Any LLM application can be decomposed into model + context + orchestration layer; the orchestration layer is the durable, portable asset and the one that absorbs domain specificity.

A harness gets the model the right context at the right time for a given task, orchestrating fixed and dynamic context.
Open source video
An agent consists of three core components: a model, context, and a harness that orchestrates them.
Open source video
Tradeoffmedium noveltymoderate evidence

Harness customization has a graded ladder: middleware (hooks and plugins) for compositional extension, versus a fully custom cognitive architecture. The choice is framed by task distribution — off-the-shelf harnesses suffice in-distribution; custom harnesses are needed out-of-distribution.

Why it matters

Provides a decision rule for when to extend an existing harness versus pay the full cost of building a proprietary cognitive architecture.

Generalization

Applies to any platform-vs-bespoke decision: extend via plug-in points until the workload leaves the platform's distribution, then fork the architecture.

Customizing harnesses can be done via middleware (hooks and plugins) or custom cognitive architectures.
Open source video
Off-the-shelf harnesses work well for in-distribution tasks, while custom harnesses are needed for out-of-distribution tasks.
Open source video
Failure Modemedium noveltyweak evidence

Failure attribution is claimed to sit in the context pipeline, not model capability: most agent errors reportedly come from poor context fed to the LLM, and the diagnostic instrument is the trace trajectory (tool calls, latencies, token costs).

Why it matters

Redirects debugging and reliability budget from model upgrades to context assembly, retrieval, and trajectory inspection — and justifies trace-level observability as day-one infrastructure.

Generalization

For any deployed LLM system, instrument the full input context and message trajectory, since the failure signal lives upstream of the model.

Most agent failures stem from poor context fed to the LLM rather than model incompetence.
Open source video
Observability into LLM inputs, outputs, and message trajectories is critical for debugging agent failures.
Open source video
Practicemedium noveltystrong evidence

The improvement loop is specified as a concrete four-step cycle over production traces: ship a v1 agent, collect lots of traces, curate via data mining, then run experiments on that curated data — a hill-climbing machine rather than ad hoc prompt tweaking.

Why it matters

Turns continuous improvement into an operational pipeline where production traces are the raw material and experiments are the unit of progress.

Generalization

Any production AI system can be modeled as trace collection → data mining/curation → offline experiment → deploy, with the trace store as the compounding asset.

A continuous learning loop acts as a hill-climbing machine to compound AI investment value.
Open source video
Collect lots of traces
Open source video
Mechanismmedium noveltymoderate evidence

Evaluation cost is decoupled from evaluation volume by using fine-tuned small language models as judges and by synthesizing feedback (online evals) instead of relying only on human labeling or UX-derived signals.

Why it matters

Makes high-frequency online evaluation economically viable, which is a precondition for running the flywheel continuously rather than periodically.

Generalization

Judge models can be distilled from larger evaluators and run inline, shifting eval economics from per-sample human cost to fixed training plus cheap inference.

Using fine-tuned SLMs as judges to reduce evaluation costs.
Open source video
Feedback is crucial for the flywheel, coming from UX design and synthetic feedback (online evals).
Open source video
Predictionhigh noveltymoderate evidence

The feedback loop is being automated at the harness level: LangSmith Engine groups failure traces into an issue board and proposes prompt or harness fixes, not merely surfacing raw traces for humans.

Why it matters

If failure triage and fix proposal are automated, the bottleneck moves from observing failures to validating and reviewing machine-proposed changes.

Generalization

Observability systems trend from dashboards toward closed-loop agents that mine their own traces and propose architectural modifications.

LangSmith Engine automates trace data analysis to create issue boards and suggest prompt or harness fixes.
Open source video
LangSmith Engine automatically grouping agent failure traces and proposing prompt modifications.
Open source video
Architecturemedium noveltymoderate evidence

Context ownership is taxonomized into episodic, semantic, and procedural memory, alongside harness ownership (orchestration) and model ownership (open weights plus model switchability) — treating memory types as owned, portable assets rather than framework internals.

Why it matters

Makes explicit that switching models is only safe if context and harness stay in-house; otherwise institutional knowledge is trapped inside a vendor's harness.

Generalization

Vendor portability requires owning the memory substrate and orchestration layer, not just retaining the right to swap API endpoints.

Context ownership includes episodic, semantic, and procedural memory.
Open source video
Model ownership involves open weights and the ability to switch models.
Open source video
To own your intelligence, you have to own all three parts.
Open source video

Deep dives

5

Failure attribution between context assembly and model capability

Research question

What fraction of traced agent failures are attributable to context-assembly defects versus model capability, and can trajectory-level traces attribute them reliably across multiple deployments?

Why

It decides where the reliability budget goes: context plumbing and retrieval, or model upgrades. If context defects dominate, trace-level observability becomes day-one runtime infrastructure instead of monitoring.

Observability into LLM inputs, outputs, and message trajectories is critical for debugging agent failures.
Open source video
LangSmith trace views showing tool calls, latencies, and token costs.
Open source video
Source video

Ownership benchmark: proprietary harness versus frontier off-the-shelf harness

Research question

Under what conditions does owning model, context, and harness outperform adopting a frontier off-the-shelf harness, and at what cost in quality, latency, and token spend?

Why

The ownership argument is presented as a principle, not an empirical result. Without a counterfactual benchmark, the build-your-own decision is unmade and organizations will fork architectures they did not need to fork.

To own your intelligence, you must own all three parts: model, context, and harness.
Open source video
LangChain as a base minimal harness vs. DeepAgents as a customizable harness.
Open source video
Source video

Crossover threshold from middleware extension to custom cognitive architecture

Research question

At what measured task-distribution distance does a middleware-extended off-the-shelf harness degrade relative to a custom cognitive architecture, and what is the cost of the later migration?

Why

Without a criterion or cost model for the crossover, teams either over-build custom loops early or hit a middleware ceiling late and pay migration twice.

Adding summarization, sandbox environments, and sub-agents as middleware.
Open source video
Agents need environments, data connections, long-run context management, parallelization, human-in-the-loop, and self-improvement.
Open source video
Source video

Fidelity–cost frontier of distilled SLM judges on agent trajectories

Research question

Where do fine-tuned small-model judges diverge from frontier-model judges on agent trajectory evaluation, and at what evaluation volume does the cost advantage dominate the measured fidelity gap?

Why

Online evals only work continuously if judging is cheap; if the cheap judge mis-scores exactly the out-of-distribution cases the flywheel must catch, the cost win is a silent reliability regression.

Private evals define quality inside an organization.
Open source video
Every mission-critical agent needs benchmarks to define what 'good' looks like and catch regressions.
Open source video
Source video

Acceptance rate of machine-proposed prompt and harness fixes

Research question

What fraction of automatically grouped failure issues and machine-proposed prompt/harness fixes survive evaluation without degrading existing evals?

Why

If triage and fix proposal are automated, the bottleneck moves to validating machine-proposed changes; without a survival rate, auto-proposed modifications flow toward production unverified.

Every mission-critical agent needs benchmarks to define what 'good' looks like and catch regressions.
Open source video
LangSmith trace views showing tool calls, latencies, and token costs.
Open source video
Source video

Article ideas

4

The Harness Is the Product — The Model Is a Swappable Dependency

Agent quality is decided in context assembly and orchestration, not model selection; teams that treat the model as the architecture will keep buying upgrades to fix defects that live upstream of the model.

Angle

Reframe agent engineering from model choice to context-getting: the harness is the layer that owns fixed versus dynamic context, environments, loop control, and extension points.

Source video

You Don't Own Your Intelligence If You Only Own the Model

Model switchability is a hollow guarantee unless the memory substrate and harness are in-house; ownership is a three-layer property, and partial ownership is vendor lock-in with extra steps.

Angle

Walk the lock-in failure mode: swapping an API endpoint while episodic, semantic, and procedural memory remain framework internals means the institutional knowledge stays with the vendor.

Source video

Middleware Until It Breaks: A Migration Rule for Agent Architecture

Extension points are the correct default and a full architectural fork should be triggered by a measured shift in task distribution, not by ambition, so the engineering question is the crossover criterion rather than the aesthetic of a custom loop.

Angle

Give the decision rule with its missing measurement: define in-distribution versus out-of-distribution signals, name the middleware ceiling symptoms, and price the late migration.

Source video

Traces Are the Asset, Dashboards Are the Cost

Observability only compounds when traces feed a mine, curate, experiment, deploy loop; standalone trace viewers are expensive logging that documents failures without reducing them.

Angle

Argue the trace store is the durable, reusable artifact and eval runners plus issue boards are the machinery that turns it into hill-climbing progress.

Source video

Project ideas

4

Crossover Bench

beyond-evals

A middleware-extended off-the-shelf harness matches a custom cognitive architecture on in-distribution tool-use tasks and degrades measurably past a reproducible out-of-distribution perturbation threshold.

Proof of concept

Implement the same domain task suite twice — a base harness with middleware (summarization, sandbox, sub-agents) and a custom loop — and sweep task perturbations from in-distribution to out-of-distribution on a shared eval runner.

Measurement

Quality per engineering-hour and per token as a function of distribution distance, plus the perturbation level at which the middleware harness falls below the custom architecture.

Source video

Context Autopsy

gatehouse

A large majority of traced agent failures can be attributed automatically to context-assembly defects rather than model capability, and resolving the attributed context defect eliminates the failure without a model change.

Proof of concept

Instrument a deployed agent to capture full LLM inputs, message trajectories, tool calls, and latencies; build a classifier that labels each failure as context-defect or model-capability; apply targeted context fixes and re-run the same traces.

Measurement

Share of failures attributed to context defects, resolution rate after context-only fixes, and regression rate on existing private evals.

Source video

Judge Divergence Ledger

beyond-evals

Distilled SLM judges agree with frontier-model judges on in-distribution traces but diverge systematically on out-of-distribution traces — exactly the region the improvement flywheel must detect.

Proof of concept

Fine-tune an SLM judge on agent trajectory labels and evaluate it against a frontier judge over held-out traces bucketed by distribution distance and failure type.

Measurement

Agreement and directional error by bucket, and the trace volume at which the SLM cost advantage exceeds the measured fidelity gap.

Source video

Fix Proposal Survival Rate

new

Most machine-proposed prompt or harness fixes derived from grouped failure traces do not survive evaluation without degrading existing evals, and the survival rate is predictable from issue-cluster features.

Proof of concept

Build a pipeline that clusters failure traces into an issue board, proposes prompt or harness modifications, applies each proposal to a shadow harness, and runs the full eval suite plus private evals before acceptance.

Measurement

Fraction of proposals passing all evals, regression rate per accepted proposal, and correlation between survival and issue-cluster features such as cluster size.

Source video

Architectural implications

4

The harness becomes an explicit, named layer boundary between model and context rather than implicit glue code inside an application.

Before

Prompt strings, retrieval calls, and tool loops scattered through application code with the model treated as the architecture.

After

A harness component owns context assembly, loop control, environments, and extension points, with the model as a replaceable dependency.

Consequence

Enables model switching and middleware-based extension without rewriting domain logic, but requires investing in context plumbing up front.

Source video

Evaluation becomes transportable infrastructure via a standard runner (Harbor), making harness/model comparisons a routine benchmark operation.

Before

Per-team, bespoke eval scripts that are not comparable across harnesses or models.

After

Shared open-source eval runners where custom and off-the-shelf harnesses are measured on the same tasks.

Consequence

Harness choices become empirically comparable and regressions detectable, but pushes teams toward standard task suites that may not reflect private domain quality.

Source video

Debugging interfaces shift from model-output inspection toward trajectory-level context inspection.

Before

Tuning prompts and upgrading models in response to bad answers.

After

Inspecting full LLM inputs, outputs, and message trajectories to locate context defects.

Consequence

Observability tooling (trace views with tool calls, latencies, token costs) becomes a first-class runtime dependency rather than a monitoring afterthought.

Source video

The improvement loop is closed by automation that reads traces and emits proposed changes to prompts and harnesses.

Before

Humans read traces, cluster failures by hand, and write fixes.

After

Systems mine traces, group failures into an issue board, and propose prompt or harness modifications for review.

Consequence

Human effort concentrates on validating and reviewing proposed fixes, introducing a new review bottleneck and a need to measure auto-proposed changes.

Source video

Tradeoffs and failure modes

4

Middleware extension versus custom cognitive architecture

Benefit

Middleware (hooks and plugins) delivers summarization, sandboxes, and sub-agents without rebuilding the loop, keeping harness code minimal.

Cost or risk

Extension points can hit a ceiling on out-of-distribution tasks, forcing a later, more expensive migration to a fully custom architecture.

Customizing harnesses can be done via middleware (hooks and plugins) or custom cognitive architectures.
Open source video
Source video

Open weights and model switchability as a lock-in defense

Benefit

Owning the model layer avoids vendor lock-in and lets teams use the best available model per task.

Cost or risk

Switching models is only safe if context and harness are also owned; otherwise institutional memory and orchestration logic stay trapped with the vendor.

To own your intelligence, you have to own all three parts.
Open source video
Source video

Automated trace mining and fix proposal

Benefit

Grouping failure traces and proposing prompt or harness fixes removes manual triage from the improvement loop.

Cost or risk

Machine-proposed modifications arrive without stated validation, creating risk of unverified changes flowing toward production and a new review burden.

LangSmith Engine automatically grouping agent failure traces and proposing prompt modifications.
Open source video
Source video

Synthetic feedback and SLM judges

Benefit

Fine-tuned small-model judges reduce evaluation costs and enable online evals at production frequency.

Cost or risk

Judge quality is not established in the summary; a distilled judge can systematically mis-score exactly the out-of-distribution cases the flywheel is meant to catch.

Using fine-tuned SLMs as judges to reduce evaluation costs.
Open source video
Source video

Open questions

4

What is the threshold at which an out-of-distribution task justifies building a fully custom cognitive architecture instead of extending middleware?

Why unresolved

The summary asserts the in-distribution/out-of-distribution distinction but gives no criteria, measurement, or cost model for the crossover point.

Research direction

Run the same domain tasks through an off-the-shelf harness with middleware versus a custom architecture and measure quality-per-engineering-hour against a benchmark runner.

Source video

How can synthetic evaluation cost be minimized at scale without losing judge fidelity?

Why unresolved

SLM judges are proposed as a cost reduction, but no accuracy or calibration result is reported.

Research direction

Compare fine-tuned SLM judges against frontier-model judges on held-out agent trajectories and characterize where the distilled judge diverges.

Source video

Can automatically grouped failure traces and machine-proposed fixes be trusted, and how would that be measured?

Why unresolved

The summary describes the automation but provides no acceptance rate, regression data, or review protocol.

Research direction

Instrument an issue-board pipeline and track the fraction of proposed prompt/harness changes that survive evaluation without degrading existing evals.

Source video

Does owning model, context, and harness actually produce better outcomes than adopting a frontier off-the-shelf harness, and under what conditions?

Why unresolved

The ownership argument is presented as a principle ('to own your intelligence'), not an empirical result, and the counterfactual cost is unquantified.

Research direction

Benchmark proprietary harnesses against general ones (Codex, Claude Code, DeepAgents) on standardized coding tasks using a shared eval runner and compare quality, latency, and token cost.

Source video

Key claims

8
opinionVerification not requested

To own your intelligence, you have to own all three parts: model, context, and harness.

Evidence

To own your intelligence, you have to own all three parts.

Source video
comparativeVerification needed

Off-the-shelf harnesses work well for in-distribution tasks, while custom harnesses are needed for out-of-distribution tasks.

Evidence

Off-the-shelf harnesses work well for in-distribution tasks, while custom harnesses are needed for out-of-distribution tasks.

Question

On which task families does middleware-extended off-the-shelf harness performance measurably degrade relative to a custom cognitive architecture?

Source video
causalVerification needed

Most agent failures stem from poor context fed to the LLM rather than model incompetence.

Evidence

Most agent failures stem from poor context fed to the LLM rather than model incompetence.

Question

What fraction of traced agent failures are attributable to context assembly defects versus model capability, across multiple production deployments?

Source video
predictionVerification needed

Harbor is becoming an industry-standard open source eval runner.

Evidence

Harbor is becoming an industry-standard open source eval runner.

Question

What is the actual adoption of Harbor as an eval runner relative to alternative harness benchmark runners?

Source video
factualVerification not requested

The base agent loop is an LLM calling tools in a loop until a task completes.

Evidence

The base agent loop is an LLM calling tools in a loop until a task completes.

Source video
causalVerification needed

A continuous learning loop acts as a hill-climbing machine to compound AI investment value.

Evidence

A continuous learning loop acts as a hill-climbing machine to compound AI investment value.

Question

Does a trace-mining and experiment loop produce compounding, non-saturating quality gains over repeated iterations?

Source video
factualVerification needed

LangSmith Engine automates trace data analysis to create issue boards and suggest prompt or harness fixes.

Evidence

LangSmith Engine automates trace data analysis to create issue boards and suggest prompt or harness fixes.

Question

How accurate are the automatically generated issue groupings and suggested fixes when evaluated against human triage?

Source video
comparativeVerification needed

Using fine-tuned SLMs as judges reduces evaluation costs.

Evidence

Using fine-tuned SLMs as judges to reduce evaluation costs.

Question

What is the quality gap between fine-tuned SLM judges and frontier-model judges on agent trace evaluation, and at what volume does the cost advantage dominate?

Source video

Connections

5