AI Engineer · Published 2026-08-12

Improving Agents is a Data Mining Problem — Vivek Trivedy, LangChain

privateReview: unreviewedConfidence: high
agentstracesObservabilitycontinual-learningfine-tuningharness-engineeringllm-evaluationcost-optimization
Open on YouTube ↗

Summary

Overview

  • Speaker: Vivek Trivedy
  • Channel: AI Engineer
  • Main topic: Continual learning and agent improvement through data mining and tracing
  • Purpose: To explain how data mining over agent traces, observability, and continual learning drive agent improvement and cost efficiency. Vivek Trivedy from LangChain discusses how improving AI agents over time is fundamentally a data mining problem. He outlines a recipe for building, running, and improving agents by collecting traces, data mining, and running experiments. He explores the challenges of reading traces at scale, leveraging open-source models for cost-efficient trace judging, and drawing parallels between machine learning and agent harness engineering.

Topic Map

The Recipe for Continuously Improving Agents

  • Explanation: A four-step recipe for building and improving agents: ship a v1 agent, collect traces, data mine the traces, and run experiments.
  • Key claims:
    • No one knows what an agent will do until they run it.
    • Agents trade determinism for autonomy.
    • Every continual learning company is an observability company.
  • Examples:
    • Running an agent, collecting traces from tool calls, output messages, API calls, and CLI usage.
  • Terminology:
    • Traces
    • Observability
    • Continual Learning
    • Data Mining
  • Why it matters: Provides a systematic framework for moving from initial agent deployment to data-driven iterative improvement.

Why Reading Traces at Scale is Hard

  • Explanation: Discusses the challenges of handling massive volumes of trace data generated by long-horizon agents.
  • Key claims:
    • Reading traces at scale is super expensive.
    • Traces don't fit in context windows.
    • Modern agent runs can be millions of tokens long.
  • Examples:
    • Coding agents like Claude Code, Codex, or deep agents producing massive trace files.
  • Terminology:
    • Trace Data
    • Context Window
    • Token Cost
    • Long-Horizon Agents
  • Why it matters: Highlights why naive approaches to reviewing all agent traces fail and necessitates specialized tools like LangSmith Engine.

Open Models for Cost-Efficient Trace Judges

  • Explanation: Explores using open-source models (like DeepSeek or Llama models) instead of frontier models to judge traces at a lower cost.
  • Key claims:
    • Open models have hidden inflection points in intelligence.
    • Fine-tuning open models can match or exceed frontier performance on specific tasks at a fraction of the cost.
    • Moving from token costs to hardware cluster costs for high-inference workloads.
  • Examples:
    • Using Opus vs. open models on the Harvey legal benchmark.
  • Terminology:
    • Open Models
    • Fine-Tuning
    • Harness Engineering
    • Trace Judging
  • Why it matters: Demonstrates how teams can dramatically reduce the cost of trace evaluation and agent supervision.

Fitting Systems to Data: The Model-Harness-Task Fit

  • Explanation: Draws an analogy between classical machine learning (Scikit-learn estimators) and modern agent engineering.
  • Key claims:
    • Agent engineering parallels classical machine learning fit methods.
    • The process involves Model + Harness + Task fit.
    • Auto-research and auto-feedback loops are central to agent optimization.
  • Examples:
    • Scikit-learn fit method applied to models, harnesses, and tasks.
  • Terminology:
    • Model-Harness-Task Fit
    • Auto-Research
    • Fit Functions
    • Distillation
  • Why it matters: Bridges traditional ML engineering paradigms with modern LLM agent development.

Key Points

Observability and Continual Learning are tightly coupled

  • Explanation: You cannot have continual learning without traces, and tracing provides the raw data needed for agent adaptation.
  • Evidence: Agents produce trace data during environment execution, which feeds human or automated learning loops.
  • Practical implication: Teams must prioritize robust tracing and logging infrastructure from day one of agent deployment.

Harness Engineering vs. Fine-Tuning

  • Explanation: Harness engineering provides fast feedback loops, while fine-tuning helps scale intelligence once harness limits are reached.
  • Evidence: Harness engineering gives feedback in minutes; fine-tuning optimizes base models for specific vertical tasks.
  • Practical implication: Use a sandwich approach: start with harness engineering, fine-tune to break ceilings, and continue harness engineering.

Frameworks, Models & Processes

Recipe for Continuously Improving Agents

  • How it works: A linear pipeline moving from deployment to data extraction and experimentation.
  • Components:
    • Build a v1 agent and start running it
    • Collect lots of traces
    • Curate the trace data (data mining)
    • Run experiments on your data
  • When to use: When deploying autonomous agents in production that need iterative improvement.

Model-Harness-Task Fit

  • How it works: Adapting learning systems to data similar to classical ML estimator fitting.
  • Components:
    • Model
    • Harness
    • Task
    • Fit Function
  • When to use: When optimizing agent performance across specific enterprise domains.

Examples & Case Studies

Running agents on terminal bench tasks and analyzing traces.

  • Illustrates: How terminal bench outputs binary pass/fail results, requiring densified feedback signals from traces to guide improvement.
  • Lesson: Traces provide the granular behavioral data needed to understand why an agent succeeded or failed.

Actionable Takeaways

  • Immediate:
    • Turn on tracing for your agents immediately to capture behavioral data.
    • Use open-source models for trace judging and evaluation to reduce costs.
  • Strategic:
    • Treat agent improvement as a data mining and continual learning pipeline.
    • Balance harness engineering and model fine-tuning based on performance ceilings and cost.
  • Questions to investigate:
    • How do we scale sleep-time compute and memory management for long-horizon agents?
    • How can we automate the densification of feedback signals from agent traces?

Claims Worth Verifying

  • Modern agent runs will soon exceed all human lifetime data production. (projection)
  • Fine-tuning open models can match frontier model trace judging capabilities at 1-2 orders of magnitude lower cost. (benchmark claim)

Notable Quotes

"Improving Agents is a Data Mining Problem." (at 0:25) "Every Continual Learning Company is an Observability Company." (at 3:42) "No one knows what their agent will do until they run it. Agents trade determinism for autonomy." (at 5:26)

Compressed Summary

  • Agent improvement is fundamentally a data mining problem over trace logs.
  • Traces at scale present cost and context window challenges.
  • Open models offer cost-efficient alternatives for trace judging and fine-tuning.
  • Continual learning requires integrating observational trace data back into agent state and harnesses.
  • Keywords: agents, traces, observability, continual-learning, fine-tuning
  • Core insight: Improving autonomous AI agents at scale requires treating trace data collection, mining, and continual learning as a unified engineering pipeline.

Core insights

7
Architecturemedium noveltystrong evidence

Agent runs are now large enough that trace data structurally cannot be reviewed inside a model context window — modern long-horizon runs can reach millions of tokens — so trace analysis must be done as an out-of-context data-mining pipeline over external storage plus a smaller judging model, not by prompting a frontier model with the trace.

Why it matters

Determines where trace analysis belongs: it is a data-processing/batch-mining workload with its own retrieval, filtering, and sampling stages, not an inference call. Anyone planning to 'just ask GPT to read the logs' will hit a hard scaling wall.

Generalization

Any observability workload over high-volume execution logs will outgrow the context window; the analysis layer must be decoupled from the model that produced the data.

Reading traces at scale is super expensive.
Open source video
Traces don't fit in context windows.
Open source video
Modern agent runs can be millions of tokens long.
Open source video
Tradeoffhigh noveltymoderate evidence

Cost-efficient trace judging comes from fine-tuning open models rather than using frontier models, and at high inference volume the cost model shifts from per-token API pricing to owning/renting hardware clusters — a different kind of engineering commitment.

Why it matters

Changes the build-vs-buy and cost-structure decision for evaluation infrastructure: teams trade pay-per-token elasticity for fixed cluster cost, fine-tuning effort, and ops burden, which is only rational above an inference-volume threshold.

Generalization

As evaluation/judging traffic scales past a threshold, dedicated fine-tuned small models on owned compute beat front-end API calls on unit economics.

Fine-tuning open models can match or exceed frontier performance on specific tasks at a fraction of the cost.
Open source video
Moving from token costs to hardware cluster costs for high-inference workloads.
Open source video
Open models have hidden inflection points in intelligence.
Open source video
Practicehigh noveltymoderate evidence

Agent improvement should be staged as a sandwich: harness engineering (fast, minute-scale feedback) first, then fine-tuning to break performance ceilings, then back to harness engineering.

Why it matters

Gives a concrete ordering heuristic for where to spend engineering effort instead of defaulting to fine-tuning or prompt-hacking; it explicitly frames fine-tuning as a ceiling-breaking move rather than a starting point.

Generalization

Iterate in the loop with the fastest feedback signal until it saturates, then change the slower, more expensive layer, then resume fast iteration.

Harness engineering gives feedback in minutes; fine-tuning optimizes base models for specific vertical tasks.
Open source video
Use a sandwich approach: start with harness engineering, fine-tune to break ceilings, and continue harness engineering.
Open source video
Mechanismhigh noveltymoderate evidence

Binary pass/fail agent benchmarks (e.g. terminal-bench-style tasks) yield feedback too sparse to guide improvement; trace data is what supplies the granular behavioral signal explaining why a run succeeded or failed — i.e., traces are used to densify the reward/feedback signal.

Why it matters

Reframes tracing as a training-signal generation problem, not just debugging. Teams can build outcome-plus-trace evaluation loops where the pass/fail label is enriched by mined behavioral evidence.

Generalization

Sparse outcome signals plus rich execution traces can be converted into denser supervision; the trace is the dense label source.

How terminal bench outputs binary pass/fail results, requiring densified feedback signals from traces to guide improvement.
Open source video
Traces provide the granular behavioral data needed to understand why an agent succeeded or failed.
Open source video
Mental Modelmedium noveltystrong evidence

Continual learning for agents is not separable from observability: the trace stream is the raw training/adaptation data, so tracing and logging infrastructure is a hard prerequisite for any improvement loop, not a nice-to-have monitoring add-on.

Why it matters

Reorders investment priorities: instrument tracing on day one of deployment because without the data there is no later learning path to build.

Generalization

Any system that claims to learn continuously from its own operation must first own a durable record of that operation.

Every continual learning company is an observability company.
Open source video
You cannot have continual learning without traces, and tracing provides the raw data needed for agent adaptation.
Open source video
Teams must prioritize robust tracing and logging infrastructure from day one of agent deployment.
Open source video
Mental Modelmedium noveltyweak evidence

Agent engineering can be modeled as a Model + Harness + Task fit problem analogous to classical ML estimator fitting, with auto-research and auto-feedback loops driving the optimization.

Why it matters

Suggests a reusable abstraction for tooling: separate the model component, the harness component, and the task specification, and build fit/evaluation procedures against each independently, which clarifies what an orchestration framework should own.

Generalization

Framing a system as (components x data x fit procedure) makes the improvement loop explicit and automatable, as in classical ML pipelines.

Agent engineering parallels classical machine learning fit methods.
Open source video
The process involves Model + Harness + Task fit.
Open source video
Auto-research and auto-feedback loops are central to agent optimization.
Open source video
Mental Modelmedium noveltymoderate evidence

Autonomy is purchased with lost determinism: no one can predict what an agent will do until it is actually run, so empirical trace collection is the only reliable source of behavioral knowledge.

Why it matters

Justifies why pre-deployment static analysis, spec review, or prompt inspection cannot substitute for running the agent and mining what it did — it forces an empirical, data-first development loop.

Generalization

As systems move from specified to learned/emergent behavior, empirical observation replaces prediction as the primary design input.

No one knows what their agent will do until they run it. Agents trade determinism for autonomy.
Open source video
No one knows what an agent will do until they run it.
Open source video

Deep dives

5

Automatic densification of trace feedback signals

Research question

Can raw multi-million-token agent runs plus a binary outcome be automatically converted into reliable per-step critiques or rewards, and does that densified signal measurably improve downstream agent performance?

Why

The digest identifies densified feedback as the mechanism that makes trace mining useful for improvement, but offers no method. Without a per-step signal, trace mining remains debugging rather than a training loop, and binary pass/fail benchmarks stay uninformative.

Traces don't fit in context windows.
Open source video
Modern agent runs can be millions of tokens long.
Open source video
Reading traces at scale is super expensive.
Open source video
Source video

The crossover point for fine-tuned open trace judges

Research question

At what judging volume does a fine-tuned open model on owned or rented compute beat frontier-model token billing for trace evaluation, once fine-tuning and serving overhead are included?

Why

The cost claim underpins the entire evaluation-architecture decision, yet no threshold is given. Teams need a measured crossover volume to know whether to buy elasticity or build a fixed-cost judge.

Fine-tuning open models can match or exceed frontier performance on specific tasks at a fraction of the cost.
Open source video
Moving from token costs to hardware cluster costs for high-inference workloads.
Open source video
Open models have hidden inflection points in intelligence.
Open source video
Source video

Detecting harness saturation before switching to fine-tuning

Research question

What instrumented metric over harness iterations predicts that further harness engineering yields diminishing returns and that fine-tuning will unlock the next gain?

Why

The sandwich strategy prescribes a switch point but no detection criterion. Without it, teams either grind on a saturated harness or jump to fine-tuning and lose minutes-scale iteration for no reason.

Harness engineering gives feedback in minutes; fine-tuning optimizes base models for specific vertical tasks.
Open source video
Fine-tuning open models can match or exceed frontier performance on specific tasks at a fraction of the cost.
Open source video
Source video

Out-of-context trace analysis architecture

Research question

What pipeline shape (storage schema, filtering, deduplication, sampling, segmented judging) is required to mine traces whose length exceeds any context window, and what is lost versus full-fidelity reading?

Why

It determines where trace analysis lives in the system: a batch data-mining workload with its own retrieval and sampling stages, decoupled from the agent runtime, rather than an interactive inference call over a pasted log.

Traces don't fit in context windows.
Open source video
Reading traces at scale is super expensive.
Open source video
Modern agent runs can be millions of tokens long.
Open source video
Source video

Transfer of trace-judge fine-tuning across domains

Research question

Does a fine-tuned open model used as a trace judge preserve judgment quality across verticals and rubric changes, or is the gain task-specific and brittle under distribution shift?

Why

The evidence is a single benchmark comparison, so generality is unestablished. If judgment quality degrades across domains, the cost advantage of open judges only applies to narrowly stable workloads, which changes the deployment calculus.

Fine-tuning open models can match or exceed frontier performance on specific tasks at a fraction of the cost.
Open source video
Open models have hidden inflection points in intelligence.
Open source video
Source video

Article ideas

4

Your Agent's Trace Log Is Your Training Set, Not Your Debugger Output

Continual learning for agents is indistinguishable from observability work: teams that treat tracing as an ops add-on added after the agent works have foreclosed their own improvement loop, because the trace stream is the raw input to every later adaptation. Instrumentation and retention-schema decisions are day-one architectural commitments, not monitoring polish.

Angle

Reframe tracing from post-hoc debugging to the data substrate of the improvement loop, and enumerate the day-one design decisions (schema, retention, sampling) that become load-bearing later.

Source video

Stop Pasting Traces Into a Frontier Model

Trace review is a batch data-mining workload, not an inference call. Long-horizon runs reach millions of tokens and cannot fit in a context window, so any workflow that streams a full run into a frontier model silently degrades into ad-hoc sampling while paying frontier prices; the analysis layer must be decoupled from the model that produced the data and given its own filtering, sampling, and judging stages.

Angle

Concrete cost and scaling argument showing where naive in-context review breaks, plus the pipeline stages that replace it.

Source video

Harness First, Fine-Tune to Break the Ceiling

Fine-tuning should be a ceiling-breaking move, not a starting point. Because harness changes return feedback in minutes while fine-tuning optimizes a base model for a vertical, the correct ordering is fast-iteration harness engineering until it saturates, then a fine-tuning cycle, then back to the harness. Teams that invert this pay base-model customization cost while giving up their fastest feedback signal.

Angle

Ordering heuristic for where to spend engineering effort, framed against the default reflex to fine-tune or prompt-hack first.

Source video

Binary Benchmarks Cannot Tell You Why Your Agent Failed

Pass/fail task suites are cheap and unambiguous the way a coin flip is: they carry almost no information about the failure. Traces are the dense label source that turns a binary outcome into actionable supervision, which means evaluation infrastructure should be measured by the density of the feedback it produces, not the number of tasks it scores.

Angle

Reframe benchmarking as a feedback-density problem and argue for outcome-plus-trace evaluation loops over scoreboard-style leaderboards.

Source video

Project ideas

4

Dense-Label Trace Miner

beyond-evals

A segmented trace-mining pipeline that emits per-step critiques from sampled trace segments will produce supervision that improves downstream agent task success more than whole-trace review or binary-outcome-only training, at a lower judging cost.

Proof of concept

Take a set of long-horizon agent runs with binary outcomes, split each trace into segments under a fixed token budget, judge segments with a small model against the outcome label, aggregate into per-step critiques, then compare fine-tuning or harness revisions driven by dense labels versus binary labels only.

Measurement

Delta in held-out task success rate between dense-label and binary-label improvement loops, plus judging cost per run and agreement between segment critiques and the ground-truth outcome.

Source video

Trace-Judge Cost Crossover Bench

beyond-evals

There exists a measurable monthly judging volume above which a fine-tuned open judge on dedicated compute is cheaper than frontier-model token billing for equivalent judgment quality, inclusive of fine-tuning and serving overhead.

Proof of concept

Build one judge task with a fixed rubric, run it through a frontier API judge and a fine-tuned open judge over increasing volumes, and chart total cost of ownership (fine-tuning hours, cluster/hosting, ops) against per-token API spend at matched quality.

Measurement

Crossover volume in judgments per month, quality parity margin between judges on the same rubric, and total cost curves including non-token overhead.

Source video

Harness Saturation Meter

movement-lab

Marginal task-success improvement per harness iteration decays predictably before a fine-tuning cycle is warranted, so the decay curve can be used as a switch signal between harness engineering and fine-tuning.

Proof of concept

Instrument an agent improvement loop to record every harness change with its evaluation delta and wall-clock feedback latency, then test whether a flattening marginal-gain curve precedes the point where a fine-tuning cycle produces the next significant gain.

Measurement

Correlation between measured marginal gain per iteration and the observed benefit of the subsequent fine-tuning cycle across several task suites.

Source video

Trace Retention Contract

gatehouse

Teams that commit to a durable trace schema and retention policy at first deployment can later run adaptation loops that teams adding logging post-hoc cannot, and the difference is visible in how quickly they can reconstruct dense labels from historical runs.

Proof of concept

Instrument two agents — one with day-one full-fidelity trace capture plus schema, one with logging added after it works — then attempt the same trace-derived label-mining task against each history and compare how much usable supervision can be reconstructed.

Measurement

Fraction of historical runs convertible into usable supervision, time to first mined improvement loop, and storage cost per retained run.

Source video

Architectural implications

3

Trace review at scale requires dedu, sampling, and judging stages rather than in-context reading; the summary cites specialized tooling (LangSmith Engine) as the response.

Before

Engineer pastes or streams a run's trace into a frontier model and reasons about it interactively.

After

Trace data is stored, mined, filtered, and judged by a separate pipeline (potentially with fine-tuned open models), with only aggregated or sampled findings reaching a frontier model.

Consequence

Observability/evaluation becomes a data-infrastructure component with its own cost curve, scaling limits, and model choices, decoupled from the agent runtime.

Source video

High-volume trace judging pushes costs from per-token API billing toward fixed hardware cluster costs with fine-tuned open models.

Before

Evaluation spend scales linearly with tokens on frontier-model pricing.

After

Evaluation spend is dominated by cluster ownership/ops plus a one-time fine-tuning investment for a task-specific judge.

Consequence

Evaluation economics become capacity-planning decisions; the threshold where this pays off should be measured rather than assumed.

Source video

Tracing is positioned as the substrate for continual learning rather than a debugging adjunct.

Before

Add logging/observability after the agent works; treat it as ops tooling.

After

Instrument tracing at first deployment because traces are the raw input to every later improvement loop.

Consequence

Behavioral data capture becomes a day-one design requirement, and data-retention/schema decisions become load-bearing for future adaptation.

Source video

Tradeoffs and failure modes

5

Reading traces at scale

Benefit

Full-fidelity trace review gives the richest understanding of agent behavior.

Cost or risk

Cost and context limits make exhaustive reading infeasible for long-horizon runs; naive review silently degrades to sampling.

Reading traces at scale is super expensive.
Open source video
Source video

Harness engineering vs. fine-tuning

Benefit

Harness engineering yields feedback in minutes; fine-tuning scales intelligence for a vertical once the harness saturates.

Cost or risk

Over-investing in the harness hits a ceiling, while jumping to fine-tuning early sacrifices fast iteration and adds base-model customization cost.

Harness engineering gives feedback in minutes; fine-tuning optimizes base models for specific vertical tasks.
Open source video
Source video

Open models as trace judges

Benefit

Fine-tuned open models can reportedly match or exceed frontier performance on specific judging tasks at a fraction of the cost.

Cost or risk

Open models have hidden inflection points in intelligence, so capability may be mis-estimated; also implies fine-tuning effort and hardware/ops ownership.

Open models have hidden inflection points in intelligence.
Open source video
Source video

Sparse benchmark signals

Benefit

Binary pass/fail tasks like terminal-bench are cheap to score and unambiguous.

Cost or risk

They carry almost no information about why a run failed, requiring trace densification to produce actionable feedback.

How terminal bench outputs binary pass/fail results, requiring densified feedback signals from traces to guide improvement.
Open source video
Source video

Autonomy vs. determinism

Benefit

Autonomous agents handle open-ended tasks without full specification.

Cost or risk

Behavior is unpredictable before execution, so correctness cannot be established by inspection and must be inferred from mined traces.

No one knows what their agent will do until they run it. Agents trade determinism for autonomy.
Open source video
Source video

Open questions

5

How should feedback signals from traces be densified automatically — i.e., how do you turn a raw multi-million-token run plus a binary outcome into a reliable per-step reward or critique?

Why unresolved

The summary names the need (densified feedback from traces) but offers no method, labeling it a question to investigate.

Research direction

Build and benchmark trace-to-label pipelines (LLM as judge over segments, fine-tuned open judges, heuristic signal extraction) and measure whether the densified signal improves downstream agent performance.

Source video

When has a harness actually hit its ceiling, signaling it is time to fine-tune?

Why unresolved

The sandwich strategy prescribes fine-tuning to 'break ceilings' but the summary gives no criterion for detecting the ceiling.

Research direction

Define and instrument saturation metrics for harness iteration (e.g., marginal improvement per iteration) and test whether they predict when fine-tuning yields gains.

Source video

What is the true crossover point where fine-tuned open models on owned hardware beat frontier-model token pricing for trace judging?

Why unresolved

The 'fraction of the cost' claim is presented as a benchmark claim needing verification, and no volume threshold is given.

Research direction

Cost-model and empirically measure a judge workload across volumes, comparing token-billed frontier judging against fine-tuned open judges on dedicated compute including fine-tuning and ops overhead.

Source video

How can memory management and sleep-time compute be scaled for long-horizon agents?

Why unresolved

Listed explicitly as a question to investigate with no mechanism or result offered.

Research direction

Prototype offline (sleep-time) compute budgets for memory consolidation and measure their effect on long-horizon task success versus runtime cost.

Source video

Does fine-tuning an open model for trace judging preserve judgment quality across domains, or is the gain task-specific and brittle?

Why unresolved

The evidence cited is a single benchmark comparison (Harvey legal benchmark), so generality is unestablished.

Research direction

Replicate the frontier-vs-fine-tuned-open comparison across multiple verticals and judging rubrics, including drift checks after task distribution shifts.

Source video

Key claims

8
opinionVerification not requested

Improving agents is fundamentally a data mining problem.

Evidence

Improving Agents is a Data Mining Problem.

Source video
causalVerification needed

Every continual learning company is an observability company.

Evidence

Every Continual Learning Company is an Observability Company.

Question

Do organizations that sustain continual agent learning consistently build observability/tracing infrastructure as a prerequisite, and can learning be achieved without it?

Source video
causalVerification needed

No one knows what an agent will do until it is run; agents trade determinism for autonomy.

Evidence

No one knows what their agent will do until they run it. Agents trade determinism for autonomy.

Question

How much of an agent's behavior at deployment can be predicted from static artifacts (prompts, tools, model choice) versus only observable by execution?

Source video
factualVerification needed

Modern agent runs can be millions of tokens long and do not fit in context windows.

Evidence

Traces don't fit in context windows.

Question

What is the distribution of trace lengths across production long-horizon agents, and what fraction exceeds typical context limits?

Source video
causalVerification needed

Reading traces at scale is very expensive.

Evidence

Reading traces at scale is super expensive.

Question

What is the measured cost per trace reviewed at frontier-model pricing versus via fine-tuned open judging models?

Source video
comparativeVerification needed

Fine-tuning open models can match or exceed frontier performance on specific tasks at roughly one to two orders of magnitude lower cost.

Evidence

Fine-tuning open models can match or exceed frontier performance on specific tasks at a fraction of the cost.

Question

On which benchmarks and tasks does a fine-tuned open model match frontier trace-judging quality, and does the 1-2 order-of-magnitude cost gap hold once fine-tuning and serving overhead are included?

Source video
comparativeVerification needed

Harness engineering provides feedback in minutes while fine-tuning optimizes base models for specific vertical tasks.

Evidence

Harness engineering gives feedback in minutes; fine-tuning optimizes base models for specific vertical tasks.

Question

What is the actual iteration latency of harness engineering changes end-to-end, and how does it compare to a fine-tuning cycle in practice?

Source video
predictionVerification needed

Modern agent runs will soon exceed all human lifetime data production.

Evidence

Modern agent runs will soon exceed all human lifetime data production. (projection)

Question

What is the projected total token volume generated by agent runs, and how does it compare with an estimate of all human-generated text to date?

Source video

Connections

5