Alex Lieberman · Published 2026-09-13

You're Not AI-Fluent Until You Understand Evals

privateReview: unreviewedConfidence: high
evalsagentsEnvironmentsVerifierstracesHarnessagent-environmentstrace-miningLLM-as-judgeagent-harnessSandboxing
Open on YouTube ↗

Summary

Overview

  • Speaker: Alex Lieberman and guest AI researcher
  • Channel: Alex Lieberman
  • Main topic: AI Evaluation (Evals), Environments, and Harness Engineering for LLM Agents
  • Purpose: To educate developers and non-technical leaders alike on how eval suites, testing environments, verifiers, and trace mining enable teams to build robust, production-grade AI agents. Alex Lieberman discusses AI evaluations (evals) with an AI researcher from a leading AI company, breaking down the concept across Easy Mode, Hard Mode, and God Mode. They cover what evals are, how they work, how environments simulate production systems safely, the role of verifiers, and how teams can systematically improve autonomous agent performance using trace data and eval suites.

Topic Map

Introduction to Evals (Easy Mode)

  • Explanation: Explaining the foundational concept of evals at the simplest level: checking whether an agent's output or action is 'right' according to human or organizational principles encoded into software.
  • Key claims:
    • Evals check whether what the agent did is correct.
    • Simple evals can be narrow Python functions checking database state before and after agent execution.
    • Evals range from simple boolean checks (did a tool run?) to nuanced alignment with organizational guidelines.
  • Examples:
    • Checking whether a CRM note field was populated or if an email draft was created correctly.
  • Terminology:
    • Eval
    • Agent
    • Python function
    • Verifier
  • Why it matters: Without evals, developers cannot reliably know whether agent modifications or new prompt iterations improve performance or cause regressions.

Environments and Production Simulation (Hard Mode)

  • Explanation: Explaining how agents need realistic simulated environments to do work safely without corrupting real production data.
  • Key claims:
    • Real production systems cannot be directly exposed to raw agent execution due to the risk of destructive errors.
    • Environments are recreated clones or simulated versions of production software (like Salesforce, CRM, email) with isolated test data.
    • Harbor is presented as an open-source framework used to manage tasks, agents, environments, and verifiers.
  • Examples:
    • Simulating a CRM and email table where agents can read, edit, and write drafts against test data.
  • Terminology:
    • Environment
    • Harbor framework
    • Production environment
    • Sandbox
    • Docker container
  • Why it matters: Safe testing requires isolating production data from agent experimentation while keeping the testing environment realistic enough to provide valid signals.

Mining Production Traces for Evals (God Mode)

  • Explanation: Discussing advanced workflows where production traces of agent failures are automatically mined and turned into new eval tasks to continuously improve agent performance.
  • Key claims:
    • Continuous improvement loops rely on turning real production failures into new eval tasks.
    • Trace data records every tool call, message, and intermediate decision made by the agent.
    • Humans review agent failures in production and define what 'good' looks like to update eval suites and agent prompts.
  • Examples:
    • Using LangSmith to inspect agent tool call loops, detect failures, and generate automated eval tests.
  • Terminology:
    • Trace data
    • LangSmith
    • Trace mining
    • Continuous improvement loop
    • Tool call loop
  • Why it matters: This represents the frontier of agent engineering, moving from manual prompt writing to self-improving systems driven by production data.

Key Points

What is an Eval?

  • Explanation: An eval is an automated test that checks if an agent successfully completed a specific task correctly.
  • Evidence: Comparing simple boolean verifiers to nuanced qualitative rubric checks.
  • Practical implication: Teams must define clear success criteria and verifiers before trusting agents with complex workflows.

The Importance of Environments

  • Explanation: Agents require realistic sandboxed environments mirroring production tools to execute workflows safely.
  • Evidence: Using cloned CRM and email tables inside Docker containers.
  • Practical implication: Do not let autonomous agents operate directly on live production databases without safety rails.

Trace Data as the Foundation for Improvement

  • Explanation: Traces capture every step, message, and tool call an agent makes, providing the audit trail needed to diagnose failures.
  • Evidence: Using platforms like LangSmith to store and analyze agent traces at scale.
  • Practical implication: Turning production failures into new test cases (evals) closes the loop on agent reliability.

Frameworks, Models & Processes

Harbor Framework

  • How it works: Provides abstractions for managing agent tasks, simulated environments, and verifiers so developers do not have to build testing infrastructure from scratch.
  • Components:
    • Task instruction
    • Agent harness
    • Environment (CRM, Email, Test data)
    • Verifier
  • When to use: When building and evaluating LLM-powered agents against complex multi-step workflows.

Examples & Case Studies

An agent is given a task to log a meeting and draft a follow-up email without sending it.

  • Illustrates: How an environment and verifier check state after agent execution.
  • Lesson: Verifiers read system state to ensure the agent followed constraints like 'Don't send it'.

Actionable Takeaways

  • Immediate:
    • Turn on tracing for any production AI agent immediately.
    • Define basic binary verifiers for critical agent tasks before scaling deployment.
  • Strategic:
    • Build simulated environments that mirror production systems without risking live customer data.
    • Establish a closed-loop system where production failures feed directly into new eval suites.
  • Questions to investigate:
    • How do we balance the cost of running comprehensive eval suites against the speed of model iteration?
    • At what point can human review of traces be fully automated by secondary verifier agents?

Claims Worth Verifying

  • Calibrating LLM evaluators against 1-to-10 scoring scales often results in neutral clustering, whereas binary pass/fail checks provide more reliable signals. (empirical)
  • Open-source frameworks like Harbor and LangSmith provide sufficient primitives for most companies to build production agent eval suites. (technological)

Notable Quotes

"My smartest AI friends told me if you want to call yourself AI fluent, you need to understand evals and how they work." (at 0:00) "The way that I think about evals at the simplest level is essentially your agents are already doing work for you. All you really want to do with an eval is check whether that thing that the agent did is like right, essentially." (at 0:29) "Not everything is immediately verifiable." (at 10:07) "An environment at a very high level is like where an agent does work." (at 16:49) "Evals are how you improve your agent." (at 46:55)

Compressed Summary

  • Evals verify whether agent actions and outputs meet defined correctness criteria.
  • Environments simulate production software safely using sandboxed databases and mock APIs.
  • Verifiers check system state after agent execution to score performance.
  • Trace mining turns real production failures into new eval test cases for continuous improvement.
  • Keywords: evals, agents, environments, verifiers, traces, harness
  • Core insight: Agent fluency requires moving beyond simple prompt engineering to building rigorous evaluation suites and simulated environments that turn production failures into continuous improvement loops.

Core insights

6
Empirical Resulthigh noveltymoderate evidence

Verifier design should default to binary pass/fail rather than graded LLM rubric scores (e.g., 1–10), because graded scales tend to collapse into neutral clustering while binary checks yield more reliable signal.

Why it matters

Directly shapes how eval suites are designed: score granularity is a reliability knob, not a cosmetic choice. Affects how teams compare prompt/model iterations and detect regressions.

Generalization

Applies to any LLM-as-judge or automated scoring pipeline where decision thresholds matter: coarse, decision-relevant labels may outperform fine-grained ordinal scales because model score distributions compress near the middle of a wide range.

Calibrating LLM evaluators against 1-to-10 scoring scales often results in neutral clustering, whereas binary pass/fail checks provide more reliable signals.
Open source video
Mechanismmedium noveltystrong evidence

Verification is performed against external system state (e.g., database contents before and after agent execution), not against the agent's textual output — so constraints like 'don't send it' are verified as the absence of a side effect.

Why it matters

Moves eval logic out of the agent transcript and into the world the agent mutates, which changes where eval code lives and what it must be able to inspect (DB rows, email drafts, CRM fields).

Generalization

Any agent operating on stateful external systems needs post-execution state assertions; transcript-only grading will miss or mis-score side effects and constraint violations.

Simple evals can be narrow Python functions checking database state before and after agent execution.
Open source video
Verifiers read system state to ensure the agent followed constraints like 'Don't send it'.
Open source video
Architecturemedium noveltymoderate evidence

Agent evaluation is decomposed into four separable components — task instruction, agent harness, environment, and verifier — so test infrastructure is composed from abstractions rather than built per-project.

Why it matters

Separating harness from environment from verifier means the same agent harness can be re-run against new environments and verifiers, and verifiers can be swapped without touching agent code — a prerequisite for regression testing at scale.

Generalization

Mirrors the test-runner/test-fixture/assertion split in conventional software testing; agents need the same boundary discipline to make evals composable and reusable.

Provides abstractions for managing agent tasks, simulated environments, and verifiers so developers do not have to build testing infrastructure from scratch.
Open source video
Task instruction
Open source video
Agent harness
Open source video
Environment (CRM, Email, Test data)
Open source video
Verifier
Open source video
Practicemedium noveltymoderate evidence

The improvement loop is closed by mining real production traces: observed agent failures are converted into new eval tasks, which then feed back into prompts and eval suites.

Why it matters

Turns production failure data into a durable test corpus, shifting agent improvement from manual prompt writing toward a data-driven regression backlog.

Generalization

Generalizes to any production AI system with instrumentation: the same trace-to-test pipeline that debugs incidents also compounds into a benchmark suite over time.

Continuous improvement loops rely on turning real production failures into new eval tasks.
Open source video
Turning production failures into new test cases (evals) closes the loop on agent reliability.
Open source video
Mental Modelmedium noveltymoderate evidence

Verifiability is a hard constraint on which agent tasks can be evaluated: some outcomes are not immediately checkable, which bounds what an eval suite can cover and implicitly argues for designing agent tasks/tool interfaces toward checkable outcomes.

Why it matters

Teams must decide scope by verifiability, not just by business value; otherwise they build agents whose real failures cannot be detected automatically.

Generalization

Applies wherever automated grading is used — pick tasks and interfaces whose success leaves an observable, machine-readable trace, or accept manual review cost.

Not everything is immediately verifiable.
Open source video
Architecturelow noveltystrong evidence

Agents are tested against cloned or simulated versions of production software (e.g., Salesforce, CRM, email) with isolated test data, typically containerized, so raw agent execution never touches live data.

Why it matters

Environment fidelity determines whether eval signals transfer to production, while isolation bounds blast radius of destructive agent actions.

Generalization

Same pattern as staging environments in conventional ops, but harder for agents because they need write access and realistic multi-tool surfaces, not just read-only reads.

Real production systems cannot be directly exposed to raw agent execution due to the risk of destructive errors.
Open source video
Environments are recreated clones or simulated versions of production software (like Salesforce, CRM, email) with isolated test data.
Open source video

Deep dives

5

Binary vs graded verifier scoring reliability

Research question

Do binary pass/fail verifiers produce higher inter-run and human agreement than 1-to-10 LLM rubric scores on the same agent tasks?

Why

Score granularity is a reliability knob for eval suites; if binary checks are more stable, teams should redesign verifiers and regression thresholds around decision-relevant labels.

Calibrating LLM evaluators against 1-to-10 scoring scales often results in neutral clustering, whereas binary pass/fail checks provide more reliable signals.
Open source video
Source video

State-based verification of agent side effects

Research question

How should verifiers assert on external system state before and after agent execution to detect constraint violations as missing or unexpected side effects?

Why

Eval correctness no longer depends on the agent's self-reporting; it moves into the mutated world, determining what environment APIs must expose.

Simple evals can be narrow Python functions checking database state before and after agent execution.
Open source video
Verifiers read system state to ensure the agent followed constraints like 'Don't send it'.
Open source video
Source video

Human review ceiling in trace-to-eval loops

Research question

At what trace volume does human review become the bottleneck in converting production failures into eval tasks, and can verifier agents triage traces with acceptable agreement?

Why

The continuous improvement loop depends on production failures becoming new eval tasks; manual review throughput caps how fast the eval suite can track real failures.

Continuous improvement loops rely on turning real production failures into new eval tasks.
Open source video
Humans review agent failures in production and define what 'good' looks like to update eval suites and agent prompts.
Open source video
Source video

Verifiability as a task and interface design constraint

Research question

How can agent tasks and tool interfaces be designed so successful outcomes leave machine-observable traces, and when is deferred verification appropriate?

Why

Bounds what an eval suite can automatically cover; teams should scope agent tasks by verifiability, not only business value.

Not everything is immediately verifiable.
Open source video
Source video

Environment fidelity threshold for eval transfer

Research question

What level of environment fidelity is required before agent eval results predict production behavior, and where does added realism stop changing pass/fail verdicts?

Why

Cloned environments are a standing engineering cost; if low-fidelity clones yield the same verdicts, teams can avoid overbuilding sandboxes.

Real production systems cannot be directly exposed to raw agent execution due to the risk of destructive errors.
Open source video
Environments are recreated clones or simulated versions of production software (like Salesforce, CRM, email) with isolated test data.
Open source video
Source video

Article ideas

4

Your Eval Suite Is Probably Measuring the Wrong Thing

Graded LLM rubrics collapse into neutral clustering, so agent teams should replace 1-to-10 scoring with binary pass/fail checks tied to external system state.

Angle

Empirical critique of LLM-as-judge scoring scales with concrete verifier redesign recommendations.

Source video

Evals Are Not a Chat Transcript: Verification Needs System State

Agent evaluation must assert on external system state before and after execution; transcript-only grading misses constraint violations and side effects.

Angle

Move eval logic from agent transcript to environment assertions, with examples of post-execution state checks.

Source video

The Four-Part Eval Stack: Task, Harness, Environment, Verifier

Treating harness, environment, and verifier as interchangeable components is the difference between ad-hoc scripts and reusable regression suites.

Angle

Architectural separation of concerns for agent testing infrastructure, mirroring test-runner/test-fixture/assertion splits.

Source video

Production Traces Are the Real Eval Backlog

Continuous agent improvement depends on turning production failures into eval tasks, making trace capture a first-class production dependency.

Angle

Trace mining as a data-driven regression backlog instead of manually authored eval suites.

Source video

Project ideas

4

Binary Verifier Bench

beyond-evals

Binary pass/fail verifiers achieve higher inter-run stability and human agreement than 1-to-10 LLM rubric scores on the same agent tasks.

Proof of concept

Build 25 agent tasks in a sandbox CRM/email environment; score each run with binary state assertions and with a 1-to-10 LLM rubric, repeated five times.

Measurement

Inter-run variance, human agreement, and regression-detection rate for each scoring mode.

Source video

Trace-to-Eval Loop

movement-lab

Trace-mined eval cases improve downstream agent success faster than hand-authored cases at equal review effort.

Proof of concept

Instrument an agent in a sandbox, mine 50 failure traces into new eval tasks, and compare against 50 hand-written evals through prompt revisions.

Measurement

Time-to-detect regressions, coverage of observed production failure modes, and agent pass rate after prompt revisions.

Source video

Verifier Agent Triage

gatehouse

Secondary verifier agents can triage production traces into pass/fail candidates with human agreement above a usable threshold for high-frequency failure classes.

Proof of concept

Have a verifier agent classify 200 human-reviewed traces; measure agreement with human labels by task type.

Measurement

Precision/recall of failure detection, human review time saved, and false-positive cost.

Source video

Environment Fidelity Ladder

new

Beyond a minimal fidelity threshold, richer cloned environments do not change pass/fail verdicts for a class of CRM/email agent tasks.

Proof of concept

Implement three environment fidelities (mock functions, stateful local clone, containerized CRM/email clone) and run the same harness and verifier suite.

Measurement

Pass/fail verdict agreement across fidelity levels and cost per eval run.

Source video

Architectural implications

4

Verifiers assert on external system state before/after execution rather than grading the agent's messages.

Before

Eval logic embedded in the agent loop or judged from the conversation transcript.

After

Verifier is a separate component with read access to environment state, invoked after the agent run.

Consequence

Eval correctness no longer depends on the agent's self-reporting; constraint violations are caught as missing/unexpected side effects.

Source video

Harness, environment, and verifier are treated as interchangeable components rather than a monolithic test script.

Before

Ad-hoc scripts that construct a prompt, run the agent, and check output in one place.

After

Task + harness + environment + verifier bound together as a reusable eval configuration.

Consequence

Regression suites can be assembled by recombining environments and verifiers, and the same harness runs across many environments.

Source video

New eval cases are sourced from production traces instead of being authored purely by hand.

Before

Eval suites written up front and manually maintained, disconnected from real usage.

After

Trace mining feeds production failures into the eval suite as new tasks.

Consequence

Eval coverage tracks actual failure modes, but requires trace capture as a non-optional production dependency.

Source video

Agent execution is sandboxed against cloned production software with isolated test data.

Before

Agents pointed at live systems, relying on prompt instructions to behave.

After

Agents run inside containerized replicas of CRM/email systems with test data.

Consequence

Environment build and maintenance becomes a standing engineering cost, and a fidelity gap between clone and production becomes a source of eval invalidity.

Source video

Tradeoffs and failure modes

4

Score granularity in LLM-based evaluation

Benefit

Binary pass/fail checks produce more reliable signal and simpler aggregation.

Cost or risk

Loses nuance for tasks whose quality is genuinely continuous or subjective, and can hide partial success.

Calibrating LLM evaluators against 1-to-10 scoring scales often results in neutral clustering, whereas binary pass/fail checks provide more reliable signals.
Open source video
Source video

Sandboxed cloned environments

Benefit

Prevents destructive agent actions against live production data while still exercising realistic multi-step workflows.

Cost or risk

Simulated environments must be built and kept in sync with production; low fidelity produces eval signals that do not transfer.

Real production systems cannot be directly exposed to raw agent execution due to the risk of destructive errors.
Open source video
Source video

Human review of production traces

Benefit

Humans define what 'good' looks like and turn failures into eval tasks and prompt updates.

Cost or risk

Manual trace review does not scale with agent volume, creating a throughput ceiling on the improvement loop.

Humans review agent failures in production and define what 'good' looks like to update eval suites and agent prompts.
Open source video
Source video

Comprehensive eval suites

Benefit

Broader eval coverage catches more regressions before production.

Cost or risk

Running large suites competes directly with iteration speed and compute cost, forcing a coverage-versus-velocity trade-off.

How do we balance the cost of running comprehensive eval suites against the speed of model iteration?
Open source video
Source video

Open questions

4

How should teams balance the cost of running comprehensive eval suites against the speed of model iteration?

Why unresolved

The summary poses this as an open question without a resolution or empirical benchmark.

Research direction

Measure marginal regression-detection value per eval added, and characterize suite subsets that preserve most detection power at lower cost.

Source video

At what point can human review of traces be fully automated by secondary verifier agents?

Why unresolved

The summary raises automated trace review as a possibility but supplies no evidence on reliability thresholds or failure conditions.

Research direction

Build verifier agents that triage traces into pass/fail candidates and measure agreement with human reviewers as a function of task type.

Source video

How do you evaluate agent tasks whose outcomes are not immediately verifiable?

Why unresolved

The summary asserts that not everything is immediately verifiable but does not specify workarounds or when deferred verification becomes possible.

Research direction

Explore deferred/eventual verification (checking state at later time horizons) and task redesign that makes outcomes machine-observable.

Source video

What level of environment fidelity is required before eval results predict production behavior?

Why unresolved

The summary describes cloned environments but provides no criteria for how closely they must mirror production.

Research direction

Compare eval outcomes across graded environment fidelities to find the point where added realism stops changing pass/fail verdicts.

Source video

Key claims

7
comparativeVerification needed

Graded 1-to-10 LLM evaluator scales tend to produce neutral clustering, while binary pass/fail checks provide more reliable signal.

Evidence

Calibrating LLM evaluators against 1-to-10 scoring scales often results in neutral clustering, whereas binary pass/fail checks provide more reliable signals.

Question

On a held-out set of agent tasks, do binary verifiers produce higher inter-run and human agreement than 1-to-10 rubric scores?

Source video
causalVerification needed

Raw agent execution should not be pointed at real production systems because of the risk of destructive errors.

Evidence

Real production systems cannot be directly exposed to raw agent execution due to the risk of destructive errors.

Question

What is the observed rate and severity of destructive actions across agent workloads when run against live versus sandboxed systems?

Source video
opinionVerification needed

Open-source frameworks like Harbor and LangSmith provide sufficient primitives for most companies to build production agent eval suites.

Evidence

Open-source frameworks like Harbor and LangSmith provide sufficient primitives for most companies to build production agent eval suites.

Question

Which eval capabilities remain unaddressed by these frameworks in production deployments, and how much custom infrastructure is still required?

Source video
causalVerification needed

Continuous improvement of agents depends on converting real production failures into new eval tasks.

Evidence

Continuous improvement loops rely on turning real production failures into new eval tasks.

Question

Does trace-mined eval suites improve downstream agent success rates faster than hand-authored eval suites given equal review effort?

Source video
factualVerification not requested

Trace data records every tool call, message, and intermediate decision made by the agent, serving as the audit trail for diagnosing failures.

Evidence

Trace data records every tool call, message, and intermediate decision made by the agent.

Source video
factualVerification not requested

Not all agent outcomes are verifiable, constraining what an eval suite can automatically check.

Evidence

Not everything is immediately verifiable.

Source video
factualVerification not requested

Verifiers check system state after agent execution to confirm constraints such as not sending an email were honored.

Evidence

Verifiers read system state to ensure the agent followed constraints like 'Don't send it'.

Source video

Connections

4