AI Engineer · Published 2026-10-05

From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

privateReview: unreviewedConfidence: high
AI agentsevaluationtracingObservabilityLLM-as-a-judgeai-agentsevalsopentelemetryopeninferenceerror-analysis
Open on YouTube ↗

Summary

Overview

  • Speaker: Laurie Voss
  • Channel: AI Engineer
  • Main topic: Evaluating and shipping production-ready AI agents using observability, tracing, code evaluations, LLM-as-a-judge evaluators, datasets, experiments, and meta-evaluation.
  • Purpose: To educate AI engineers on how to build, trace, evaluate, and systematically improve LLM-based AI agents using structured testing, evaluation rubrics, and observability tooling. Laurie Voss, Head of Developer Relations at Arize AI, presents a comprehensive 101 workshop on moving AI agents from casual 'vibe checks' to reliable production systems. He explains why standard software unit tests and manual reviews fail for non-deterministic AI outputs, introducing a rigorous framework of OpenTelemetry tracing (via Arize AX), code evals, LLM-as-a-judge evals, golden datasets, error analysis, and meta-evaluation to iteratively improve agent prompts and codebases.

Topic Map

Introduction and Agenda

  • Explanation: Introduction of the speaker, background, and the core agenda covering evals, tracing in Arize AX, building AI agents with the Claude Agent SDK, error analysis, code evals, LLM-as-a-judge evals, meta-evaluation, and dataset/experiment iteration loops.
  • Key claims:
    • Agents make evaluation much harder than single-LLM applications due to cascading errors and multi-step tool calls.
    • Most teams ship AI features based on vague vibe checks, which fails when models encounter new inputs.
  • Examples:
    • Mention of building a financial analysis chatbot agent using the Claude Agent SDK.
  • Terminology:
    • Arize AX
    • Claude Agent SDK
    • evals
    • traces
    • spans
  • Why it matters: Establishes why traditional testing methods are insufficient for AI applications and sets the framework for the workshop.

What are Evals and Why You Need Them (The Vibes Problem)

  • Explanation: Discussion of the limitations of manual vibe checks and unit tests for LLM applications, explaining why traces are logs for AI and evals are tests for AI.
  • Key claims:
    • Traces are logs for AI; evals are tests for AI.
    • Without evals, changing a prompt to fix one tone issue can silently hallucinate product features elsewhere.
    • Manual human review does not scale, doesn't catch regressions, and doesn't run in CI.
  • Examples:
    • Changing system prompts and breaking unexpected parts of an application.
  • Terminology:
    • vibes problem
    • traces
    • spans
    • evals
    • regression test
  • Why it matters: Explains the foundational mental model needed to transition from prototyping to engineering production AI.

Two Types of Evals: Code Evals vs. LLM-as-a-Judge Evals

  • Explanation: Comparison of code-based deterministic checks versus semantic LLM-as-a-judge evaluations, highlighting their pros, cons, and appropriate use cases.
  • Key claims:
    • Code evals are deterministic, free, fast, and reproducible like unit tests, but can be brittle.
    • LLM-as-a-judge evals use a second LLM to grade outputs against a rubric, handling meaning and semantics rather than just strings.
    • LLM judges cost money, are non-deterministic, and require calibration against human judgment.
  • Examples:
    • Code eval checking if output parses as JSON or contains a ticker symbol.
    • LLM judge checking faithfulness or tone.
  • Terminology:
    • code evals
    • LLM-as-a-judge
    • rubric
    • deterministic
  • Why it matters: Helps engineers choose the right evaluation mechanism depending on whether the test is syntactic or semantic.

Why Agents Make Evaluation Harder

  • Explanation: Analysis of multi-agent and multi-turn agent execution complexity, tool calls, and cascading errors.
  • Key claims:
    • Single LLM calls have simple input-output pairs, but agents make a series of tool calls and reasoning steps.
    • Errors cascade across steps: bad retrieval leads to bad reasoning, leading to confidently wrong output.
    • Multi-agent handoffs add another layer where triage and specialist routing can fail.
  • Examples:
    • Agent searching the web for Tesla, finding info on an 18th-century inventor instead, and writing a polished report about them.
  • Terminology:
    • cascading failures
    • multi-agent complexity
    • tool calls
    • hallucination
  • Why it matters: Demonstrates why monitoring intermediate agent steps (spans) is essential for debugging agent workflows.

Setting Up Tracing with Arize AX

  • Explanation: Demonstration of setting up OpenTelemetry tracing and OpenInference instrumentation with the Claude Agent SDK to capture raw data for evaluations.
  • Key claims:
    • You can't evaluate what you can't observe; observability comes before evaluation.
    • Auto-instrumentation via OpenInference captures prompt text, completions, token counts, model names, and tool invocations with just two lines of code.
  • Examples:
    • Code snippet registering tracer provider with Arize AX using space ID and API key.
  • Terminology:
    • OpenTelemetry
    • OpenInference
    • tracing
    • Arize AX
    • auto-instrumentation
  • Why it matters: Provides the practical implementation steps required to collect execution data from an agent application.

Error Analysis and Looking at Your Data

  • Explanation: The importance of reading traces, conducting error analysis, and categorizing failures before writing any evals.
  • Key claims:
    • Reading a dozen traces end-to-end is the most important practice in agent development.
    • 'The response was wrong' is not actionable; you must ask why and categorize by root cause (retrieval failure, reasoning error, hallucination, scope violation).
  • Examples:
    • Reviewing financial analyst agent runs for Apple, Tesla, and Nvidia to spot missing budget breakdowns or unverified private company data.
  • Terminology:
    • error analysis
    • open coding
    • axial coding
    • root cause analysis
  • Why it matters: Bridges raw trace data to actionable engineering insights by identifying specific failure modes.

Building and Running Code Evals

  • Explanation: Writing simple deterministic code evaluators in Python to check specific constraints like ticker symbol mentions.
  • Key claims:
    • Code evals provide instant, free, deterministic checks for format, structure, constraints, and required fields.
    • Grade the outcome, not the path; don't punish agents for taking a creative or unexpected path to the correct answer.
  • Examples:
    • Writing a Python evaluator to check if a financial report mentions the required stock ticker.
  • Terminology:
    • code evaluator
    • deterministic evaluation
    • evaluation result
    • suppress tracing
  • Why it matters: Shows how to implement fast, cheap programmatic quality gates into the evaluation pipeline.

Building LLM-as-a-Judge Evals and Rubrics

  • Explanation: Implementing semantic evaluations using LLM judges, covering correctness and faithfulness, structuring rubrics with XML tags, and managing tradeoffs.
  • Key claims:
    • An LLM judge rubric should have four parts: define the judge's role, define explicit pass/fail criteria, label data with XML tags, and define output choices outside the prompt.
    • Include labeled examples of good and bad outputs in your rubric to drastically improve judge consistency.
    • Chain-of-thought prompting (asking the judge to explain its reasoning before scoring) measurably improves evaluation accuracy.
  • Examples:
    • Using Built-In Correctness and Faithfulness evaluators with Claude Sonnet 4.6 as a judge.
  • Terminology:
    • LLM-as-a-judge
    • rubric
    • faithfulness
    • correctness
    • XML tags
    • chain-of-thought
  • Why it matters: Enables evaluation of complex semantic attributes like adherence to source documents and actionability.

Meta-Evaluation: Testing Your Tests

  • Explanation: Evaluating the evaluator by comparing LLM judge predictions against human annotations to measure agreement, precision, and recall.
  • Key claims:
    • Your judge is a classifier that makes predictions (pass/fail or actionable/not actionable) that can be compared against human ground truth.
    • Human inter-rater reliability is often low (0.2–0.3 Cohen's Kappa); if your judge is more consistent than humans, that's a win.
    • When judges disagree with humans, read explanations to uncover ambiguity in your rubric.
  • Examples:
    • Calculating agreement rate between human annotations and classification evaluator scores in a pandas DataFrame.
  • Terminology:
    • meta-evaluation
    • confusion matrix
    • precision
    • recall
    • inter-rater reliability
  • Why it matters: Ensures that the evaluation suite itself is trustworthy and correctly calibrated.

The Eval-Iterate Cycle and Automated Prompt Improvement

  • Explanation: Using experiments, golden datasets, and coding agents to automatically improve agent prompts based on evaluation failure explanations.
  • Key claims:
    • Save failing production or test traces as a regression dataset to test prompt improvements systematically.
    • Feed failing trace explanations back into a coding agent or LLM to automatically rewrite and improve prompt templates.
    • The impact hierarchy for improvements is: data quality fixes first, prompting improvements second, model selection third, and hyperparameter tuning lowest.
  • Examples:
    • Exporting failing traces from Arize AX, saving them as a dataset named 'aie-financial-demo-fails', and running an experiment with improved prompts.
  • Terminology:
    • eval-driven development
    • golden dataset
    • regression test set
    • impact hierarchy
    • experiments
  • Why it matters: Completes the software development lifecycle loop for AI, turning production failures into automated regression tests.

Key Points

Treat Evals as Infrastructure

  • Explanation: Evaluations should not be an afterthought; they must be treated with the same rigor and systematic integration as unit tests in traditional software engineering.
  • Evidence: Real-world engineering teams at Descript, Bolt, and Anthropic followed the exact arc of shipping on vibes, realizing it didn't scale, and building evals.
  • Practical implication: Establish telemetry and evaluation loops early in the development process rather than trying to retrofit testing after production deployment.

Grade the Outcome, Not the Path

  • Explanation: When evaluating AI agents that perform multi-step reasoning and tool calls, grading intermediate trajectories rigidly punishes creative and valid problem-solving paths.
  • Evidence: Discussion of multi-turn customer service benchmarks where agents find valid alternative execution paths that rigid trajectory testing marks as failures.
  • Practical implication: Design evaluators to check the final output and compliance with success criteria rather than enforcing a strict sequence of tool calls.

Data Quality Fixes Have the Highest Impact

  • Explanation: When iterating to improve agent performance, fixing knowledge base and retrieval data quality yields significantly higher returns than prompt tweaks or model switching.
  • Evidence: Presented in the impact hierarchy: 1. Data quality fixes, 2. Prompting improvements, 3. Model selection, 4. Hyperparameter tuning.
  • Practical implication: Invest engineering effort into retrieval augmented generation (RAG) data hygiene and grounding sources before spending time on hyperparameter tuning.

Frameworks, Models & Processes

The Swiss Cheese Model of Quality

  • How it works: Combines multiple layers of defense to catch agent errors, recognizing that no single evaluation method catches everything.
  • Components:
    • Automated code evals (catches format and syntax issues)
    • LLM-as-a-judge reviews (catches semantic and reasoning gaps)
    • Manual transcript reviews / early access programs (catches novel failures)
    • Production monitoring, A/B testing, and user feedback (catches edge cases in production)
  • When to use: When designing an end-to-end quality assurance strategy for production AI applications.

The Eval-Iterate Cycle

  • How it works: A continuous feedback loop for AI software development that mirrors traditional CI/CD pipelines.
  • Components:
    • Find failures via tracing
    • Read explanations via error analysis
    • Fix prompts or data
    • Run experiments against test datasets
    • Repeat
  • When to use: During the development, debugging, and maintenance phases of building AI agents.

The Impact Hierarchy

  • How it works: Prioritization framework for where to invest engineering resources when optimizing AI agent performance.
  • Components:
    • Data quality fixes (highest impact)
    • Prompting improvements
    • Model selection
    • Hyperparameter tuning (lowest impact)
  • When to use: When deciding what part of an AI system to refactor or tune following evaluation failures.

Examples & Case Studies

An agent tasked with finding financial data on Tesla searched the web, retrieved information on an 18th-century inventor named Tesla, and wrote a polished report about him.

  • Illustrates: Cascading failures in agent workflows where bad retrieval leads to bad reasoning and confident hallucinations.
  • Lesson: Without tracing and evaluation steps inspecting intermediate outputs, polished agent outputs can mask critical retrieval errors.

An agent tasked with booking an economy class flight creatively found a way to upgrade the flight to first-class because the travel policy allowed it under specific conditions, failing a rigid evaluation script.

  • Illustrates: Creatively correct agent behavior being incorrectly flagged as a failure by overly restrictive evaluation tests.
  • Lesson: Grade agent outcomes based on explicit success criteria rather than enforcing rigid execution paths.

A financial analyst agent correctly analyzed Amazon's AWS segment performance but failed to mention Amazon's retail stock ticker overall because the prompt asked for AWS profitability.

  • Illustrates: Subtle evaluation failures where an agent does good research but misses specific multi-part prompt constraints.
  • Lesson: Use diverse test datasets and specific evaluators to catch edge cases before user deployment.

Actionable Takeaways

  • Immediate:
    • Instrument your agent application with OpenTelemetry tracing using two lines of code.
    • Read at least a dozen raw traces end-to-end to understand actual failure modes before writing evals.
    • Start with simple code evals for deterministic checks and add LLM judges for semantic checks.
  • Strategic:
    • Treat evaluations as core infrastructure rather than an afterthought.
    • Build golden datasets from production failures to act as permanent regression test suites.
    • Involve domain experts (product managers, customer success) in defining success criteria and writing evaluation rubrics.
  • Questions to investigate:
    • What is the inter-rater reliability between human reviewers and our LLM judge rubrics?
    • How can we systematically sample production traffic (e.g., 10%) to minimize evaluation costs while maintaining statistical significance?
    • Which dimensions of agent performance require guardrail evals versus North Star metric evals?

Claims Worth Verifying

  • Human inter-rater reliability is often as low as 0.2 to 0.3 (Cohen's Kappa). (statistic)
  • Human annotators miss up to 50% of defects due to fatigue. (statistic)
  • Halving the margin of error in statistical evaluation requires 4x the sample size. (statistical principle)

Notable Quotes

"Traces are logs, for AI. Evals = tests, for AI." (at 7:42) "You can't evaluate what you can't observe, so observation comes first." (at 21:52) "The response was wrong is not actionable. Ask why." (at 76:58) "Today's failure is tomorrow's regression test." (at 105:32)

Compressed Summary

  • AI agents introduce cascading errors and multi-step reasoning that make traditional unit testing and vibe checks insufficient.
  • Arize AX and OpenTelemetry tracing provide the observability foundation required to capture agent spans and tool calls.
  • Evaluations split into code evals (deterministic, fast, cheap) and LLM-as-a-judge evals (semantic, flexible, requires calibration).
  • Effective LLM judge rubrics require defined roles, explicit criteria, XML tagging, and few-shot examples.
  • The eval-iterate-shippable loop turns production failures into regression datasets, closing the AI software development lifecycle.
  • Keywords: ai agents, evaluation, tracing, observability, llm-as-a-judge
  • Core insight: AI agents require moving past casual vibe checks by establishing observability through tracing, rigorous evaluation rubrics, and automated feedback loops that turn production failures into regression tests.

Core insights

7
Practicehigh noveltymoderate evidence

Grade the outcome, not the execution path: agent evaluators should score the final result against requirements rather than penalizing agents for taking unexpected or creative routes to a correct answer.

Why it matters

Reward-shaping on the path over-constrains agents, suppresses valid tool-use strategies, and creates brittle evals that fail agents that are actually correct. Outcome grading keeps evals aligned with real user value while leaving the agent free to explore alternative trajectories.

Generalization

Applies to any evaluated non-deterministic system where multiple valid solution paths exist (agents, planning systems, search): evaluate on terminal artifacts and constraints, not intermediate action sequences, unless the path itself is a safety or cost constraint.

Grade the outcome, not the path; don't punish agents for taking a creative or unexpected path to the correct answer.
Open source video
Practicemedium noveltymoderate evidence

Error analysis must precede eval authoring: reading a dozen traces end-to-end and categorizing failures by root cause (retrieval failure, reasoning error, hallucination, scope violation) is the highest-leverage step, and 'the response was wrong' is not an actionable failure label.

Why it matters

Evals written before failure taxonomies exist encode the wrong assumptions and miss real failure modes. Root-cause categorization turns trace data into a prioritized, targeted eval suite rather than a generic quality check.

Generalization

Failure taxonomy should be derived empirically from observed production/debug traces before constructing automated checks; this mirrors open/axial coding and applies to any observability-driven improvement loop.

Reading a dozen traces end-to-end is the most important practice in agent development.
Open source video
'The response was wrong' is not actionable; you must ask why and categorize by root cause (retrieval failure, reasoning error, hallucination, scope violation).
Open source video
Mechanismmedium noveltymoderate evidence

An LLM judge rubric should be structured as four explicit parts: define the judge's role, define explicit pass/fail criteria, label the data with XML tags, and define the output schema.

Why it matters

Forcing role, criteria, delimiter-tagged inputs, and output contract into a fixed structure makes judge prompts reproducible and parseable, reducing variance and enabling automated downstream handling of judge verdicts.

Generalization

Structured prompt contracts (role + explicit criteria + delimited data + output schema) are reusable across any LLM-based classification, scoring, or routing task, not just grading.

An LLM judge rubric should have four parts: define the judge's role, define explicit pass/fail criteria, label data with XML tags, and define output ch
Open source video
Architecturemedium noveltystrong evidence

Observability is a prerequisite for evaluation: you cannot evaluate what you cannot observe, so OpenTelemetry/OpenInference tracing must be instrumented before any eval is meaningful.

Why it matters

Evals consume trace data; without captured prompt text, completions, token counts, model names, and tool invocations, judges and code evals have nothing to grade. This fixes the layering order: instrumentation → trace collection → error analysis → evaluation.

Generalization

Any evaluation or improvement system for non-deterministic pipelines depends on a prior structured observability substrate; evals without trace capture are ungrounded.

You can't evaluate what you can't observe; observability comes before evaluation.
Open source video
Auto-instrumentation via OpenInference captures prompt text, completions, token counts, model names, and tool invocations with just two lines of code.
Open source video
Failure Modemedium noveltystrong evidence

Agents require span-level, not just input-output-level, evaluation because errors cascade across multi-step tool calls and reasoning steps, and multi-agent handoffs introduce an additional routing/triage failure surface.

Why it matters

Single-call eval patterns miss the causal chain where a retrieval failure produces confidently wrong downstream output. Evaluating only the final answer hides which span is the true locus of failure, making debugging and prompt fixes guesswork.

Generalization

For any multi-step system, evaluation and debugging must be decomposable per step; terminal-output-only evaluation cannot localize cascading failures.

Errors cascade across steps: bad retrieval leads to bad reasoning, leading to confidently wrong output.
Open source video
Agent searching the web for Tesla, finding info on an 18th-century inventor instead, and writing a polished report about them.
Open source video
Tradeofflow noveltystrong evidence

Code evals and LLM-as-a-judge evals are complementary, not substitutable: code evals are deterministic, free, fast, and reproducible but brittle, while LLM judges handle semantics and meaning but cost money, are non-deterministic, and require calibration against human judgment.

Why it matters

This dictates a two-tier eval architecture: cheap deterministic gates for syntactic/structural constraints (format, required fields, ticks) and calibrated judges for semantic properties (faithfulness, tone). Treating either as the whole solution introduces either brittleness or unsustainably noisy signal.

Generalization

Deterministic-check-plus-modeled-scoring layering recurs in any validation/testing regime where some properties are syntactic and others require semantic interpretation.

Code evals are deterministic, free, fast, and reproducible like unit tests, but can be brittle.
Open source video
LLM judges cost money, are non-deterministic, and require calibration against human judgment.
Open source video
Mental Modelmedium noveltymoderate evidence

Manual human review is structurally inadequate as a production quality gate because it does not scale, does not reliably catch regressions, and cannot run in CI—so evals must be automatable artifacts.

Why it matters

Reframes evals as CI-integrated regression tests rather than ad-hoc review activities, which determines whether quality can be maintained as prompts and code change continuously.

Generalization

Any rapidly iterating non-deterministic system needs automated, CI-runnable checks; human review remains for calibration and novel failure discovery, not gatekeeping.

Manual human review does not scale, doesn't catch regressions, and doesn't run in CI.
Open source video
Without evals, changing a prompt to fix one tone issue can silently hallucinate product features elsewhere.
Open source video

Deep dives

5

Calibrating LLM-as-a-judge evaluators for CI gating

Research question

What judge-human agreement level is achievable, and what calibration investment is required before an LLM judge can serve as a trustworthy CI gate?

Why

Without calibration, judge verdicts are non-deterministic and biased, so automated gates either block valid changes or pass regressions. Calibration determines whether LLM-as-a-judge can be a production quality gate rather than a noisy signal.

LLM judges cost money, are non-deterministic, and require calibration against human judgment.
Open source video
Code evals are deterministic, free, fast, and reproducible like unit tests, but can be brittle.
Open source video
Source video

Regression coverage for silent cross-behavior prompt breakage

Research question

How can an eval suite reliably detect unknown-unknown regressions when a localized prompt edit fixes one behavior and breaks another?

Why

Prompt changes are cross-cutting risks. Without regression coverage, CI cannot catch silent breakage, and manual review does not scale or catch regressions.

Without evals, changing a prompt to fix one tone issue can silently hallucinate product features elsewhere.
Open source video
Manual human review does not scale, doesn't catch regressions, and doesn't run in CI.
Open source video
Source video

Path-sensitive constraints in outcome-graded agent evals

Research question

When must agent evals grade the trajectory (tool-call count, token spend, disallowed actions) in addition to the final outcome?

Why

Outcome-only grading can pass inefficient, expensive, or unsafe trajectories. Production evals need hybrid outcome+constraint checks to preserve user value without ignoring cost, latency, or safety.

Grade the outcome, not the path; don't punish agents for taking a creative or unexpected path to the correct answer.
Open source video
Multi-agent handoffs add another layer where triage and specialist routing can fail.
Open source video
Source video

Sufficiency of two-line auto-instrumentation for span-level root-cause analysis

Research question

Does two-line OpenInference auto-instrumentation capture enough span detail to localize cascading multi-step agent failures without manual instrumentation?

Why

Observability precedes evaluation. If auto-instrumentation misses span-level causality, span-aware evals and debugging cannot localize the true origin of cascading failures.

Auto-instrumentation via OpenInference captures prompt text, completions, token counts, model names, and tool invocations with just two lines of code.
Open source video
You can't evaluate what you can't observe; observability comes before evaluation.
Open source video
Source video

Error analysis as eval authoring prerequisite

Research question

How does root-cause categorization of traces change the eval suite compared to writing evals from assumptions?

Why

Evals written before failure taxonomies exist encode the wrong assumptions and miss real failure modes. Root-cause categorization turns trace data into a prioritized, targeted eval suite.

Reading a dozen traces end-to-end is the most important practice in agent development.
Open source video
'The response was wrong' is not actionable; you must ask why and categorize by root cause (retrieval failure, reasoning error, hallucination, scope violation).
Open source video
Source video

Article ideas

4

Your Agent Eval Is Probably Grading the Wrong Thing

Agent evaluators should score terminal outcomes against requirements rather than penalize valid creative paths; path-based scoring should be reserved for explicit safety, cost, or latency constraints.

Angle

Argue against path-sensitive rubrics that suppress valid tool-use strategies and show where trajectory constraints are still necessary.

Source video

Observability Is Not Optional: Why Evals Without Traces Are Ungrounded

Evaluation consumes trace data, so instrumentation with OpenTelemetry/OpenInference must precede eval authoring; evals built without trace capture systematically miss cascading failures.

Angle

Architecture layering argument: instrumentation → trace collection → error analysis → evaluation.

Source video

The Two-Tier Eval Architecture: Deterministic Gates and Calibrated Judges

Code evals and LLM-as-a-judge evals are complementary, not substitutable; production systems need cheap deterministic gates for structural constraints and calibrated semantic judges for meaning.

Angle

Tradeoff analysis showing brittleness of code-only evals and noise/cost of judge-only evals, then prescribing a layered design.

Source video

Prompt Changes Are Cross-Cutting Risks: Moving Agent Quality into CI

Localized prompt edits can silently break unrelated behaviors, so agent quality must be enforced by CI-runnable regression evals rather than manual review.

Angle

Engineering discipline: treat prompt changes as dependency-affecting changes and gate them with a regression eval suite.

Source video

Project ideas

4

JudgeCal: Human-Alignment Calibration Harness for LLM Judges

beyond-evals

A rubric-structured judge can reach ≥90% agreement with human labels on a held-out agent-output set after calibration; below that threshold it should not gate CI.

Proof of concept

Build judge prompts with role, pass/fail criteria, XML-tagged data, and output schema; run across multiple models/rubrics; compare verdicts to human labels; plot agreement and variance.

Measurement

Judge-human agreement (accuracy/Cohen's kappa), judge verdict variance across repeated runs, and cost per 1,000 evaluations.

Source video

PromptRegress: Cross-Behavior Regression Suite for Prompt Edits

gatehouse

A CI eval suite covering unrelated behaviors can detect silent regressions caused by localized prompt edits that human review misses.

Proof of concept

Define a base agent with several behaviors; make a tone-only prompt edit; run human review and an automated eval suite; count regressions in unrelated behaviors.

Measurement

Recall of regressions (human review vs CI eval suite), false positive rate, and CI runtime/cost.

Source video

Trace2Eval: Span-Level Cascading Failure Localizer

beyond-evals

Two-line OpenInference auto-instrumentation captures enough span detail to localize known cascading failures (bad retrieval → bad reasoning → wrong answer) as reliably as manually instrumented span graphs.

Proof of concept

Inject known retrieval failures into a multi-step agent; compare auto-instrumented traces vs manual spans; have evaluators identify the failure-origin span.

Measurement

Percentage of failures localized to the correct span, time-to-localize, and instrumentation overhead.

Source video

HybridOutcomeGuard: Outcome Grading with Trajectory Constraints

beyond-evals

Combining terminal-outcome scoring with explicit trajectory constraints (tool-call budget, disallowed actions, token spend) detects unsafe or inefficient agent runs that outcome-only grading passes.

Proof of concept

Build an evaluator that first scores outcome against requirements, then checks constraints on spans; run on agent traces with known unsafe but correct-answer trajectories.

Measurement

Detection rate of constraint-violating runs, false positive rate on valid creative paths, and cost/latency of hybrid eval.

Source video

Architectural implications

3

Tracing/observability must be established as a layer beneath evaluation, since evals consume trace data rather than raw application state.

Before

Evaluation treated as an ad-hoc, end-of-pipeline check on final outputs, or as manual review of sample responses.

After

An instrumentation layer (OpenTelemetry/OpenInference) captures spans, token counts, model names, and tool calls, which evals then consume as input.

Consequence

Evaluation becomes a downstream consumer of a shared trace substrate, so instrumenting the agent is a prerequisite for any quality measurement, and evals can interrogate intermediate steps.

Source video

Prompt changes become cross-cutting risks that can silently break unrelated behaviors, requiring regression coverage rather than point fixes.

Before

Changing a system prompt to fix a localized issue (e.g., tone) is assumed to be a contained edit.

After

Prompt edits are treated as dependency-affecting changes that must pass a suite of evals covering unrelated behaviors.

Consequence

Prompt and agent codebases need a regression eval suite and CI gating, shifting prompt management from craft to engineering discipline.

Source video

Agent failure localization moves from the final output to the span graph, because errors cascade across tool calls and reasoning steps.

Before

Debugging centered on inspecting the agent's final response and guessing where it went wrong.

After

Each intermediate span (retrieval, reasoning, tool invocation, handoff) is inspected to find the true origin of cascading failure.

Consequence

Debugging and eval tooling must be span-aware, and multi-agent handoffs introduce an additional routing/triage failure surface that requires its own evaluation.

Source video

Tradeoffs and failure modes

4

Deterministic code evals

Benefit

Deterministic, free, fast, and reproducible like unit tests, and can gate format, structure, constraints, and required fields.

Cost or risk

Brittle—they cannot judge meaning, so they may pass semantically wrong outputs or fail valid variations.

Code evals are deterministic, free, fast, and reproducible like unit tests, but can be brittle.
Open source video
Source video

LLM-as-a-judge evals

Benefit

Handle meaning and semantics (faithfulness, tone) rather than just string matching.

Cost or risk

Cost money, are non-deterministic, and require calibration against human judgment, and therefore can give unstable or biased verdicts.

LLM judges cost money, are non-deterministic, and require calibration against human judgment.
Open source video
Source video

Path-sensitive vs outcome-sensitive grading

Benefit

Outcome grading aligns evals with user value and permits diverse valid strategies.

Cost or risk

Ignoring the path can let inefficient, expensive, or unsafe trajectories pass as long as the final answer is right.

Grade the outcome, not the path; don't punish agents for taking a creative or unexpected path to the correct answer.
Open source video
Source video

Multi-agent handoffs

Benefit

Specialist routing and triage enable decomposition of complex tasks across agents.

Cost or risk

Handoffs add another layer where triage and specialist routing can fail, compounding cascading errors.

Multi-agent handoffs add another layer where triage and specialist routing can fail.
Open source video
Source video

Open questions

4

How should LLM-as-a-judge evaluators be calibrated against human judgment, and how much human labeling is required to reach a trustworthy calibration?

Why unresolved

The summary states judges require calibration but does not specify the calibration procedure, sample sizes, or acceptance thresholds.

Research direction

Design experiments measuring judge-human agreement across rubric variants and model choices, and define minimum inter-rater thresholds for accepting a judge as a gate.

Source video

When is path-based grading necessary despite the 'grade the outcome, not the path' principle—e.g., for cost, latency, or safety constraints?

Why unresolved

The summary gives a firm outcome-grading principle but does not address cases where the trajectory itself is a source of cost, risk, or policy violation.

Research direction

Prototype hybrid evaluators combining outcome scoring with constraint checks on trajectory (tool-call count, token spend, disallowed actions).

Source video

How do you build a regression suite that reliably detects silent cross-behavior breakage when a prompt changes, given that the same prompt edit fixes one behavior and breaks another?

Why unresolved

The summary describes the failure (tone fix causing hallucination elsewhere) and prescribes CI-runnable evals but does not describe coverage strategies or how to detect unknown-unknown regressions.

Research direction

Explore dataset/experiment iteration loops and meta-evaluation to broaden coverage and measure whether eval suites catch unintended prompt side effects.

Source video

What is the minimal viable tracing instrumentation for agents, and does two-line auto-instrumentation capture enough to debug cascading multi-step failures?

Why unresolved

Auto-instrumentation is claimed to capture prompt text, completions, token counts, model names, and tool invocations, but sufficiency for span-level root-cause analysis is not established.

Research direction

Compare auto-instrumented traces against manually instrumented span graphs for ability to localize known cascading failures.

Source video

Key claims

8
opinionVerification not requested

Traces function as logs for AI systems and evals function as tests for AI systems.

Evidence

Traces are logs for AI; evals are tests for AI.

Source video
causalVerification needed

You cannot evaluate what you cannot observe, so observability precedes evaluation.

Evidence

You can't evaluate what you can't observe; observability comes before evaluation.

Question

Do evals built without trace capture systematically miss failure modes that span-level observability would reveal?

Source video
factualVerification needed

Auto-instrumentation via OpenInference captures prompt text, completions, token counts, model names, and tool invocations with only two lines of code.

Evidence

Auto-instrumentation via OpenInference captures prompt text, completions, token counts, model names, and tool invocations with just two lines of code.

Question

Does two-line OpenInference auto-instrumentation capture sufficient span-level detail to localize cascading agent failures without manual instrumentation?

Source video
causalVerification needed

Errors cascade across agent steps: bad retrieval leads to bad reasoning, which leads to confidently wrong output.

Evidence

Errors cascade across steps: bad retrieval leads to bad reasoning, leading to confidently wrong output.

Question

How often does an early-span retrieval error propagate to a confidently wrong terminal answer, and can intermediate-span evals intercept it?

Source video
causalVerification needed

Manual human review does not scale, does not catch regressions, and does not run in CI.

Evidence

Manual human review does not scale, doesn't catch regressions, and doesn't run in CI.

Question

What fraction of regressions introduced by prompt changes are missed by human review versus caught by an automated CI eval suite?

Source video
causalVerification needed

Without evals, changing a prompt to fix one tone issue can silently hallucinate product features elsewhere.

Evidence

Without evals, changing a prompt to fix one tone issue can silently hallucinate product features elsewhere.

Question

How frequently do localized prompt edits produce measurable regressions in unrelated behaviors, and does a regression eval suite reliably detect them?

Source video
comparativeVerification needed

Code evals are deterministic, free, fast, and reproducible like unit tests, but can be brittle.

Evidence

Code evals are deterministic, free, fast, and reproducible like unit tests, but can be brittle.

Question

What failure modes do deterministic code evals introduce (false passes and false failures) relative to semantic judges on the same outputs?

Source video
factualVerification needed

LLM judges cost money, are non-deterministic, and require calibration against human judgment.

Evidence

LLM judges cost money, are non-deterministic, and require calibration against human judgment.

Question

What judge-human agreement level is achievable, and what calibration investment is needed before a judge can serve as a CI gate?

Source video

Connections

5