AI Engineer · Published 2026-10-17

How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI

privateReview: unreviewedConfidence: high
evalsObservabilityphoenixLLM-as-a-judgetestingtracingci-cdgolden-datasetopentelemetrynon-determinism
Open on YouTube ↗

Summary

Overview

  • Speaker: Laurie Voss
  • Channel: AI Engineer
  • Main topic: Evaluating LLM agents using Phoenix and Claude
  • Purpose: To guide AI engineers on moving past 'vibes-based' agent development to systematic, automated evaluation using Phoenix and CI/CD pipelines. Laurie Voss, Head of Developer Relations at Arize AI, presents a comprehensive workshop on how to evaluate AI agents using Phoenix and the Claude Agent SDK in TypeScript. He introduces a 7-step process starting from tracing agent steps, reading and organizing data, writing code and LLM-as-a-judge evals, testing the judge against human labels, running experiments, and locking them into CI/CD pipelines.

Topic Map

Introduction & Setup

  • Explanation: Introduction of speaker Laurie Voss, setting up the workshop repo using TypeScript, Node 22, and Phoenix on localhost:6006.
  • Key claims:
    • AI agents need traces and evals to know if changes make them better or worse.
    • Without evals, every prompt change is a gamble.
  • Examples:
    • GitHub repo seldo/aie-nyc
  • Terminology:
    • Traces
    • Evals
    • Phoenix
    • OpenInference
    • OpenTelemetry
  • Why it matters: Establish foundational tooling for tracking and evaluating AI agent behavior.

What Are Evals?

  • Explanation: Defining traces as logs for AI and evals as tests for AI, contrasting them with unit tests.
  • Key claims:
    • Traces record what your AI did at an incredible level of detail.
    • Evals run tests on your traces to see if the output is any good.
    • LLMs are non-deterministic, so traditional unit tests don't work.
  • Examples:
    • Comparing LLM output stability across multiple runs
  • Terminology:
    • Traces
    • Evals
    • Spans
    • Non-deterministic
  • Why it matters: Provides the core mental model for AI observability and testing.

Types of Evals: Code, LLM Judge, and Human

  • Explanation: Breaking down the three types of evaluations: code evals, LLM-as-a-judge evals, and human evaluations.
  • Key claims:
    • Code evals are fast, free, and brittle (deterministic checks).
    • LLM judge evals understand meaning, use rubrics, cost money, and are non-deterministic.
    • Humans are the gold standard for quality but do not scale.
  • Examples:
    • Checking JSON parsing with code eval
    • Checking tone and grounding with LLM judge
  • Terminology:
    • Code evals
    • LLM-as-a-judge
    • Rubric
    • Gold standard
  • Why it matters: Shows how to combine complementary evaluation approaches for robustness.

Step 1 & 2: Tracing and Reading Data

  • Explanation: Instrumenting the Claude Agent SDK with Phoenix using OpenInference and OpenTelemetry, and reading traces.
  • Key claims:
    • Tracing takes about 5 lines of code using OpenInference.
    • Reading traces doesn't scale, but your coding agent does.
  • Examples:
    • Registering ClaudeAgentSDKModule with Phoenix
  • Terminology:
    • OpenTelemetry
    • OpenInference
    • Spans
    • Manual instrumentation
  • Why it matters: Enables automatic collection of agent behavior without manual logging.

Step 3: Code Evals

  • Explanation: Writing deterministic TypeScript code evals for structural and format requirements.
  • Key claims:
    • Code evals run in milliseconds and cost nothing.
    • They check formatting, structure, and required content.
  • Examples:
    • Checking if output mentions the requested stock ticker
  • Terminology:
    • Code evals
    • Determinism
  • Why it matters: Provides fast, inexpensive checks for basic output structure.

Step 4 & 5: LLM-as-a-Judge and Writing Your Own Judge

  • Explanation: Using Phoenix built-in evaluations and writing custom LLM judges using rubrics and few-shot examples.
  • Key claims:
    • LLM judges evaluate semantic qualities like correctness, faithfulness, and actionability.
    • A good eval prompt needs the judge's role, explicit criteria, marked data, labeled examples, and a constrained answer.
  • Examples:
    • Creating an actionability evaluator with binary choices
  • Terminology:
    • LLM-as-a-judge
    • Rubric
    • Precision
    • Recall
    • Golden dataset
  • Why it matters: Automates semantic evaluation of agent performance at scale.

Step 6, 7 & 8: Testing Evals, Experiments, and CI/CD

  • Explanation: Running experiments against golden datasets and integrating test suites into GitHub Actions CI.
  • Key claims:
    • Treat eval prompts like code.
    • Gate your CI/CD build on acceptance criteria metrics.
  • Examples:
    • Running npm test and npm run experiment in GitHub Actions
  • Terminology:
    • Golden dataset
    • Experiments
    • CI/CD
    • Pass@k
    • Pass^k
  • Why it matters: Prevents regressions when modifying prompts, models, or agent code in production.

Key Points

Looks good to me is not a test suite

  • Explanation: Relying on manual spot-checking of a few agent runs leads to unexpected production failures when users use the agent in novel ways.
  • Evidence: Shipping agent features based on 3 manual test runs.
  • Practical implication: Establish automated evaluation suites before shipping AI agents.

Code evals are fast, free, and brittle

  • Explanation: Deterministic functions check formatting and exact matches in milliseconds without API costs.
  • Evidence: String matching and JSON parsing checks.
  • Practical implication: Use code evals for structure and format; use LLM judges for meaning.

LLM judges understand meaning but make mistakes

  • Explanation: Using a second LLM to grade outputs against a rubric handles semantics but can be non-deterministic and costly.
  • Evidence: LLM judge evaluating grounding and tone.
  • Practical implication: Validate LLM judges against human-labeled golden datasets.

Human labels form the golden dataset

  • Explanation: Humans remain the gold standard for quality judgment.
  • Evidence: Annotating 16 traces with actionability labels in Phoenix.
  • Practical implication: Build a golden dataset of 20-100 human-labeled examples to test your judges.

Frameworks, Models & Processes

The Swiss Cheese Model of Quality

  • How it works: Stacking multiple imperfect evaluation layers so that errors are caught by at least one layer.
  • Components:
    • Automated evals
    • Manual transcript review
    • Production monitoring and A/B testing
  • When to use: When building production-grade AI agents requiring high reliability.

Five Parts of a Good Eval Prompt

  • How it works: Structuring an LLM judge prompt for consistent, reliable grading.
  • Components:
    • The judge's role
    • Explicit criteria
    • Data clearly marked
    • Labeled examples
    • A constrained answer
  • When to use: When writing custom LLM-as-a-judge evaluators.

Examples & Case Studies

An agent searching for Tesla data returned a report about an 18th-century inventor instead.

  • Illustrates: Confidently wrong model behavior.
  • Lesson: Without evals, model hallucinations and confabulations go unnoticed until customers experience them.

Adding XML tags around few-shot examples flipped the evaluator's classification results from all negative to correct.

  • Illustrates: Prompt sensitivity in LLM judges.
  • Lesson: Treat eval prompts like code; structure data clearly using XML tags to avoid parsing confusion.

Actionable Takeaways

  • Immediate:
    • Read your traces before writing evals.
    • Write code evals for structure and format.
    • Add LLM-as-a-judge evals for semantic quality.
  • Strategic:
    • Build a golden dataset with human-labeled examples.
    • Run experiments when changing prompts or models.
    • Integrate test suites into CI/CD pipelines.
  • Questions to investigate:
    • How many examples are needed in the golden dataset?
    • How to balance precision and recall in LLM judges?

Claims Worth Verifying

  • Tracing takes about five lines of code using OpenInference. (technical)
  • Human-human agreement on qualitative tasks (Cohen's kappa) is often low (0.2-0.3). (statistical)

Notable Quotes

"Without evals, every change is a gamble." (at 11:49) ""Looks good to me" is not a test suite." (at 10:12) "A prompt change you haven't measured is a vibe." (at 165:10)

Compressed Summary

  • Traces are logs; evals are tests for AI.
  • Combine code evals, LLM judges, and human review.
  • Build golden datasets from human annotations.
  • Run experiments to validate prompt and model changes.
  • Integrate eval test suites into CI/CD pipelines.
  • Keywords: evals, observability, phoenix, llm-as-a-judge, testing
  • Core insight: Systematic evaluation using traces, code evals, and LLM judges replaces guesswork and vibes with rigorous testing for AI agents.

Core insights

6
Failure Modehigh noveltymoderate evidence

An LLM judge is itself an untested model: it must be validated against a human-labeled golden set before its scores are trusted, and its failure mode can be silent and degenerate. In the workshop, adding XML delimiters around few-shot examples flipped an evaluator's classifications from uniformly negative to correct — meaning the judge had been systematically wrong, not merely noisy.

Why it matters

Judges sit in the trust path of every automated quality metric and CI gate. A judge that emits a constant label (all pass or all fail) looks stable and low-variance, so it evades detection by the usual 'the numbers look consistent' heuristic, while silently invalidating every downstream decision. This makes judge calibration a prerequisite, not an optimization.

Generalization

Any LLM used as a grader, router, or classifier in an automated pipeline needs its own labeled test set and per-class agreement metrics; delimiter/formatting sensitivity is a first-class regression surface, and metric stability is not evidence of correctness.

Adding XML tags around few-shot examples flipped the evaluator's classification results from all negative to correct.
Open source video
Build a golden dataset of 20-100 human-labeled examples to test your judges.
Open source video
Validate LLM judges against human-labeled golden datasets.
Open source video
Mechanismhigh noveltyweak evidence

The human bottleneck in observability is not collecting traces but reading them, and the proposed substitute is a coding agent reading traces on your behalf rather than a human skimming transcripts.

Why it matters

This reframes 'manual trace review' from a human ritual into an automated analysis stage that can operate over hundreds of spans, keeping qualitative inspection in the loop without it becoming the scaling limit. It also implies the trace format must be machine-readable and complete enough for an agent to reason over — a constraint on instrumentation design, not just tooling choice.

Generalization

As agent systems generate more telemetry than humans can review, the analysis layer itself becomes an agentic workload; observability infrastructure should be optimized for machine consumers (structured spans, stable schemas) as much as for human dashboards.

Reading traces doesn't scale, but your coding agent does.
Open source video
Tradeoffmedium noveltystrong evidence

Eval type should be selected by the property being tested, not by preference: deterministic code checks own format/structure/required content in milliseconds at zero cost, LLM judges own semantics (correctness, faithfulness, tone, actionability) at a cost in money and determinism, and humans own ground truth at a cost in scale.

Why it matters

Teams routinely pay LLM-judge latency and cost for checks that a string match would settle (e.g., 'does the output mention the requested ticker'), or conversely rely on brittle exact matching for semantic qualities. Routing each assertion to the cheapest sufficient mechanism changes eval suite cost and runtime by orders of magnitude.

Generalization

In any verification system, partition assertions by whether they are decidable and cheap versus semantic and expensive; layer the cheap layer first so the expensive layer only runs on outputs that already pass structural checks.

Code evals run in milliseconds and cost nothing.
Open source video
Code evals are fast, free, and brittle (deterministic checks).
Open source video
LLM judge evals understand meaning, use rubrics, cost money, and are non-deterministic.
Open source video
Humans are the gold standard for quality but do not scale.
Open source video
Practicemedium noveltymoderate evidence

Eval prompts are treated as production code and their acceptance metrics become build gates: experiments run against a golden dataset, and CI/CD fails when scores fall below acceptance criteria rather than when a human feels the diff is fine.

Why it matters

This converts prompt/model/agent changes from an unreviewable diff into a measurable regression surface with a pass/fail boundary. It also introduces a new maintenance burden — the eval suite and golden dataset must be versioned alongside the agent — and forces a decision about how to threshold non-deterministic scores.

Generalization

Any system with non-deterministic outputs needs a statistical acceptance gate in CI rather than a deterministic test suite; the gate's threshold policy becomes an architectural decision, not a tooling detail.

Treat eval prompts like code.
Open source video
Gate your CI/CD build on acceptance criteria metrics.
Open source video
Without evals, every prompt change is a gamble.
Open source video
Architecturemedium noveltymoderate evidence

Tracing is treated as an auto-instrumentation concern (~5 lines registering an SDK module) emitting OpenInference/OpenTelemetry spans, rather than manual logging statements scattered through agent code.

Why it matters

Putting trace emission at the SDK/module boundary keeps observability out of business logic and produces uniform span structure across agent steps, which is what makes traces reusable as eval inputs and dataset material. Manual logging couples instrumentation to code churn and yields inconsistent schemas.

Generalization

Observability for agent runtimes should be a middleware/instrumentation layer conforming to a shared semantic convention, so that traces are a stable substrate consumable by debuggers, eval harnesses, and dataset builders alike.

Tracing takes about 5 lines of code using OpenInference.
Open source video
Registering ClaudeAgentSDKModule with Phoenix
Open source video
Mental Modelmedium noveltyweak evidence

Quality is engineered as stacked imperfect layers — automated evals, manual transcript review, production monitoring and A/B testing — on the assumption that each layer misses different failures, so no single eval is expected to be sufficient.

Why it matters

This shifts the design question from 'how do we write a good eval' to 'what does each layer catch that the others don't,' which justifies paying for redundant coverage (e.g., an LLM judge plus sampled human review plus live monitoring) instead of optimizing one metric to exhaustion.

Generalization

For systems with open-ended failure surfaces, coverage comes from heterogeneous detectors with uncorrelated blind spots, not from maximizing the accuracy of a single detector.

Stacking multiple imperfect evaluation layers so that errors are caught by at least one layer.
Open source video
Automated evals
Open source video
Manual transcript review
Open source video
Production monitoring and A/B testing
Open source video

Deep dives

5

Meta-evaluation: certifying an LLM judge before it gates anything

Research question

How many human-labeled examples, and which agreement statistics, are required before an LLM judge's labels can be trusted as a build gate — and how do we detect a degenerate constant predictor that reports stable but uniformly wrong labels?

Why

Judges sit in the trust path of every automated quality metric. A judge that always emits the same label looks low-variance and therefore 'stable', so the usual consistency heuristic cannot catch it; only a human-labeled golden set with per-class agreement can. Without this, CI gates certify nothing.

Build a golden dataset of 20-100 human-labeled examples to test your judges.
Open source video
Validate LLM judges against human-labeled golden datasets.
Open source video
Adding XML tags around few-shot examples flipped the evaluator's classification results from all negative to correct.
Open source video
Source video

Judge prompt sensitivity and drift as a regression surface

Research question

How sensitive are LLM judge classifications to non-semantic prompt perturbations — delimiter placement, few-shot ordering and count, whitespace — and what re-validation cadence detects judge degradation after an underlying model update?

Why

The XML-delimiter flip is a single anecdote showing a silent, total classification inversion. If a formatting change can flip an evaluator from uniformly wrong to correct, the reverse can happen in production with no signal. This makes judge prompts a versioned artifact needing perturbation tests, not a one-time authoring task.

Adding XML tags around few-shot examples flipped the evaluator's classification results from all negative to correct.
Open source video
A good eval prompt needs the judge's role, explicit criteria, marked data, labeled examples, and a constrained answer.
Open source video
Treat eval prompts like code.
Open source video
Source video

Statistical release gates for non-deterministic agent outputs

Research question

For the same agent and golden set, how do single-run score thresholds, pass@k, and pass^k compare on false-alarm rate and real-regression escape rate per CI dollar spent?

Why

Gating on non-deterministic scores with a small golden set forces a choice between flaky builds and thresholds loose enough to miss regressions. The gate policy is an architectural decision — it determines CI runtime, API spend, and what class of regression can never be caught before merge.

Gate your CI/CD build on acceptance criteria metrics.
Open source video
Without evals, every prompt change is a gamble.
Open source video
Treat eval prompts like code.
Open source video
Source video

Agent-driven trace triage as a replacement for human transcript reading

Research question

Does a coding agent reading structured spans detect known-bad traces at a rate comparable to human reviewers, and at what rate does it produce confidently wrong diagnoses?

Why

The claim that coding agents substitute for human trace reading is asserted without a procedure or validation. If it holds, qualitative inspection stays in the loop without being the scaling limit; if it does not, it reintroduces 'looks good to me' at machine speed and volume. Either outcome constrains how spans must be structured for machine consumers.

Reading traces doesn't scale, but your coding agent does.
Open source video
Traces record what your AI did at an incredible level of detail.
Open source video
Tracing takes about 5 lines of code using OpenInference.
Open source video
Source video

Cost-aware routing of assertions across code evals, judges, and humans

Research question

What is the measured cost and runtime reduction from partitioning an eval suite by decidability — deterministic checks first, judges only on survivors — versus running every assertion through an LLM judge?

Why

Teams pay judge latency and cost for checks a string match would settle, and conversely use brittle exact matching for semantic qualities. Routing assertions to the cheapest sufficient mechanism changes suite cost and runtime by orders of magnitude and is directly actionable, unlike general 'use the right eval type' advice.

Code evals run in milliseconds and cost nothing.
Open source video
LLM judge evals understand meaning, use rubrics, cost money, and are non-deterministic.
Open source video
Humans are the gold standard for quality but do not scale.
Open source video
Source video

Article ideas

4

Your Judge Is a Model, Not an Oracle

An LLM judge is an untested classifier until it has been scored against a human-labeled golden set; because a systematically wrong judge produces stable, low-variance scores, metric consistency is actively misleading evidence of correctness and calibration must precede gating.

Angle

Treat the judge as the system under test: per-class agreement, degenerate-predictor detection, and the XML-delimiter flip as a case study in silent total failure.

Source video

Stop Paying an LLM to Check Your JSON

Most eval suites route assertions to the wrong mechanism; partitioning by decidability — deterministic checks first, judges only on outputs that survive — cuts suite cost and runtime by orders of magnitude while increasing semantic coverage where it actually matters.

Angle

Build the eval suite as a test pyramid with an explicit routing rule per assertion, and show where exact matching silently misgrades semantic quality.

Source video

The Prompt Diff Is the Release Gate

Once eval prompts are versioned like code and acceptance metrics gate CI, the hardest remaining problem is statistical rather than editorial: non-deterministic scores plus a small golden set make every threshold a choice between flaky builds and undetected regressions.

Angle

Frame CI gating as a statistical policy decision — repeated runs, pass@k versus pass^k, threshold selection — and treat golden-dataset refresh as roadmap-critical maintenance.

Source video

Depth in Layers, Not Accuracy in One Metric

For open-ended agent failure surfaces, coverage comes from heterogeneous detectors with uncorrelated blind spots — automated evals, human review, production monitoring — not from optimizing a single eval to exhaustion.

Angle

Reframe the design question from 'how do we write a good eval' to 'what does each layer catch that the others don't', and justify paying for redundant coverage.

Source video

Project ideas

4

Judge Certifier

beyond-evals

For a given judge and task, precision/recall estimated from a human-labeled golden set of N examples converges tightly enough at a specific N to support a build gate, and a constant-label judge will be detected by per-class agreement even though its run-to-run variance is near zero.

Proof of concept

Take 100 traces from a real agent, hand-label them, then sweep golden-set size (10, 20, 40, 80) and measure variance in judge precision/recall estimates. Include a deliberately degenerate judge that always returns 'pass' as a control to confirm the variance-based heuristic misses it while per-class agreement flags it.

Measurement

Confidence-interval width on precision/recall versus golden-set size; detection rate of the constant-label control judge; false-certification rate at each set size.

Source video

Judge Perturbation Fuzzer

beyond-evals

Non-semantic perturbations to a judge prompt (delimiter style, few-shot ordering, few-shot count, whitespace) cause classification flips on a labeled golden set at a measurable rate, and that flip rate varies more across perturbations than across repeated runs of a single prompt.

Proof of concept

Generate a matrix of judge-prompt variants around one task, run each against a fixed labeled set, and score agreement with human labels. Report per-perturbation deltas to rank which formatting dimensions are the real regression surface.

Measurement

Per-perturbation accuracy delta and flip rate versus human labels; ratio of perturbation-induced variance to sampling variance across repeated identical runs.

Source video

Gate Policy Bench

gatehouse

For the same agent and golden set, pass^k with a moderate threshold strictly dominates single-run scoring on regression escape rate per CI dollar, and the dominant cost driver is repeated sampling rather than judge pricing.

Proof of concept

Inject a known set of quality regressions into an agent, then run three gate configurations — single-run threshold, pass@k, pass^k — in CI against the same golden dataset and record which regressions each gate catches and which good builds it blocks.

Measurement

False-alarm rate, regression escape rate, CI wall-clock time, and API spend per gate configuration per injected regression.

Source video

Agent Trace Triage

new

A coding agent reading structured OpenInference spans agrees with human reviewers on labeled traces at a rate high enough to substitute for manual transcript reading, but exhibits a measurable rate of confidently wrong diagnoses that must be reported rather than hidden.

Proof of concept

Assemble a labeled trace set containing known-bad and known-good agent runs, have both a human reviewer and an agent-driven triage harness diagnose each trace, and compare diagnoses against the ground-truth labels.

Measurement

Detection rate on known-bad traces, false-positive rate on good traces, confident-but-wrong diagnosis rate, and cost per trace compared to human review time.

Source video

Architectural implications

4

Trace collection is decoupled from agent code via OpenInference/OpenTelemetry module registration.

Before

Ad-hoc console logging and hand-written log lines interleaved with agent logic.

After

A single instrumentation hook that emits structured spans for every agent step, feeding both a trace UI and downstream eval jobs.

Consequence

Span schema becomes a shared contract; changing it breaks eval pipelines and golden dataset linkage, so the trace format needs versioning discipline.

Source video

The golden dataset becomes a first-class artifact, produced by annotating traces in the observability platform.

Before

Eval inputs approximated by whatever prompts the developer had on hand.

After

A curated, versioned set of 20-100 human-labeled traces that judges are validated against and experiments are run over.

Consequence

Labeling throughput and dataset curation become roadmap-critical; the dataset must be refreshed as usage patterns drift, or the eval suite grades yesterday's failures.

Source video

A meta-eval layer exists above the agent: the judge is itself tested against human labels before its scores gate the build.

Before

One evaluation pass — the LLM judge's verdict is taken as the quality signal.

After

Two-tier pipeline: judge produces labels, a labeled golden set measures the judge's precision/recall, and only a validated judge feeds CI.

Consequence

Adds a calibration loop and per-judge maintenance burden, but makes failures in the grader auditable instead of invisible.

Source video

Prompt/agent changes move behind a statistical acceptance gate in CI/CD rather than manual sign-off.

Before

Ship after a few manual runs; quality regressions discovered in production by users.

After

Experiments against a golden dataset run in GitHub Actions, and the build fails on acceptance-criteria metrics.

Consequence

CI runtime and API spend rise, and non-determinism must be handled with repeated runs and pass@k / pass^k-style metrics rather than a single sample.

Source video

Tradeoffs and failure modes

5

Code evals: determinism vs. semantic blindness

Benefit

Millisecond, zero-cost checks for format, structure, and required content.

Cost or risk

Brittle — correct outputs that vary in surface form fail, so exact-match checks are a poor proxy for quality.

Code evals are fast, free, and brittle (deterministic checks).
Open source video
Source video

LLM-as-a-judge: semantic coverage vs. trustworthiness

Benefit

Can grade meaning — correctness, faithfulness, tone, actionability — at scale.

Cost or risk

Non-deterministic and paid per call; can be systematically wrong in ways that look like stable metrics, as when every case was labeled negative until delimiters were added.

LLM judges understand meaning but make mistakes
Open source video
Source video

Human evaluation: ground truth vs. throughput

Benefit

The gold standard for quality judgment and the source of golden-set labels.

Cost or risk

Does not scale — the documented in-session labeling effort was 16 traces, which is thin evidence for calibrating a classifier.

Humans are the gold standard for quality judgment but do not scale.
Open source video
Source video

Manual spot-checking as a release gate

Benefit

Fast and requires no infrastructure; catches gross breakage.

Cost or risk

Produces confident false confidence — agents pass three ad-hoc runs and then fail on novel user behavior in production.

Shipping agent features based on 3 manual test runs.
Open source video
Source video

CI gating on eval metrics

Benefit

Turns prompt and model changes into measurable regressions caught before merge.

Cost or risk

Non-deterministic scores plus a small golden set make thresholds noisy, inviting either flaky builds or thresholds loose enough to miss real regressions.

Gate your CI/CD build on acceptance criteria metrics.
Open source video
Source video

Open questions

4

How many human labels are actually needed to certify a judge's precision and recall well enough to gate a build?

Why unresolved

The workshop demonstrates a 16-trace annotation and recommends 20-100 labeled examples, but the summary gives no agreement statistics or confidence intervals tying label count to gate reliability.

Research direction

Run a label-count-vs-metric-confidence study: sweep golden set size, measure variance in judge precision/recall estimates, and derive minimum-set guidance per decision threshold.

Source video

How should CI thresholds be set for non-deterministic agent outputs — single-run score, pass@k, or pass^k?

Why unresolved

Both metrics are named, but the summary does not state which one the gate uses, what threshold was chosen, or how flakiness was handled in practice.

Research direction

Compare pass@k versus pass^k as release gates on the same agent, measuring false-alarm rate and escape rate of real regressions per CI dollar spent.

Source video

Does delegating trace reading to a coding agent reintroduce the 'looks good to me' problem at a larger scale?

Why unresolved

The claim that coding agents substitute for human trace reading is asserted without a described procedure, prompt, or validation of the agent's findings against human analysis.

Research direction

Build an agent-driven trace triage harness and measure its agreement with human reviewers on labeled traces, including its rate of confidently wrong diagnoses.

Source video

How brittle are judges to prompt formatting beyond delimiter placement, and how would drift be detected in production?

Why unresolved

The XML-tag flip is a single anecdote; no systematic sensitivity testing or monitoring for judge drift is described.

Research direction

Perturbation testing of judge prompts (delimiters, ordering, whitespace, few-shot count) scored against the golden set, plus periodic re-validation jobs to detect degradation after model updates.

Source video

Key claims

8
causalVerification needed

Traditional unit tests do not work for LLM outputs because the underlying models are non-deterministic.

Evidence

LLMs are non-deterministic, so traditional unit tests don't work.

Question

What measured output variance across repeated identical runs justifies replacing exact assertions, and does variance differ materially between temperature settings and providers?

Source video
causalVerification needed

An evaluator's classifications can be restored from uniformly wrong to correct purely by adding XML delimiters around few-shot examples, with no other change.

Evidence

Adding XML tags around few-shot examples flipped the evaluator's classification results from all negative to correct.

Question

Is this result reproducible across judge models, and what is the mechanism — attention to example boundaries, tokenization, or instruction-following degradation?

Source video
factualVerification needed

Adding tracing to an agent via OpenInference requires roughly five lines of code.

Evidence

Tracing takes about 5 lines of code using OpenInference.

Question

What is the actual line count and configuration surface for a non-trivial agent with nested tool calls, and what fraction of spans capture tool arguments and results?

Source video
comparativeVerification not requested

Deterministic code-based evals execute in milliseconds and incur no API cost.

Evidence

Code evals run in milliseconds and cost nothing.

Source video
comparativeVerification needed

Human trace reading does not scale, whereas a coding agent can perform the same review function.

Evidence

Reading traces doesn't scale, but your coding agent does.

Question

How does an agent reviewer's detection rate on known-bad traces compare to a human reviewer's, and at what cost per trace?

Source video
opinionVerification not requested

Eval prompts should be versioned and treated with the same rigor as code.

Evidence

Treat eval prompts like code.

Source video
opinionVerification needed

Effective eval prompts require five structural elements: judge role, explicit criteria, clearly marked data, labeled examples, and a constrained answer.

Evidence

A good eval prompt needs the judge's role, explicit criteria, marked data, labeled examples, and a constrained answer.

Question

Which of the five elements contributes most to judge accuracy, and does the constrained answer format mainly improve parseability or also classification quality?

Source video
causalVerification not requested

Manual spot-checking of a handful of agent runs is not a viable test suite and leads to production failures on novel user inputs.

Evidence

Looks good to me is not a test suite

Source video

Connections

5