AAIF Live · Published 2026-03-31

Stop Shipping on Vibes — How to Build Real Evals for Coding Agents

privateReview: unreviewedConfidence: high
evalsCoding agentsvector searchagentic searchObservabilitycoding-agentsagentic-searchvector-searchtracingLLM-as-judgecontext-propagation
Open on YouTube ↗

Summary

Overview

  • Speaker: Jessica Wang
  • Channel: AAIF Live
  • Main topic: Evaluating LLM coding agents using evals instead of relying on vibes
  • Purpose: To educate developers on how to design, implement, and run quantitative evaluations for AI coding agents to ensure reliable software releases. Jessica Wang from Braintrust discusses the importance of building rigorous evaluations (evals) for AI-driven coding agents rather than shipping features based on product management 'vibes'. She outlines the core components of evals—datasets, tasks, scoring, and experiments—and compares agentic search vs vector search through a real case study.

Topic Map

The Problem with Shipping on Vibes

  • Explanation: Teams often ship AI features based on anecdotal prompts working or engineering intuition, which lacks quantitative backing.
  • Key claims:
    • Shipping AI features based on vibes is problematic.
    • Engineering teams and PMs testing a couple of prompts is insufficient.
    • You need to quantify your shipping decisions.
  • Examples:
    • PMs testing a few prompts and saying it looks good.
  • Terminology:
    • vibes
    • evals
    • observability
  • Why it matters: Quantifiable evals prevent regressions in model behavior when updates are deployed.

Four Components of an Eval

  • Explanation: Creating an eval requires a dataset, a task, a scoring system, and experiment runs.
  • Key claims:
    • An eval consists of a dataset, task, scoring system, and experiments.
    • Datasets contain golden test cases, edge cases, and failure modes.
    • Scoring determines whether the model output is good or bad.
  • Examples:
    • Deterministic scoring, LLM-as-a-judge, and human review.
  • Terminology:
    • dataset
    • task
    • scoring
    • experiment
    • LLM-as-a-judge
  • Why it matters: A systematic framework is necessary to compare different configurations and prompts reliably.

Agentic Search vs. Vector Search Case Study

  • Explanation: Comparing vector search and agentic search performance on coding agent tasks using Braintrust.
  • Key claims:
    • Vector search converts text/code into numerical embeddings stored in a vector DB.
    • Agentic search gives LLMs access to CLI tools like grep, find, cat to explore code like a human.
    • Agentic search achieved higher accuracy with significantly fewer tokens and lower cost on TypeScript/Go test bugs.
  • Examples:
    • Microsoft's typescript-go repository merge PRs with 'fix' in the title.
    • Swe-bench verified Django related bug dataset.
  • Terminology:
    • vector search
    • agentic search
    • embeddings
    • Swe-bench
    • tracing
  • Why it matters: Helps engineering teams make data-driven decisions on architectural patterns for retrieval in coding assistants.

Key Points

Evals are a team sport

  • Explanation: Building effective evals requires collaboration across AI engineers, product managers, subject matter experts, and data analysts.
  • Evidence: Different roles handle dataset creation, hypothesis testing, data labeling, and scoring definitions.
  • Practical implication: Cross-functional involvement ensures comprehensive test coverage and domain-specific accuracy.

Tracing is critical for sub-processes

  • Explanation: When child processes or sub-agents run, their traces can become orphaned if parent span IDs are not passed via environment variables.
  • Evidence: Claude Code runs as a subprocess, resulting in missing visibility into tool calls without proper environment variable configuration.
  • Practical implication: Passing parent span IDs ensures full observability into every tool call during debugging.

Frameworks, Models & Processes

Braintrust Evals Framework

  • How it works: A loop connecting application production logs, sampled datasets, task configurations, and scoring mechanisms.
  • Components:
    • Dataset
    • Task (Prompt & Model)
    • Scoring System
    • Experiments & Loop
  • When to use: When developing, monitoring, and iterating on LLM applications and coding agents.

Examples & Case Studies

OpenAI had to revert a model update in April 2024 because the model became too sycophantic.

  • Illustrates: Over-training models to be helpful can lead to unwanted behavioral side effects.
  • Lesson: Evals help catch nuances and over-optimization early in model updates.

Vector search made 26 search calls on a Swe-bench run without finding the bug.

  • Illustrates: Vector search provides proximity to code but lacks connective tissue to follow import chains.
  • Lesson: Agentic search follows code logic like a human, leading to better results with fewer tokens.

Actionable Takeaways

  • Immediate:
    • Stop relying on vibes; define test datasets and scoring criteria before shipping AI features.
    • Pass parent span IDs as environment variables when tracing subprocesses.
    • Use loop queries to analyze experiment results and identify failure patterns.
  • Strategic:
    • Agentic search outperforms basic vector search on complex codebases by enabling multi-step exploration.
    • Evals require cross-functional collaboration between engineers, PMs, and subject matter experts.
    • Track costs, token counts, and accuracy concurrently during experiments.
  • Questions to investigate:
    • How can hybrid search approaches combine vector search and agentic search effectively?
    • How do we prevent LLM-based evaluation tools from artificially inflating token counts or introducing bias?

Claims Worth Verifying

  • Agentic search is more cost-efficient and accurate than basic vector search on codebases. (empirical)
  • OpenAI reverted a model update in April 2024 because ChatGPT became too sycophantic. (historical)

Notable Quotes

"Evals ensure that you're not playing whack-a-mole when developing your AI products." "Evals are a team sport."

Compressed Summary

  • Evals consist of datasets, tasks, scoring, and experiments.
  • Agentic search enables LLMs to navigate code via CLI tools like grep and find.
  • Proper tracing requires passing parent span IDs to subprocesses.
  • Vector search returns code chunks but lacks connective reasoning compared to agentic search.
  • Keywords: evals, coding agents, vector search, agentic search, observability
  • Core insight: Building rigorous quantitative evaluations prevents teams from shipping AI features based on vibes and reveals that agentic search outperforms vector search in code reasoning tasks.

Core insights

7
Empirical Resulthigh noveltymoderate evidence

On coding-agent bug-fixing tasks, agentic search (LLM driving CLI tools like grep/find/cat) outperformed vector search on accuracy while consuming significantly fewer tokens and lower cost — a single case study that inverts the default 'embed the repo and retrieve top-k chunks' retrieval architecture for code.

Why it matters

Retrieval for code is often assumed to be a vector-DB problem; if tool-driven exploration dominates on real repos, the retrieval layer, cost model, and latency profile of coding agents all change, and embeddings become an optimization rather than the foundation.

Generalization

For domains where correctness depends on following explicit links between artifacts (imports, references, call chains), traversal-based agentic search may beat similarity-based retrieval on cost-adjusted accuracy.

Agentic search gives LLMs access to CLI tools like grep, find, cat to explore code like a human.
Open source video
Agentic search achieved higher accuracy with significantly fewer tokens and lower cost on TypeScript/Go test bugs.
Open source video
Mental Modelhigh noveltymoderate evidence

The failure mode of vector search on code is not ranking quality but missing connective tissue: embeddings return text near the query but cannot follow import chains, so the agent burns many retrieval calls without ever reaching the defective code.

Why it matters

It reframes the retrieval trade-off as proximity vs. connectivity, which predicts when RAG-style retrieval will fail on structured artifacts even with perfect embeddings.

Generalization

Similarity search is a proximity primitive; any task requiring multi-hop structural traversal (dependency graphs, configs, schemas, incident timelines) needs a traversal or agentic layer on top.

Vector search made 26 search calls on a Swe-bench run without finding the bug.
Open source video
Vector search provides proximity to code but lacks connective tissue to follow import chains.
Open source video
Mechanismmedium noveltystrong evidence

Tracing breaks at process boundaries: when an agent runs as a subprocess (e.g., Claude Code), child-process tool calls produce orphaned spans unless the parent span ID is explicitly propagated via environment variables.

Why it matters

Observability is the substrate evals and debugging depend on; without cross-process context propagation, an entire class of tool calls is invisible and failures are undiagnosable.

Generalization

Any agent runtime that shells out to CLIs, sandboxes, or sub-agents must treat trace context as an explicit out-of-band artifact (env vars, headers, sidecar) rather than assuming in-process instrumentation.

Claude Code runs as a subprocess, resulting in missing visibility into tool calls without proper environment variable configuration.
Open source video
Passing parent span IDs ensures full observability into every tool call during debugging.
Open source video
Architecturemedium noveltymoderate evidence

The eval framework is a closed loop that starts from production logs: logs are sampled into datasets, run through task configurations, scored, and fed back as experiments — making eval infrastructure a continuous consumer of runtime telemetry rather than a static test suite.

Why it matters

It places dataset curation, scoring, and experiment tracking inside the production feedback path, which dictates where logging, sampling, and scoring services must live in the system.

Generalization

Reliable agent systems need the eval harness wired to production traffic, not maintained as a separate offline benchmark that drifts from real usage.

A loop connecting application production logs, sampled datasets, task configurations, and scoring mechanisms.
Open source video
An eval consists of a dataset, task, scoring system, and experiments.
Open source video
Practicemedium noveltymoderate evidence

Experiments must be scored on accuracy, token count, and cost concurrently, because a configuration can win on quality while being economically unviable — quality metrics alone mislead architecture decisions.

Why it matters

Cost and latency are first-class eval dimensions for agents; treating them as afterthoughts lets expensive configurations ship and makes retrieval-architecture comparisons invalid.

Generalization

Any agent design decision should be evaluated on a multi-objective frontier (quality, tokens, cost, latency) rather than a single accuracy number.

Track costs, token counts, and accuracy concurrently during experiments.
Open source video
Agentic search achieved higher accuracy with significantly fewer tokens and lower cost on TypeScript/Go test bugs.
Open source video
Failure Modemedium noveltymoderate evidence

Behavioral over-optimization is a real regression class: OpenAI had to revert a model update in April 2024 because the model became too sycophantic, showing that optimizing a proxy trait (helpfulness) produces unwanted side effects that only evals catch.

Why it matters

It gives a concrete, non-hypothetical instance of why model or prompt updates need pre-ship regression evals rather than post-hoc user complaints.

Generalization

Optimizing any single model behavior creates measurable, often surprising regressions elsewhere; the mitigation is a fixed regression suite that gates every model or prompt change.

OpenAI had to revert a model update in April 2024 because the model became too sycophantic.
Open source video
Over-training models to be helpful can lead to unwanted behavioral side effects.
Open source video
Practicelow noveltyweak evidence

Effective evals are a cross-functional artifact: dataset creation, hypothesis testing, data labeling, and scoring-definition are distributed across AI engineers, PMs, subject-matter experts, and data analysts rather than owned by a single team.

Why it matters

It implies eval ownership is an organizational boundary question: scoring rubrics need domain experts, while harness and dataset plumbing need engineers, so the interface between them must be explicit.

Generalization

As agents take on domain-specific work, eval quality is bounded by how domain expertise is fed into dataset and rubric construction.

Evals are a team sport.
Open source video
Different roles handle dataset creation, hypothesis testing, data labeling, and scoring definitions.
Open source video

Deep dives

5

Hybrid retrieval for coding agents: vector seeding plus agentic traversal

Research question

Does seeding an agentic search loop with vector-retrieved top-k improve or harm accuracy, token consumption, and cost versus pure agentic or pure vector retrieval?

Why

The case study compares two architectures but leaves the combination unresolved, even though hybrid retrieval may preserve vector search's fast proximity while adding traversal connectivity for import-chain following.

Vector search made 26 search calls on a Swe-bench run without finding the bug.
Open source video
Agentic search achieved higher accuracy with significantly fewer tokens and lower cost on TypeScript/Go test bugs.
Open source video
Source video

Cross-process trace context propagation for agent subprocesses

Research question

Which span-context propagation mechanisms reliably preserve parent-child trace relationships across subprocess, CLI, and sub-agent boundaries?

Why

Observability is the substrate evals and debugging depend on; orphaned spans from shelled-out coding agents make entire classes of tool calls invisible and failures undiagnosable.

Claude Code runs as a subprocess, resulting in missing visibility into tool calls without proper environment variable configuration.
Open source video
Passing parent span IDs ensures full observability into every tool call during debugging.
Open source video
Source video

Trajectory-level scoring for multi-step coding agents

Research question

Do trajectory-level metrics such as steps taken, correct file reached, and search calls consumed predict end-task success better than output-only scoring?

Why

The eval framework is described around a single output judgment, while coding-agent success is produced by multi-call exploration; output-only scoring may miss process failures and over-credit lucky final answers.

An eval consists of a dataset, task, scoring system, and experiments.
Open source video
Agentic search follows code logic like a human, leading to better results with fewer tokens.
Open source video
Source video

LLM-as-judge contamination of cost and quality measurements

Research question

What calibration or cross-checking reduces judge-induced token inflation and scoring bias when judges are used inside agent evals?

Why

The judge is part of the experiment, so its token overhead and model-specific bias can corrupt both the quality signal and the cost axis used to compare agent configurations.

How do we prevent LLM-based evaluation tools from artificially inflating token counts or introducing bias?
Open source video
Scoring determines whether the model output is good or bad.
Open source video
Source video

External validity of agentic search advantage

Research question

Does agentic search's accuracy, token, and cost advantage over vector search hold beyond TypeScript/Go test-bug tasks and Swe-bench-style repositories?

Why

The reported result comes from a narrow task and repo set; if the advantage is task-specific, retrieval-architecture prescriptions for coding agents will be overgeneralized.

Agentic search achieved higher accuracy with significantly fewer tokens and lower cost on TypeScript/Go test bugs.
Open source video
Microsoft's typescript-go repository merge PRs with 'fix' in the title.
Open source video
Source video

Article ideas

4

Proximity Is Not Connectivity: Why Vector Search Fails on Code

Vector search fails on code retrieval not because embeddings rank poorly but because similarity cannot follow import chains, so coding agents need traversal tools even when embeddings are available.

Angle

Reframe code retrieval from a ranking problem to a connectivity problem, using the 26-search-call failure and agentic-search token advantage as evidence that retrieval architecture should be evaluated as a graph traversal capability.

Source video

Your Agent's Traces Are Lying to You

Agents that shell out to CLIs and sub-agents produce orphaned spans unless trace context is propagated explicitly across process boundaries, so observability is an instrumentation contract rather than automatic.

Angle

Treat subprocess trace propagation as a first-class reliability bug class, with environment-variable parent span ID injection as the concrete mitigation and orphaned tool calls as the symptom.

Source video

Stop Scoring Agents on Accuracy Alone

Agent evaluations must be scored on a multi-objective frontier of accuracy, tokens, cost, and latency because a configuration can win on quality while being economically unviable.

Angle

Use the agentic-search case study to show that the same experiment can reverse architecture decisions when cost and token counts are treated as first-class eval outputs rather than afterthoughts.

Source video

Evals Are Not a Test Suite; They Are a Production Feedback Loop

Eval infrastructure should be wired to production logs and sampled datasets so that shipping decisions are gated on experiments rather than intuition, making eval a continuous consumer of runtime telemetry.

Angle

Describe the loop from production logs to sampled datasets to task configurations to scoring and experiments, and argue that offline-only benchmarks drift from the failures teams actually need to prevent.

Source video

Project ideas

4

RepoTraverse Bench

beyond-evals

Under a fixed token budget, agentic search using grep, find, and cat achieves higher bug-localization accuracy at lower token cost than vector top-k retrieval across multiple languages and repo sizes.

Proof of concept

Build a benchmark harness that runs both retrieval architectures against typescript-go merge PRs with 'fix' in the title and Swe-bench verified Django bug datasets, controlling for model, prompt, and token budget.

Measurement

Accuracy, tokens consumed, dollar cost, and number of search calls per successful bug localization, disaggregated by language and repository size.

Source video

TraceBridge

gatehouse

Injecting the parent span ID through environment variables into subprocess coding agents eliminates orphaned tool-call spans and produces complete parent-child trace trees without in-process instrumentation.

Proof of concept

Wrap a Claude Code style subprocess agent with an instrumentation shim that propagates trace context via environment variables, then compare trace completeness against a baseline with no propagation.

Measurement

Percentage of orphaned tool-call spans, number of attached child spans per parent trace, and debugging time to reconstruct failed runs.

Source video

EvalLoop Sampler

beyond-evals

Wiring production logs into sampled eval datasets and pre-ship experiment gates reduces recurring agent regressions and rollbacks compared with ad hoc prompt testing by PMs and engineers.

Proof of concept

Implement a pipeline that samples production logs into golden, edge-case, and failure-mode datasets, runs task configurations and scoring, and blocks deployment when experiment metrics regress.

Measurement

Production regression rate, rollback frequency, and time-to-detect regressions before and after gating.

Source video

JudgeCost Auditor

new

LLM-as-a-judge scoring introduces measurable token overhead and model-dependent scoring bias, and cross-checking judges against deterministic checks and human review reduces contamination of quality and cost metrics.

Proof of concept

Run a fixed agent-eval dataset through multiple judge models plus deterministic checks and human review, measuring token overhead attributable to each judge and scoring drift across judge models.

Measurement

Judge token overhead, score agreement with deterministic and human labels, and variance in configuration rankings caused by judge choice.

Source video

Architectural implications

3

Retrieval for code shifts from an index service to a tool-using agent loop.

Before

Embed the repository, store vectors, and answer queries with top-k similarity retrieval.

After

Expose filesystem/CLI tools (grep, find, cat) to the model and let it explore iteratively like a human, with embeddings optional rather than foundational.

Consequence

New infrastructure requirements emerge around tool sandboxing, per-step token accounting, step limits, and trajectory-level tracing, while retrieval cost becomes a function of exploration depth rather than index size.

Source video

Trace context must be propagated explicitly across process boundaries.

Before

Instrumentation inside a single process captures the full call tree automatically via in-process context.

After

Subprocess and sub-agent spans require the parent span ID to be injected via environment variables so child tool calls attach to the parent trace.

Consequence

Agent runtimes need a standard convention for out-of-band context propagation, and orphaned spans become a first-class reliability/instrumentation bug class rather than an edge case.

Source video

Eval harness is coupled to the observability pipeline.

Before

Ad hoc prompt testing by PMs and engineers, with no quantitative gate before shipping.

After

A loop where production logs are sampled into datasets, run against task configurations, scored, and iterated as experiments.

Consequence

Dataset versioning, sampling policy, scoring services, and experiment tracking become production components, and shipping decisions become gated on experiment results rather than intuition.

Source video

Tradeoffs and failure modes

4

Agentic search vs. vector search for code retrieval

Benefit

Higher bug-finding accuracy with significantly fewer tokens and lower cost on TypeScript/Go bug tasks.

Cost or risk

Multi-step exploration introduces non-determinism, tool-execution surface area (sandboxing/security), and variable latency; the summary reports this from one case study rather than broad measurement.

Agentic search achieved higher accuracy with significantly fewer tokens and lower cost on TypeScript/Go test bugs.
Open source video
Source video

LLM-as-a-judge scoring

Benefit

Provides a scalable way to determine whether model output is good or bad without exhaustive human review.

Cost or risk

Judge models can introduce bias or artificially inflate token counts, corrupting both the quality signal and the cost measurements used to compare configurations.

How do we prevent LLM-based evaluation tools from artificially inflating token counts or introducing bias?
Open source video
Source video

Optimizing models/prompts for a single desirable trait

Benefit

Improves the target behavior, e.g. helpfulness or user satisfaction.

Cost or risk

Produces unintended behavioral side effects that can force a rollback — as when a model update was reverted for becoming too sycophantic.

Over-training models to be helpful can lead to unwanted behavioral side effects.
Open source video
Source video

Shipping on qualitative judgment

Benefit

Fast iteration and low process overhead; a few prompts can look good to a PM or engineer.

Cost or risk

No quantitative regression signal, so failures surface in production and teams end up playing whack-a-mole against recurring issues.

Evals ensure that you're not playing whack-a-mole when developing your AI products.
Open source video
Source video

Open questions

4

How can hybrid approaches combine vector search and agentic search effectively?

Why unresolved

The case study is framed as a comparison, not a combination; no mechanism for mixing similarity retrieval with traversal-based exploration is described.

Research direction

Build a benchmark that varies retrieval mix (vector-only, agentic-only, vector-for-seeding plus agentic-traversal) and measure accuracy, tokens, and cost jointly on real repo bug tasks.

Source video

How do you prevent LLM-based evaluation tools from artificially inflating token counts or introducing bias?

Why unresolved

The summary raises the risk but offers no calibration method, judge-selection criterion, or cross-check against deterministic/human scoring.

Research direction

Compare judges against deterministic checks and human review on a fixed dataset, measuring token overhead attributable to the judge and scoring drift across judge models.

Source video

How should scoring be defined when the unit of work is a multi-step tool-call trajectory rather than a single model output?

Why unresolved

The eval framework is described around a dataset, a task, and a binary good/bad output judgment, while the agentic behavior demonstrated is a multi-call exploration process.

Research direction

Define trajectory-level rubrics (steps taken, whether the correct file was reached, search calls consumed) and test whether they predict end-task success better than output-only scoring.

Source video

Does agentic search's efficiency advantage hold beyond TypeScript/Go test-bug tasks and beyond Microsoft's typescript-go and Swe-bench-style datasets?

Why unresolved

Evidence comes from a narrow set of repositories and bug types, and the claim is explicitly empirical rather than established.

Research direction

Replicate the comparison across languages, repo sizes, and task types (feature work, refactors, incident response) with fixed token budgets.

Source video

Key claims

7
comparativeVerification needed

Agentic search is more accurate and more token/cost efficient than basic vector search on coding tasks.

Evidence

Agentic search achieved higher accuracy with significantly fewer tokens and lower cost on TypeScript/Go test bugs.

Question

On a fixed benchmark, what is the accuracy, token, and cost delta between agentic search and vector search across languages and repo sizes?

Source video
factualVerification needed

A vector-search run made 26 search calls without locating the bug.

Evidence

Vector search made 26 search calls on a Swe-bench run without finding the bug.

Question

Is the 26-call failure reproducible, and does it stem from a bad index, bad query formulation, or a genuine inability to follow import chains?

Source video
factualVerification needed

OpenAI reverted a model update in April 2024 because the model became too sycophantic.

Evidence

OpenAI had to revert a model update in April 2024 because the model became too sycophantic.

Question

Was the revert caused specifically by sycophancy-related over-optimization, and what eval would have caught it pre-ship?

Source video
causalVerification needed

Child processes that run as sub-agents or CLIs produce orphaned traces unless the parent span ID is passed through environment variables.

Evidence

Claude Code runs as a subprocess, resulting in missing visibility into tool calls without proper environment variable configuration.

Question

Which span-context propagation mechanisms (env vars, headers, sidecars) reliably preserve parent-child trace relationships across agent subprocess boundaries?

Source video
factualVerification not requested

An eval is composed of a dataset, a task, a scoring system, and experiment runs.

Evidence

An eval consists of a dataset, task, scoring system, and experiments.

Source video
causalVerification needed

Evals prevent recurring production failures and rollbacks in AI products.

Evidence

Evals ensure that you're not playing whack-a-mole when developing your AI products.

Question

Do teams with gating eval suites show measurably fewer production regressions and rollbacks than teams without them?

Source video
causalVerification needed

Agentic search follows code logic the way a human would, which is why it finds bugs with fewer tokens.

Evidence

Agentic search follows code logic like a human, leading to better results with fewer tokens.

Question

Is the mechanism genuinely import-chain traversal, or is it simply that more exploration steps are available within the same budget?

Source video

Connections

4