Vanishing Gradients · Published 2026-09-02

Stop Shipping AI Nobody Can Verify with Hamel Husain

privateReview: unreviewedConfidence: high
evalsagentsverificationdata-scienceproduct-designproduct-smellprovenanceactive-learningtrace-reviewagentic-engineeringhuman-in-the-loopWebMCP
Open on YouTube ↗

Summary

Overview

  • Speaker: Hugo Bowne-Anderson & Hamel Husain
  • Channel: Vanishing Gradients
  • Main topic: AI Evals, Agentic Engineering, and Product Verification
  • Purpose: To educate engineers, product managers, and data scientists on how to design AI products that are verifiable, how to use AI agents effectively in data workflows, and how to approach evaluations with domain expertise. Hugo Bowne-Anderson and Hamel Husain discuss the state of AI evaluation, agentic engineering, and why traditional data science practices are more critical than ever in the age of AI agents. They emphasize that 'it's hard to eval' is a product smell, indicating poor product design rather than an evaluation failure, and explore how tools like notebooks, web MCP, and iterative error analysis can bridge the gap between human expertise and AI execution.

Topic Map

The State of Data Science & AI Agents

  • Explanation: Reflecting on how data science has evolved from the hype of the mid-2010s to the current era of AI agents, emphasizing that data science is not dead—AI has made it more valuable due to the increase in noisy data and unstructured signals.
  • Key claims:
    • Data science is more valuable than ever because AI generates massive amounts of unstructured data and noisy signals.
    • Coding agents are powerful, but people get caught up in job titles instead of focusing on skills.
  • Examples:
    • Transitioning from early data science hype to agentic workflows.
  • Terminology:
    • Agentic engineering
    • AI agents
    • Non-determinism
  • Why it matters: Understanding the shifting role of data scientists in an AI-dominated world.

Evals with Agents & Data Bottlenecks

  • Explanation: Examining why evaluating AI agents is difficult and how looking at data is the biggest bottleneck in machine learning evaluations.
  • Key claims:
    • Looking at data is tedious and painful, which causes people to avoid evals.
    • A hybrid approach where AI assists with active learning and surfacing high-value examples accelerates human annotation and learning rates.
  • Examples:
    • Active learning by surfacing low-confidence examples to human annotators.
  • Terminology:
    • Active learning
    • Exploration vs exploitation
    • Error discovery
  • Why it matters: Overcoming the human friction in data review and evaluation.

It's Hard to Eval is a Product Smell

  • Explanation: Analyzing Hamel Husain's blog post arguing that difficulty in evaluating an AI product indicates poor product design for verification rather than an intrinsic eval problem.
  • Key claims:
    • If a product is hard to evaluate, it is poorly designed for human verification.
    • Users need to be able to inspect assumptions, intermediate calculations, and source provenance easily.
  • Examples:
    • Data agent answering a business question with zero context vs. with assumption breakdowns and notebook links.
  • Terminology:
    • Product smell
    • Semantic layer
    • Provenance
  • Why it matters: Guides developers to design products that make verification transparent and straightforward.

Human Empathy and Domain Expertise in Verification

  • Explanation: Discussing the importance of mimicking domain experts' thought processes and fact-finding workflows when building AI tools.
  • Key claims:
    • Domain experts will not sign off on black-box outputs they cannot verify.
    • AI products must augment human experts by helping them surface contradictions, key facts, and open questions.
  • Examples:
    • Workers' comp medical report review tool highlighting contradictions and key facts.
  • Terminology:
    • Domain expert
    • Fact finding
    • Human in the loop
  • Why it matters: Ensures AI outputs are trusted and adopted in high-stakes domains.

Tools for Agentic Engineering: Runme & WebMCP

  • Explanation: Showcasing developer tooling like Runme and WebMCP for running notebooks and exposing browser-based tools directly to AI agents.
  • Key claims:
    • Notebooks and literate programming remain powerful for debugging and documenting agent workflows.
    • WebMCP allows browser-based applications to expose tools directly to AI agents without server backends.
  • Examples:
    • OpenAI developer blog post on automating repetitive work at OpenAI with Codex and Runme.
  • Terminology:
    • Runme
    • WebMCP
    • Literate programming
  • Why it matters: Provides practical workflows for managing and debugging complex agentic tasks.

Key Points

It's Hard to Eval is a Product Smell

  • Explanation: Difficulty in evaluating an AI system usually means the product was not designed to be easily verified by its users.
  • Evidence: Hamel's blog post and examples showing data agents, PE lesson plan builders, and medical report review tools.
  • Practical implication: Redesign products to surface assumptions, queries, and intermediate steps so users can verify them quickly.

Start with Data Analysis Before Evals

  • Explanation: Before jumping into rigorous automated evaluation frameworks, look at a small sample of traces and data to understand the problem.
  • Evidence: Looking at even 10 to 20 traces motivates what needs to be measured and uncovers unknown errors.
  • Practical implication: Use coding agents to quickly render interfaces for reviewing traces before building complex scoring pipelines.

Reduce Dimensionality of Complex Problems

  • Explanation: When dealing with massive exploratory state spaces or failure matrices, simplify the data to find hotspots.
  • Evidence: Transition failure matrices showing state-to-state failures in agent workflows.
  • Practical implication: Apply statistical thinking and error analysis to identify key failure modes rather than trying to evaluate everything at once.

Frameworks, Models & Processes

The Agentic Workflow Verification Loop

  • How it works: Monitor agent execution in real-time, log errors and intermediate steps, surface high-value examples for human review, and iteratively refine prompts and tools.
  • Components:
    • Real-time monitoring
    • Error surfacing
    • Active learning
    • Meta-analysis
  • When to use: When building and deploying AI agents in production environments.

Examples & Case Studies

A data agent answers 'What was net revenue for Product A last quarter?' with a single number.

  • Illustrates: Bad product design because the user cannot verify the math or data provenance.
  • Lesson: Provide assumptions, intermediate calculations, metric definitions, and links to underlying notebooks.

A physical education teacher uses an AI lesson planner that allows accepting or rejecting edits from vetted plans.

  • Illustrates: Good product design that builds trust by showing provenance and allowing incremental editing.
  • Lesson: Design workflows that align with how human experts actually work and verify information.

A workers' comp medical report tool extracts records and highlights contradictions, key facts, and open questions.

  • Illustrates: Augmenting domain experts by helping them do their job rather than replacing them with a black box.
  • Lesson: Focus on improving human recall and reducing the friction of verification in high-stakes domains.

Actionable Takeaways

  • Immediate:
    • Review 10 to 20 raw traces of your AI application before writing evaluation code.
    • Expose intermediate steps and assumptions in your AI product's user interface.
  • Strategic:
    • Treat evaluation as an inherent part of product design, not an afterthought.
    • Embrace data analysis and statistical thinking as core competencies for working with AI agents.
  • Questions to investigate:
    • How can we design user interfaces that make verification faster and more intuitive?
    • What are the key latency and failure signals in our agentic workflows?

Claims Worth Verifying

  • Difficulty in evaluating an AI product is primarily a product design flaw rather than an evaluation algorithm limitation. (theoretical)
  • WebMCP allows browser-based applications to expose tools directly to AI agents without a server backend. (technical)

Notable Quotes

"it's hard to eval is actually a product smell" "Science is something you do, not something you are." "Search tends to be the Achilles heel of a lot of AI products."

Compressed Summary

  • Difficulty in evaluating AI products indicates poor product design for verification.
  • AI products must expose assumptions, intermediate steps, and source provenance.
  • Start evals by looking at raw data traces to understand failure modes.
  • Data science skills in statistical thinking and error analysis are vital for managing noisy AI outputs.
  • Keywords: evals, agents, verification, data-science, product-design
  • Core insight: Evaluating AI is hard because products are poorly designed for transparency; making systems verifiable through intuitive UI and provenance turns AI from a black box into a trusted collaborator.

Core insights

7
Architecturehigh noveltystrong evidence

Difficulty in evaluating an AI product is often a product smell: the product was not designed for human verification. Make assumptions, intermediate calculations, and source provenance explicit so users can check the system.

Why it matters

It shifts the burden from building post-hoc eval harnesses to embedding verification affordances into the product surface. Without this, eval cost remains high and user trust fails regardless of eval metrics.

Generalization

Any AI output consumed by humans should expose enough of its derivation (assumptions, intermediate artifacts, code/notebook links) that a domain user can spot-check it.

If a product is hard to evaluate, it is poorly designed for human verification.
Open source video
Users need to be able to inspect assumptions, intermediate calculations, and source provenance easily.
Open source video
Practicemedium noveltystrong evidence

Start product and eval work by manually examining 10-20 raw traces before building automated evaluation frameworks. This reveals actual error modes and what to measure.

Why it matters

Automated eval frameworks built too early optimize for assumed failure categories and miss unknown errors; small trace inspection grounds the eval in observed reality.

Generalization

Every new agent or LLM feature should begin with a small manual trace review that shapes both product changes and eval metrics.

Looking at even 10 to 20 traces motivates what needs to be measured and uncovers unknown errors.
Open source video
Before jumping into rigorous automated evaluation frameworks, look at a small sample of traces and data to understand the problem.
Open source video
Mechanismmedium noveltymoderate evidence

Use coding agents to build custom trace-review interfaces quickly; reducing the friction of 'looking at data' is the key to making evals happen at all.

Why it matters

Manual inspection is the bottleneck in eval work because it is tedious and painful. Agents can render purpose-built UIs for trace review, making it scalable enough to actually be done.

Generalization

The same AI agents being evaluated can be used to build the observability and eval tooling that inspects their own behavior.

Use coding agents to quickly render interfaces for reviewing traces before building complex scoring pipelines.
Open source video
Looking at data is tedious and painful, which causes people to avoid evals.
Open source video
Mental Modelmedium noveltystrong evidence

For high-stakes domains, AI products must imitate the domain expert's fact-finding workflow rather than return black-box conclusions. They should surface contradictions, key facts, and open questions, because experts will not sign off on outputs they cannot verify.

Why it matters

Adoption in regulated or expert domains depends on matching how professionals actually gather and verify evidence, not on producing a confident final answer.

Generalization

Expert-facing AI should be designed as an evidence-dossier generator (facts + contradictions + open items) instead of an automated answer machine.

Domain experts will not sign off on black-box outputs they cannot verify.
Open source video
AI products must augment human experts by helping them surface contradictions, key facts, and open questions.
Open source video
Architecturehigh noveltymoderate evidence

Emerging agent infrastructure is making existing software surfaces agent-native: WebMCP lets browser-based applications expose tools directly to agents without server backends, and notebook/literate-programming runtimes (Runme) provide debuggable, documented agent workflows.

Why it matters

This changes where agent-tooling responsibility sits. Instead of rebuilding server APIs for every app, the browser becomes the tool layer, and notebooks become reproducible agent transcripts for debugging and eval.

Generalization

As agents become users of the same software as humans, UI components and interactive notebooks become a new class of agent tool interface.

WebMCP allows browser-based applications to expose tools directly to AI agents without server backends.
Open source video
Notebooks and literate programming remain powerful for debugging and documenting agent workflows.
Open source video
Mechanismmedium noveltymoderate evidence

Eval with human annotators should be organized as active learning: AI surfaces low-confidence or high-value examples for human review, accelerating learning and annotation rates.

Why it matters

Since trace inspection is the data bottleneck, naive random sampling wastes human effort. Intelligent surfacing concentrates human attention on examples that most change the model or eval understanding.

Generalization

Any human-in-the-loop eval loop can benefit from exploration-vs-exploitation sampling: show the model examples it is least certain about and use the resulting labels to refine prompts, tools, or the eval set itself.

A hybrid approach where AI assists with active learning and surfacing high-value examples accelerates human annotation and learning rates.
Open source video
Active learning by surfacing low-confidence examples to human annotators.
Open source video
Practicemedium noveltymoderate evidence

Agent failures are concentrated in state-to-state transitions; reduce dimensionality by analyzing transition failure matrices and hotspots instead of trying to evaluate every possible path uniformly.

Why it matters

Agent workflows have huge exploratory state spaces. Trying to evaluate everything at once is intractable and obscures the small number of systemic failure modes.

Generalization

For large agent workflows, construct a state-transition or error matrix, find cells where failures cluster, and focus eval, tracing, and fixes on those hotspots.

Transition failure matrices showing state-to-state failures in agent workflows.
Open source video
Apply statistical thinking and error analysis to identify key failure modes rather than trying to evaluate everything at once.
Open source video

Deep dives

4

Verification affordance design for AI outputs: from 'answer-only' to evidence receipts

Research question

What set of intermediate artifacts (assumptions, calculations, source provenance, and notebook links) should an AI answer expose so a domain expert can verify it without being overwhelmed, and how should that set vary with decision risk?

Why

This is the core product-smell idea: if AI products expose verifiable internals at the UX level, eval becomes a design property rather than a separate expensive test.

Users need to be able to inspect assumptions, intermediate calculations, and source provenance easily.
Open source video
If a product is hard to evaluate, it is poorly designed for human verification.
Open source video
Source video

The empirical basis of 10-20 trace reviews in eval discovery

Research question

What is the marginal value of manually inspecting successive raw traces from a new agent workflow—when does novel failure discovery flatten, and does 10-20 traces generalize across task complexity?

Why

The summary recommends trace-level review as the correct starting point, but there is no empirical curve to tell teams when to stop manually reviewing and start measuring; without that curve, teams either under-inspect or automate prematurely.

Looking at even 10 to 20 traces motivates what needs to be measured and uncovers unknown errors.
Open source video
Before jumping into rigorous automated evaluation frameworks, look at a small sample of traces and data to understand the problem.
Open source video
Source video

State-transition failure matrices as first-class instrumentation for agent evals

Research question

How can agent telemetry capture state-to-state transitions, and what methods turn those records into failure hotspot matrices that change where to focus testing and repair?

Why

Transition failures account for much of agent error but are usually evaluated uniformly; a matrix of transitions can expose dimensionality and focus human/logical effort on hotspots.

Transition failure matrices showing state-to-state failures in agent workflows.
Open source video
Apply statistical thinking and error analysis to identify key failure modes rather than trying to evaluate everything at once.
Open source video
Source video

Explore-exploit sampling for human-in-the-loop agent evals

Research question

When selecting traces for human review, how should evaluation sets trade off low-confidence examples (exploitation of known unknowns) with random ones (exploration of unknown unknowns) in terms of downstream eval quality and annotation cost?

Why

Looking at data is the human bottleneck, and active learning can accelerate it, but current advice does not quantify explore/exploit losses; that gap stalls eval process design.

A hybrid approach where AI assists with active learning and surfacing high-value examples accelerates human annotation and learning rates.
Open source video
Active learning by surfacing low-confidence examples to human annotators.
Open source video
Source video

Article ideas

4

'I can't eval it' is a product smell: how to make AI explainable enough to test

If an AI feature is difficult to evaluate, it is not an evaluation problem—it is a product design failure: once a product exposes its assumptions, intermediate steps and provenance, users can verify it and eval becomes tractable.

Angle

Contrarian engineering argument: instead of investing in more eval harnesses, invest in interfaces that expose verifiable artifacts.

Source video

The 20-trace ritual: why manual inspection is the foundation of agent evaluation

Automated eval frameworks that are built before any manual trace inspection optimize for the wrong categories; a mandatory 10-20 trace review is the only way to choose metrics that actually track user-facing failures.

Angle

Process and culture for eval teams, drawing on active-learning/data-bottleneck arguments.

Source video

Your AI assistant should produce an evidence dossier, not an answer

For high-stakes domains, expert sign-off requires surfacing contradictions, key facts, and open questions in an implicit fact-finding workflow; black-box outputs will never be approved, no matter how good the eval metrics look.

Angle

Design model for safe adoption in regulated/professional domains.

Source video

Let agents build the test harnesses agents deserve

Coding agents excel at building trace-review UIs, and using them to reduce the tedium of manual data inspection is the fastest path to evals that people actually run; evaluators should treat agents as their first eval infrastructure.

Angle

Counterpoint to fear of agents: they can be used to build their own eval tooling.

Source video

Project ideas

4

VeriTracer: notebook-dossier answers for data agents

movement-lab

When users are shown an AI data answer with assumption breakdown, intermediate calculations, and a notebook link, time-to-verification and correct acceptance rate improves by at least 50% over a zero-context answer-only UI.

Proof of concept

Enhance a simple Q/A data agent to emit a Runme-compatible notebook (assumptions, steps, source links) as its output; A/B test with domain analysts on real business questions.

Measurement

Time-to-verification (seconds), correct acceptance rate, false-acceptance rate, and number of user clicks on source artifacts.

Source video

AgentFailMapper

gatehouse

Recording agent state-to-state transitions and computing a failure matrix will surface the highest-frequency failure transition with 3x fewer logs examined than random manual trace search.

Proof of concept

Build a thin tracing layer around an agent framework (e.g., LangGraph) that logs states/transitions, aggregates failure rates into matrices, and produces ranked hotspot lists.

Measurement

Precision/recall of detected hotspot transition vs. ground-truth from manual log review; reduction in failure rate after focused fix.

Source video

EvalLens Active-learning sampler

beyond-evals

Low-confidence surfacing of agent traces will identify novel error categories at least 1.5x faster per annotation hour than random sampling during initial eval-set construction.

Proof of concept

Implement a small tool that scores raw agent traces by confidence, selects top-N for human review, and periodically feeds findings into a structured eval set; run on a realistic batch of traces.

Measurement

Novel error categories discovered per hour, annotation time, learner accuracy before/after labels, error coverage.

Source video

Browser-API parity testbed for WebMCP

new

A WebMCP browser-native tool exposure for a standard app can match a server-backend API on task success rate while cutting integration boilerplate in half, though with an additional measurable per-action failure rate due to browser context.

Proof of concept

Stand up one common web app (e.g., a collaborative spreadsheet UI) and create two tool endpoints (WebMCP and HTTP server API). Run same set of agent tasks against both and compare.

Measurement

Task success rate, lines of integration code/time, per-action failure rate, and browser-layer exception rate.

Source video

Architectural implications

4

Typical agent APIs return a single final answer, leaving no way for users to inspect assumptions, calculations, or source provenance.

Before

Agent output is a final answer string; verification is a separate offline evaluation step.

After

Agent outputs are structured as assumption statements, intermediate steps, metric definitions, and links to underlying notebooks, so users can verify in context.

Consequence

Verification becomes a product property rather than a post-hoc eval concern, and trust is established at the point of interaction.

Source video

The product team finds evals hard because nobody wants to look at raw traces; they lack a usable interface for trace review.

Before

Developers avoid trace inspection and instead jump to automated scoring pipelines.

After

Development loop includes a trace-review panel, generated on demand by coding agents, integrated into the iteration workflow.

Consequence

Eval iteration speed increases because people actually look at failures rather than trusting aggregate metrics blindly.

Source video

Exposing browser apps and notebooks to agents, via WebMCP and Runme, makes new producer/consumer boundaries possible for agent tools.

Before

Agents require dedicated, often server-backed APIs for every tool or data source.

After

Browser-based applications can expose functions directly to agents, and notebook runtimes can execute and document agent workflows.

Consequence

The cost of integrating agents with existing software drops, and agent transcripts can be versioned and inspected as literate programs.

Source video

Human review is treated as a final approval step, not as part of an active-learning loop that improves the agent.

Before

Annotations from production review are collected manually and do not inform model or prompt refinement.

After

A hybrid loop surfaces low-confidence and high-value traces to humans; labels are fed back into the system via active learning.

Consequence

Every human review hour contributes to eval discovery and agent improvement instead of being firewalled at approval.

Source video

Tradeoffs and failure modes

4

Manual trace review vs. automated eval frameworks

Benefit

Manual review of 10-20 traces uncovers unknown errors and motivates what to measure.

Cost or risk

Looking at data is tedious and painful, so teams routinely avoid it and instead automate prematurely, measuring dimensions that don't matter.

Looking at data is tedious and painful, which causes people to avoid evals.
Open source video
Source video

Verification by provenance and rich output vs. simple answer UX

Benefit

Users can inspect assumptions, intermediate calculations, and notebook links, which makes trust possible.

Cost or risk

A zero-context single-number answer is easier for users to consume, but it cannot be verified and fails in high-stakes decisions.

Data agent answering a business question with zero context vs. with assumption breakdowns and notebook links.
Open source video
Source video

Active learning surfacing vs. uniform sampling in eval

Benefit

Active learning concentrates human annotation on low-confidence examples and accelerates learning rates.

Cost or risk

Pure low-confidence sampling can bias understanding of true error rates; exploration-vs-exploitation tradeoffs must be managed.

Active learning by surfacing low-confidence examples to human annotators.
Open source video
Source video

Reducing dimensionality vs. exhaustive evaluation

Benefit

Failure matrices and statistical thinking identify high-yield hotspots, making eval actionable.

Cost or risk

Aggressively simplifying the state space may hide important but infrequent failure modes outside the identified hotspots.

Apply statistical thinking and error analysis to identify key failure modes rather than trying to evaluate everything at once.
Open source video
Source video

Open questions

4

How should product designers choose which intermediate assumptions and calculations to expose, without overwhelming users or leaking chain-of-thought?

Why unresolved

The summary states that users need to inspect assumptions and provenance, but it does not give criteria for where transparency becomes noise or a usability burden.

Research direction

UX studies on verification affordances and the semantics of a well-designed 'receipt' for AI outputs.

Source video

What principled method scales from the initial 10-20 trace review to a continuously maintained, statistically meaningful eval set over time?

Why unresolved

Starting small is recommended, but the summary does not describe how to grow or version eval data as prompts, tools, and failure modes evolve.

Research direction

Trace sampling policies, active learning for eval-set curation, and drift detection in agent behavior.

Source video

How do transition failure matrices get built and monitored in real time for production agents?

Why unresolved

The summary gives the matrix as a useful diagnostic, but not the telemetry or state-machine abstractions needed to produce it automatically.

Research direction

Instrumentation frameworks that record agent state transitions and render hotspot matrices for ongoing analysis.

Source video

Will WebMCP-style browser-native tool exposure become the dominant agent-tool pattern, and what are its security and governance limits?

Why unresolved

The capability is stated, but the summary offers no production patterns, constraints, or permissioning model.

Research direction

Prototype WebMCP agents, measure reliability versus server-backed tool APIs, and define scoped-permission models.

Source video

Key claims

7
opinionVerification needed

Difficulty in evaluating an AI system usually means the product was not designed to be easily verified by its users.

Evidence

If a product is hard to evaluate, it is poorly designed for human verification.

Question

Can two otherwise equal systems—one exposing provenance and one not—show measurable differences in eval difficulty and user trust?

Source video
opinionVerification needed

Reviewing even 10-20 traces motivates what needs to be measured and uncovers unknown errors.

Evidence

Looking at even 10 to 20 traces motivates what needs to be measured and uncovers unknown errors.

Question

What is the marginal value curve of adding traces to eval discovery? At what point do new insights flatten?

Source video
causalVerification needed

AI-assisted active learning accelerates human annotation and learning rates.

Evidence

A hybrid approach where AI assists with active learning and surfacing high-value examples accelerates human annotation and learning rates.

Question

Does low-confidence-based surfacing outperform random sampling in a measured annotation study?

Source video
factualVerification needed

WebMCP allows browser-based applications to expose tools directly to AI agents without server backends.

Evidence

WebMCP allows browser-based applications to expose tools directly to AI agents without server backends.

Question

Can a production-grade WebMCP agent integration run without additional infrastructure, including in restricted network environments?

Source video
opinionVerification needed

Notebooks and literate programming remain powerful for debugging and documenting agent workflows.

Evidence

Notebooks and literate programming remain powerful for debugging and documenting agent workflows.

Question

Are notebook transcripts as effective for agent failure analysis as structured trace logs?

Source video
predictionVerification needed

Domain experts will not sign off on black-box outputs they cannot verify.

Evidence

Domain experts will not sign off on black-box outputs they cannot verify.

Question

In real deployments, does adding externally verifiable artifacts change expert sign-off rates?

Source video
causalVerification needed

Data science has become more valuable because AI generates massive amounts of unstructured data and noisy signals.

Evidence

Data science is more valuable than ever because AI generates massive amounts of unstructured data and noisy signals.

Question

Can this be quantified in terms of job roles, productivity, or demand signals?

Source video

Connections

6