AI Engineer · Published 2026-10-12

Containment Is Lying to You: Measuring AI Agents Like Employees — Druid AI

privateReview: unreviewedConfidence: high
AI agentsevaluation frameworkcontainmentgovernanceroiagent-evaluationObservabilitydriftmaturity-modelhuman-baseline
Open on YouTube ↗

Summary

Overview

  • Speaker: Michael Yang and Dan Balaceanu
  • Channel: AI Engineer
  • Main topic: Evaluating AI agent performance, ROI, and business impact using a reference framework and scoring model
  • Purpose: To provide developers and business leaders with a practical framework and maturity scale (Missing, Measured, Managed) for evaluating AI agents beyond simple containment metrics. The presenters discuss the common pitfall in evaluating AI agents solely by 'containment' percentage. Drawing an analogy to hiring interns, they introduce a comprehensive 6-pillar governance and measurement framework consisting of 26 components (covering agent mandate, platform capabilities, skills, tools, tasks, and outputs) to evaluate AI agent ROI, business impact, and performance like real employees.

Topic Map

The Problem with Containment

  • Explanation: Containment percentage (conversations handled without human intervention) is often the only metric used on dashboards, but it can be misleading when agents successfully answer routine questions but fail at high-value business intent like refinancing.
  • Key claims:
    • Containment is a vanity metric but some vendors price on it.
    • An agent with 96% containment can provide the wrong business offer compared to one with 80% containment.
  • Examples:
    • Banking customer service interns: Intern A (80% contained, flagged) vs Intern B (96% contained, on track) where Intern A correctly routes mortgage refinancing inquiries while Intern B mishandles them.
  • Terminology:
    • Containment
    • Escalation
    • Vanity metric
  • Why it matters: Relying solely on containment misleads business stakeholders about actual revenue impact and business value.

The Agent Evaluation Framework (6 Pillars)

  • Explanation: A structured reference model containing 26 components across six pillars, governed by metrics like error tolerance, risk, compliance, observability, drift, accuracy, and tone.
  • Key claims:
    • You can't claim a result you can't inspect.
    • Governance is the most heavily weighted cluster (39/100) because business outcomes depend on expected boundaries.
  • Examples:
    • Evaluating banking customer service agents across Agent Mandate, Platform Capabilities, Skills, Tools, Tasks, and Outputs.
  • Terminology:
    • Agent Mandate
    • Platform Capabilities
    • Skills
    • Tools
    • Tasks
    • Outputs
    • Governance
  • Why it matters: It replaces subjective debates about whether an agent is ready with objective measurements based on mathematical rules.

Maturity Scoring Scale: Missing, Measured, Managed

  • Explanation: A 0-1-2 maturity scale used to score each component of the framework, weighted and normalized to 100.
  • Key claims:
    • 0 = Missing (never reviewed), 1 = Measured (exists and checked, even by hand), 2 = Managed (tracked and versioned over time, continuously managed).
    • Not all measured metrics need to be automated; manual inspection is equally valid for defining risk.
  • Examples:
    • Scoring compliance against SOC 2 and back policies manually vs automatically.
  • Terminology:
    • Missing
    • Measured
    • Managed
    • Maturity scale
  • Why it matters: Provides a rigorous numerical score (e.g., Agent A scoring 83 vs Agent B scoring 45) to determine real agent readiness.

Key Points

Inspect the whole agent, not just outputs

  • Explanation: Evaluating only output success rates misses crucial governance, platform capabilities, and mandate alignment.
  • Evidence: Agent B had higher containment and more skills, but lacked explainability, drift monitoring, and compliance tracking, resulting in a lower overall score.
  • Practical implication: Establish clear boundaries and error tolerances during agent definition rather than just measuring runtime success.

The Interview Never Ends

  • Explanation: Unlike human interns who get an annual review, AI agents must be evaluated weekly against live traffic and human baselines.
  • Evidence: Agent performance drifts over time, requiring continuous instrumentation, evaluation, improvement, and re-testing.
  • Practical implication: Implement weekly automated re-runs against human baselines to catch performance drift early.

Frameworks, Models & Processes

Druid AI Agent Evaluation Framework

  • How it works: Scores 26 components across 6 pillars (Governance, Agent Mandate, Platform Capabilities, Outputs, Tools, Skills) using a 0-2 maturity scale and weights normalized to 100.
  • Components:
    • Agent Mandate
    • Platform Capabilities
    • Skills
    • Tools
    • Tasks
    • Outputs
    • Governance (Error tolerance, Risk, Compliance, Observability, Accuracy & hallucination, Tone & sentiment)
  • When to use: When deploying enterprise AI agents into production and evaluating their ROI, business impact, and performance.

Examples & Case Studies

Hiring two banking customer service interns where Intern B had 96% containment and Intern A had 80% containment.

  • Illustrates: High containment can mask missed business opportunities (like mortgage refinancing handoffs).
  • Lesson: Containment is not enough; measuring correct routing and business intent matters more.

Actionable Takeaways

  • Immediate:
    • Define agent mandates and boundaries before deployment.
    • Move beyond containment metrics to include governance and compliance.
  • Strategic:
    • Treat AI agents like employees with weekly reviews and continuous evaluation.
    • Measure what the agent frees your human experts to do to prove ROI to CFOs.
  • Questions to investigate:
    • How does your agent handle high-intent business handoffs?
    • Are your agent's governance and compliance metrics actively tracked and versioned?

Claims Worth Verifying

  • Higher education AI agents achieve 99.5% containment rate according to Druid 2026 AI Adoption Benchmark. (Benchmark statistic)

Notable Quotes

"How do we actually tell if our AI agent is effective? Which numbers matters? Which artifact should we inspect?" (at 0:43) "You can't claim a result you can't inspect." (at 10:18) "Containment is the vanity metric, but some vendors price on it." (at 5:13)

Compressed Summary

  • Containment percentage is an insufficient vanity metric for AI agent evaluation.
  • A comprehensive 6-pillar, 26-component framework evaluates agents across governance, mandate, platform, tools, tasks, and outputs.
  • Maturity scoring uses a Missing (0), Measured (1), Managed (2) scale weighted to 100.
  • AI agents require weekly evaluation against human baselines to prevent performance drift.
  • Keywords: ai agents, evaluation framework, containment, governance, roi
  • Core insight: Evaluating AI agents requires inspecting their entire configuration and runtime behavior through a structured governance framework rather than relying solely on containment metrics.

Core insights

6
Failure Modehigh noveltystrong evidence

Containment (share of conversations handled without human intervention) is a confounded metric: it can be maximized by smoothly mishandling high-value intents. An agent scoring 96% containment can produce a worse business outcome (wrong offer, missed refinancing handoff) than one scoring 80%.

Why it matters

If containment is the dashboard headline (and even a pricing basis), teams will optimize and vendors will sell against an objective that is anti-correlated with revenue-critical outcomes. Evaluation must key on correctness of routing/business intent, not on non-escalation.

Generalization

Any single scalar proxy metric that rewards sunk conversation within a boundary will be gamed at the expense of value-bearing outcomes; evaluation needs intent-conditional correctness, not a global containment rate.

Containment is the vanity metric, but some vendors price on it.
Open source video
An agent with 96% containment can provide the wrong business offer compared to one with 80% containment.
Open source video
Mental Modelmedium noveltymoderate evidence

The evaluation framework weights governance most heavily of the six pillars (39/100), placing error tolerance, risk, compliance, observability, accuracy/hallucination, and tone above raw capability artifacts like skills and tools.

Why it matters

It encodes the claim that controllability and inspectability dominate capability when agents face real business outcomes. A team's effort budget should reflect that weighting rather than piling on skills/tools.

Generalization

In production agent systems, weighting boundaries and observability above feature breadth is a defensible prior for readiness scoring.

Governance is the most heavily weighted cluster (39/100) because business outcomes depend on expected boundaries.
Open source video
Mechanismhigh noveltystrong evidence

The maturity scale separates whether a metric exists from whether it is automated: 1=Measured counts even when checked by hand, while 2=Managed requires tracking and versioning over time. Manual inspection is treated as a legitimate state, not a deficiency.

Why it matters

It gives a low-infrastructure on-ramp: a team can score progress with human-in-the-loop review before investing in full instrumentation, and the only hard requirement for the top state is versioned tracking, not automation.

Generalization

Maturity ladders can decouple 'tracked' from 'automated', letting observability debt be retired incrementally without blocking measurement.

0 = Missing (never reviewed), 1 = Measured (exists and checked, even by hand), 2 = Managed (tracked and versioned over time, continuously managed).
Open source video
Not all measured metrics need to be automated; manual inspection is equally valid for defining risk.
Open source video
Architecturemedium noveltystrong evidence

Evaluation must inspect the whole agent configuration (mandate, platform capabilities, skills, tools, tasks, outputs), not just runtime success rates. Agent B had higher containment and more skills yet scored lower because it lacked explainability, drift monitoring, and compliance tracking.

Why it matters

It locates the evidence for readiness in static config and platform scaffolding as much as in outputs, which changes what an evaluation harness must read: not only result logs but mandate definitions, tool inventories, and governance signals.

Generalization

Agent evaluation becomes a whole-system audit spanning declarative configuration and runtime traces, not a black-box output scoring pipeline.

Agent B had higher containment and more skills, but lacked explainability, drift monitoring, and compliance tracking, resulting in a lower overall score.
Open source video
You can't claim a result you can't inspect.
Open source video
Practicemedium noveltymoderate evidence

Agent evaluation is a continuous loop ('the interview never ends'): weekly automated re-runs against live traffic and human baselines to catch performance drift, rather than a one-time pre-deployment review.

Why it matters

It converts evaluation from a launch gate into an ongoing operational cadence with a specific suggested interval (weekly) and reference (human baseline), which is concrete enough to prototype and schedule.

Generalization

Deployed agents accumulate drift, so evaluation must be periodic and baseline-anchored; the human baseline serves as ground truth against which drift is detected.

AI agents require weekly evaluation against human baselines to prevent performance drift.
Open source video
Agent performance drifts over time, requiring continuous instrumentation, evaluation, improvement, and re-testing.
Open source video
Mechanismmedium noveltymoderate evidence

A weighted 0-2 score across 26 components normalized to 100 yields a single comparative readiness number between agents (e.g. 83 vs 45), replacing subjective readiness debates with arithmetic.

Why it matters

It gives a decision artifact—a numeric readiness score—that can be used to arbitrate between competing agent candidates and to make tradeoff discussions explicit through weights.

Generalization

Composite weighted maturity scoring turns qualitative readiness arguments into auditable, tunable arithmetic where weight choices are the real policy knob.

Provides a rigorous numerical score (e.g., Agent A scoring 83 vs Agent B scoring 45) to determine real agent readiness.
Open source video
Scores 26 components across 6 pillars (Governance, Agent Mandate, Platform Capabilities, Outputs, Tools, Skills) using a 0-2 maturity scale and weights normalized to 100.
Open source video

Deep dives

4

Intent-conditional correctness metrics for AI agents

Research question

How can intent-level correctness (e.g., correct routing or handoff for high-value intents) be operationalized and measured, and does it discriminate between agents that containment treats as equivalent?

Why

Because containment is a vanity metric that can be maximized by mishandling high-value intents, a replacement metric is needed to evaluate true business impact.

Containment is a vanity metric but some vendors price on it.
Open source video
An agent with 96% containment can provide the wrong business offer compared to one with 80% containment.
Open source video
Source video

Human baselines for continuous agent evaluation

Research question

How should human baselines be constructed, sampled, and kept current for weekly agent evaluation, and what are the costs and drift characteristics of such baselines?

Why

Weekly evaluation against human baselines is prescribed, but without a method, teams cannot implement it; the baseline is the ground truth for drift detection.

AI agents require weekly evaluation against human baselines to prevent performance drift.
Open source video
Agent performance drifts over time, requiring continuous instrumentation, evaluation, improvement, and re-testing.
Open source video
Source video

Optimal cadence for detecting agent performance drift

Research question

What is the earliest detectable signal of agent performance drift, and how does detection time vary with evaluation frequency (daily, weekly, monthly), and what is the cost-sensitive optimal cadence?

Why

The framework recommends weekly evaluation, but without evidence on how fast drift manifests, teams may miss drift or over-invest in evaluation.

AI agents require weekly evaluation against human baselines to prevent performance drift.
Open source video
Agent performance drifts over time, requiring continuous instrumentation, evaluation, improvement, and re-testing.
Open source video
Source video

Robustness of weighted maturity scores for agent readiness

Research question

How sensitive are agent readiness rankings to changes in the 26-component weights, and can weights be tuned to align with specific business priorities without destabilizing rankings?

Why

The framework produces a single score, but if rankings are highly weight-dependent, the score may be more of a policy artifact than an objective measure.

Governance is the most heavily weighted cluster (39/100) because business outcomes depend on expected boundaries.
Open source video
Scores 26 components across 6 pillars (Governance, Agent Mandate, Platform Capabilities, Outputs, Tools, Skills) using a 0-2 maturity scale and weights normalized to 100.
Open source video
Source video

Article ideas

4

Containment is a Vanity Metric: Why AI Agent Evaluation Must Measure Intent-Level Correctness

Containment percentage is actively harmful as a primary evaluation metric because it rewards non-escalation over correct handling of high-value intents; organizations should replace it with intent-conditional success metrics.

Angle

Critique of current industry practice, with a proposal for an alternative metric and evidence from banking examples.

Source video

Governance Over Capability: The Case for Weighting Boundaries and Observability Above Skills and Tools in Agent Readiness

The most heavily weighted pillar in agent evaluation should be governance—error tolerance, risk, compliance, observability, drift, accuracy, and tone—because controllability and inspectability dominate capability when agents face real business outcomes.

Angle

Argue against feature-count-driven agent selection, using the 39/100 governance weighting and Agent B example.

Source video

The Interview Never Ends: Why AI Agent Evaluation Must Be a Continuous, Baseline-Anchored Cadence

Agent evaluation cannot be a one-time pre-deployment gate; it must be a weekly, human-baseline-anchored loop to catch performance drift, and manual inspection is a legitimate interim state.

Angle

Operationalizing continuous evaluation with a maturity scale that decouples measurement from automation.

Source video

From Subjective Debates to Auditable Arithmetic: A Weighted Maturity Score for Agent Readiness

A weighted 0-2 score across 26 components normalized to 100 provides an auditable decision artifact that replaces subjective readiness debates, but its weights are a policy knob that must be tuned transparently.

Angle

Explain the scoring mechanism and its implications for procurement and internal readiness decisions.

Source video

Project ideas

4

Intent-Aware Containment Replacement (IACR)

beyond-evals

An intent-conditional correctness metric (correct routing/handoff rate for high-value intents) will rank agents differently than containment, and in a banking customer service simulation, an agent with high containment but low intent-correctness will be correctly identified as worse than a lower-containment agent with high intent-correctness.

Proof of concept

Build a simulated banking customer service environment with defined intents (e.g., refinancing, balance inquiry). Instrument two agents: one optimized for containment, one for correct routing. Measure both containment and intent-correctness on a test set of high-value intents.

Measurement

Rank correlation between containment and intent-correctness; delta in incorrect high-value outcomes.

Source video

Human Baseline as a Service (HBaaS)

gatehouse

A sampled human baseline (expert-scored live traffic) can be maintained at acceptable cost and used to detect weekly agent drift, and baseline drift itself is measurable and manageable.

Proof of concept

For a given domain (e.g., banking customer service), sample live agent conversations weekly, have human experts score correctness and routing. Track baseline scores over 6 weeks to measure baseline stability and labeling cost.

Measurement

Cost per labeled conversation, baseline variance, and detection rate of agent drift against baseline.

Source video

Drift Detection Cadence Optimizer

movement-lab

There exists a cost-optimal evaluation cadence (e.g., daily vs weekly vs monthly) that minimizes total cost (evaluation + missed drift damage) for a given domain, and weekly is not universally optimal.

Proof of concept

Instrument a deployed agent's performance on live traffic, compute a drift signal at daily granularity. Simulate evaluation at different cadences using historical data to estimate time-to-detection and cumulative drift damage. Optimize cadence.

Measurement

Time-to-detection at each cadence, total cost (evaluation cost + drift damage cost).

Source video

Weight Sensitivity Analyzer for Agent Readiness Scores

new

Agent readiness rankings are sensitive to the weighting of the 26 components, and small changes in weights (e.g., reducing governance from 39 to 30) can flip the ranking of two agents, indicating that weights must be set explicitly according to business priorities.

Proof of concept

Implement the 26-component maturity scoring model. For a set of 10 agent configurations with component scores, run a Monte Carlo simulation varying weights within plausible ranges. Measure ranking stability (e.g., Kendall tau) and identify weight thresholds that flip rankings.

Measurement

Ranking stability metrics (e.g., probability of rank inversion), sensitivity indices for each weight.

Source video

Architectural implications

3

Evaluation evidence lives in both static configuration and runtime behavior.

Before

Evaluation reads only output logs to compute a success or containment rate.

After

Evaluation reads declarative artifacts (mandate, tools, skills, platform capabilities) plus governance signals (drift, compliance, explainability) alongside runtime outputs.

Consequence

The evaluation harness must couple to the agent's config/registry layer, not just its telemetry pipeline, making config a versioned and inspectable first-class input.

Source video

Governance signals such as drift, compliance tracking, and explainability must exist to be scored.

Before

Governance is a launch-time review or policy document.

After

Governance metrics are continuously emitted, tracked, and versioned as part of agent runtime.

Consequence

Observability and drift detection become prerequisites for hitting the 'Managed' maturity state rather than optional add-ons.

Source video

A 'Managed' state requires tracked and versioned metrics over time.

Before

Metrics are point-in-time snapshots used for a readiness decision.

After

Metrics are stored as time series with versions, enabling drift analysis against baselines.

Consequence

Requires persistent metric history and baseline comparison infrastructure, which also enables the weekly re-run cadence the framework prescribes.

Source video

Tradeoffs and failure modes

3

Optimizing for containment

Benefit

Simple, single-number dashboard metric that some vendors price on, and can be used as a quick operational signal.

Cost or risk

Rewards non-escalation over correct handling of high-value intent, so high-containment agents can silently mishandle revenue-critical requests like refinancing.

Containment is the vanity metric, but some vendors price on it.
Open source video
Source video

Automating measurement versus manual review

Benefit

Manual inspection is valid for defining risk and avoids the upfront cost of building automated instrumentation.

Cost or risk

Manual review does not scale to weekly, continuous re-runs against live traffic and human baselines, capping progress at the 'Measured' state rather than 'Managed'.

Not all measured metrics need to be automated; manual inspection is equally valid for defining risk.
Open source video
Source video

Capability breadth versus governance maturity

Benefit

Adding skills and tools increases what the agent can do and can raise contained-conversation volume.

Cost or risk

More capabilities without explainability, drift monitoring, and compliance tracking lowers overall readiness score despite higher containment.

Agent B had higher containment and more skills, but lacked explainability, drift monitoring, and compliance tracking, resulting in a lower overall score.
Open source video
Source video

Open questions

4

How should correctness on high-intent business handoffs be weighted and measured relative to containment?

Why unresolved

The framework asserts containment is insufficient but the summary does not define a replacement metric for intent-level correctness.

Research direction

Define an intent-conditional success metric (e.g., correct routing/handoff rate) and test whether it discriminates between agents that containment treats as equivalent.

Source video

How are human baselines established and kept current for weekly agent comparison?

Why unresolved

Weekly evaluation against human baselines is prescribed, but the summary gives no method for constructing or maintaining that baseline.

Research direction

Prototype a human-baseline capture process (sampled live traffic scored by experts) and measure baseline drift and labeling cost.

Source video

What is the earliest detectable signal of agent performance drift, and does weekly cadence miss it?

Why unresolved

The summary states performance drifts and recommends weekly re-runs without evidence on how fast drift manifests or whether weekly is adequate.

Research direction

Instrument live traffic and measure time-to-detection of drift at daily/weekly/monthly evaluation frequencies to find the cost-sensitive cadence.

Source video

What is the sensitivity of the weighted maturity score to the weighting scheme, given governance alone is 39/100?

Why unresolved

The summary reports weights and an example score but provides no robustness analysis of how rankings change with different weights.

Research direction

Run a sensitivity/ablation study on the 26-component weights to test whether agent rankings are stable or weight-dependent.

Source video

Key claims

7
opinionVerification needed

Containment is a vanity metric but is used as a pricing basis by some vendors.

Evidence

Containment is the vanity metric, but some vendors price on it.

Question

Which vendors price on containment-based metrics, and does pricing correlate with containment targets in contracts?

Source video
comparativeVerification needed

An agent with 96% containment can provide the wrong business offer compared to one with 80% containment.

Evidence

An agent with 96% containment can provide the wrong business offer compared to one with 80% containment.

Question

Is there empirical evidence that higher-containment agents produce worse intent-level outcomes, or is this an illustrative example?

Source video
factualVerification needed

Governance is the most heavily weighted cluster at 39/100 because business outcomes depend on expected boundaries.

Evidence

Governance is the most heavily weighted cluster (39/100) because business outcomes depend on expected boundaries.

Question

What is the exact weight breakdown across the six pillars, and how was 39/100 derived?

Source video
opinionVerification not requested

Manual inspection is an equally valid measurement state and metrics need not be automated.

Evidence

Not all measured metrics need to be automated; manual inspection is equally valid for defining risk.

Source video
causalVerification needed

AI agents require weekly evaluation against human baselines to prevent performance drift.

Evidence

AI agents require weekly evaluation against human baselines to prevent performance drift.

Question

Does a weekly cadence measurably prevent or reduce performance drift versus longer intervals?

Source video
comparativeVerification needed

Agent B had higher containment and more skills but scored lower overall due to missing explainability, drift monitoring, and compliance tracking.

Evidence

Agent B had higher containment and more skills, but lacked explainability, drift monitoring, and compliance tracking, resulting in a lower overall score.

Question

What were the actual component scores and weights that produced Agent B's lower total?

Source video
factualVerification needed

Higher education AI agents achieve a 99.5% containment rate according to the Druid 2026 AI Adoption Benchmark.

Evidence

Higher education AI agents achieve 99.5% containment rate according to Druid 2026 AI Adoption Benchmark.

Question

Is the 99.5% figure from the Druid 2026 AI Adoption Benchmark reproducible and does it reflect correctly routed intents rather than mere non-escalation?

Source video

Connections

4