Your Evals Are Only as Good as the Domain Expert Who Wrote Them
Novice-authored agent evals systematically under-report deployable performance by roughly 2x, which makes eval authorship and failure adjudication a domain hiring decision, not a tooling or prompt decision; teams that staff evals with generic engineers will mis-estimate real deployment success and mis-rank their backlog.
AngleTreat the 33% vs 15% gap as an eval-validity finding rather than a model finding, and derive staffing and review-instrumentation consequences from it.
Source video ↗Put the Agent Below the Approval Gate, Not Behind It
The cheapest high-value enterprise agent is one that flags front-of-queue blocking exceptions and drafts the outbound communication while a human keeps adjudication; withholding authority removes the need for high-stakes correctness guarantees and any new governance path, and the remaining human step is a bounded, measurable cost rather than a failure of the design.
AngleFrame capability restriction as an architecture decision with a deployability payoff, and be explicit about what throughput ceiling it accepts in exchange.
Source video ↗Rank Your AI Backlog in Days of Delay, Not Items Processed
Automation volume is the wrong triage metric: a small exception population can dominate accumulated queue latency, so intervention targets should be selected by exception-caused delay and judged by a before/after cycle-time baseline, which is also the only artifact that makes deployment impact provable.
AngleContrast volume-ranked and delay-ranked backlog prioritization against a single worked queue and show what each chooses differently.
Source video ↗Shipping the Tool Is the Midpoint, Not the Job
The system is incomplete at launch; defining the deliverable to include an outcome metric in operational time saved, false-alarm tracking, and an edge-case remediation loop is what separates a measurable intervention from an unmeasured tool, and it should be written into scope rather than handed off to ops.
AngleArgue that post-launch measurement artifacts are engineering outputs, using false-alarm tracking as the concrete artifact teams skip.
Source video ↗