Your 99% Agent Is a 36% Agent
Per-step accuracy is a vanity metric: because agent value is terminal and failure costs are cumulative, the only honest production KPI is artifact yield per unit of expected spend, and reliability must be budgeted across a horizon rather than improved locally. Teams that report step-level accuracy are reporting the one number that cannot predict production outcomes.
AngleTake the 0.99^100 figure and push it into an engineering planning rule: given a target success rate, derive the maximum allowable horizon and required per-step reliability, then show how that reshuffles where engineering effort goes (horizon shortening beats prompt tuning).
Source video ↗Delete the Model Wherever the Answer Is Unique
The highest-leverage reliability work in an agent system is subtractive: every step with a single verifiable correct answer — login, artifact verification, download checking — should be removed from the model's responsibility and executed deterministically, because a probability distribution is the wrong tool for a task with one right answer. Tool-wrapping is not a convenience pattern; it is how you shrink the exponent on compounding failure.
AngleReframe deterministic tooling as an explicit architectural boundary decision, with a step-audit checklist (ambiguous vs. non-ambiguous) rather than a library recommendation.
Source video ↗Demo Success Is a Volume Artifact
The demo-to-production gap is not a capability gap, it is a scale and drift gap: one watched run against a stable site is a fundamentally different statistical object than thousands of unattended runs against shifting sites and models having off days. Treating a demo as evidence of reliability is a category error, and the fix is operational (retries, artifact verification, drift watch) rather than model-side.
AngleArgue that demo success predicts nothing about production and specify what evidence would — artifact-verified yield curves over N unattended runs with drift events included.
Source video ↗Compile Your Agent: Graduating Converged Paths into Skills
Long-lived agents should get cheaper and more reliable over time by compiling observed convergent behavior into deterministic artifacts, turning runtime re-planning into a fixed critical path. The interesting engineering problem is not whether an agent can plan, but whether it can recognize when planning is no longer necessary — and that requires a promotion gate, not a prompt.
AngleFrame the runtime-to-artifact promotion pipeline (observe → converge → gate → freeze → watch for drift) and argue it is the missing maintenance model for agent deployments.
Source video ↗Trajectory Budget Planner
newFor any workflow with measured per-step reliability p and per-step cost c, there exists a retry count r* that minimizes expected cost per successful artifact, and r* is strictly smaller than the retry count that maximizes raw success rate; additionally, beyond a computable horizon length, no realistic per-step reliability improvement is economically rational for a fixed cost-per-step.
Proof of conceptInstrument 3 browser workflows of 20, 50, and 100 steps, measure per-step success and cost, then sweep retries 1-8 in simulation and in live runs to produce expected-cost-per-artifact curves against a target success threshold.
MeasurementExpected cost per successful artifact as a function of retry count and horizon length; the horizon at which cost per success diverges; the gap between cost-optimal and success-optimal retry counts under varying step cost.
Source video ↗Artifact Oracle
gatehouseGrounding success in a machine-verifiable external artifact (confirmation email, order ID, database record) instead of model self-report reveals a substantial silent-failure rate, and deterministic verifiers can cover the majority of common workflows without any model judgment in the verification step.
Proof of conceptBuild deterministic verifiers for three workflow types (bill-pay confirmation email, order receipt ID, DB record write) and wire them as the termination signal in an unattended run loop; compare agent-declared success against artifact-verified success across the same runs.
MeasurementDelta between self-reported success rate and artifact-verified success rate; false-positive completion rate; verification latency and cost per run; fraction of workflows coverable by deterministic verifiers alone.
Source video ↗Wrap-vs-Step A/B
beyond-evalsWrapping brittle multi-step browser sequences into single deterministic tools reduces end-to-end artifact success rate failure on 50+ step workflows by more than the failure rate introduced by the wrapper artifacts themselves — i.e., failure is genuinely reduced, not relocated to the tool layer.
Proof of conceptImplement the same 3 production-representative workflows twice — raw step-level agent and tool-wrapped agent, same model, same sites — and run N≥200 unattended trials each with artifact verification and per-step decision-point logging.
MeasurementEnd-to-end artifact success rate, model decision points per run, cost per success, and attribution of failures to model decisions vs. tool-layer artifacts (equal-failure-rate outcome falsifies the hypothesis).
Source video ↗Skill Graduation Pipeline
movement-labA convergence detector combining action-sequence similarity and repeated artifact identity can auto-promote stable trajectories into deterministic replayable skills such that cost per success falls measurably without dropping success rate below the pre-promotion baseline.
Proof of conceptLog action sequences and artifacts across repeated runs of a fixed workflow, cluster trajectories, auto-promote clusters crossing a stability threshold into replayable deterministic skills, and keep a drift watch that re-runs the live path against the frozen skill.
MeasurementRuns-to-promotion, cost and step-count per success before vs. after promotion, success-rate regression against baseline, and mean time-to-detect when a promoted skill stops matching the live site.
Source video ↗