Reliability Is a Harness Problem, Not a Model Capability Problem
Teams waiting on stronger foundation models are optimizing the wrong variable: because every browser agent run starts from zero with no persistence of execution traces, harness investment (trace capture, replay, and a train-skill loop) compounds run over run while model upgrades alone reset to the same baseline.
AngleReframe the reliability debate from a model-capability question to an infrastructure question, using the doer-teacher loop as the worked example and the cost/latency/success-rate curve as the argument for where the next engineering dollar goes.
Source video ↗The Two Preconditions: Where Self-Improving Agents Work and Where They Stall
A self-improving loop is only as good as its environment, and the replayable-plus-verifiable test cleanly partitions the agent landscape: browser and coding agents with deterministic tests can hill-climb cheaply, while open-ended research agents cannot judge their own iterations and will silently optimize noise.
AngleTurn an architectural observation into a decision checklist teams can apply before building any self-improvement loop, with the open-ended-research case as the falsification test.
Source video ↗Skills Are a Package Registry: The Coming Supply Chain of Agent Reliability
Once per-site reliability knowledge is shareable as skill.md artifacts, the unit of engineering shifts from per-project scraping scripts to a versioned catalog, and the token-cost savings from suggested selectors will drive adoption faster than any correctness guarantee — importing third-party skills therefore propagates unknown, unvalidated heuristics into production agents.
AngleArgue that the registry layer needs versioning, validation against current site state, and provenance metadata before adoption outpaces the ability to audit what a skill actually does.
Source video ↗ConvergenceBench
beyond-evalsMarginal success-rate gain per doer-teacher iteration collapses sharply after a small number of cycles, and an early-stopping rule keyed to strategy.md change-magnitude will reach a production-grade skill.md in far fewer iterations than a fixed-budget loop, at no loss of final success rate.
Proof of conceptRun the same task set (e.g., flight search, restaurant booking) through the loop with per-iteration instrumentation of success rate, token cost, latency, and a structural diff size on strategy.md; A/B a fixed 20-iteration budget against a diff-magnitude early-stopping rule.
MeasurementIterations-to-graduation, final success rate, and total token cost per task; success criterion is equal-or-better success rate at a materially lower iteration count and lower cumulative cost.
Source video ↗Replay Fixture Harness
movement-labDOM snapshot fixtures plus request-level mocking can reproduce the specific failure surfaces that make browser agents unreliable (slider drag, anti-bot page) faithfully enough that a teacher agent's strategy verdict on replay matches its verdict on the live site in the majority of cases.
Proof of conceptCapture traces and DOM snapshots from real failing runs on two sites, build a replay substrate, then run the teacher agent on both live and replayed traces and compare the resulting strategy.md edits and pass/fail verdicts.
MeasurementAgreement rate between live and replayed verdicts, replay fidelity per failure-mode class, and cost reduction per evaluated iteration versus live execution.
Source video ↗Skill Drift Sentinel
gatehouseRuntime success-rate monitoring on a graduated skill.md can detect staleness before agent reliability visibly degrades in production, and auto-demotion plus re-entry into the doer-teacher loop recovers reliability faster and cheaper than full re-derivation from scratch.
Proof of conceptDeploy graduated skills against a set of sites, inject controlled UI/DOM mutations, and compare three responses: no action, full re-derivation, and drift-triggered re-entry with the existing skill.md as the teacher's starting point.
MeasurementDetection latency (runs after drift before alert), recovery time and token cost per recovery path, and post-recovery success rate relative to pre-drift baseline.
Source video ↗Skill Registry Auditor
newThird-party skills can be statically decomposed into their selector and navigation heuristics, and skills whose selectors no longer resolve against current site state measurably inflate token cost and lower success rate when imported — so validation gates plus selector-cost scoring will predict skill quality without executing them.
Proof of conceptImport a sample of catalog skills into a fixed agent, measure token cost and success rate against a selector-free baseline, and correlate those outcomes with a static audit score (selector resolution, staleness, instruction length).
MeasurementToken reduction and success-rate delta per imported skill, plus rank correlation between the static audit score and measured performance; a strong correlation validates pre-execution quality screening.
Source video ↗