Your Self-Improving Agent Is Grading Its Own Homework
Recursive self-improvement is not a model capability but an evaluation-architecture problem: until the scorer is an isolated, versioned, tamper-resistant boundary that the optimizer cannot write to or read, every reported gain in the loop is a measurement artifact, and the correct engineering response is to treat the eval harness, not the generator, as the critical path.
AngleTake the loop's own component list seriously and show that the ops artifacts it implies — sandboxed execution, versioned prompts and code, rollback on regression, an untouchable scoring path — are the actual deliverable, then contrast that with funding-announcement evidence of the capability timeline.
Source video ↗A 60-Point Benchmark Jump Is a Marketing Document Until You Ablate the Harness
Single-benchmark deltas reported at release time are commercial artifacts shaped by launch strategy and scaffold tuning, so they should never be the input to a production routing decision; teams that route on public leaderboard movement are systematically overpaying for capability they cannot measurably use.
AngleWalk through the ablation that would be needed to separate weights from harness — frozen scaffold across versions, same model across scaffolds, variance decomposition — and specify what a private per-task routing eval must contain to be trustworthy.
Source video ↗48% Is Not a Passing Grade
Human-indistinguishability claims are unfalsifiable without a published chance baseline, rater protocol, and confidence interval, and reporting a below-chance-baseline number as a first-ever pass is a template example of how eval design failure becomes a capability narrative that then drives security-relevant product decisions.
AngleUse the gap between the stated result and the stated pass criterion to argue for mandatory baseline disclosure as a publication norm for any 'model beats humans' claim, including forced-choice human-human positive controls.
Source video ↗Capital Is Evidence of Consensus, Not of Mechanism
Large raises for recursively self-improving superintelligence and headline timelines like a century of breakthroughs in a decade are evidence that investors agree on a direction, not that the loop closes; engineers should adopt the loop pattern on its own architectural merits while treating the capability timeline as an unfalsifiable opinion.
AngleSeparate the falsifiable engineering claim (the loop pattern and its eval obligations) from the timeline claim, and show what measured result would move the timeline from opinion to demonstrated capability.
Source video ↗Scorer Isolation Harness
gatehouseA generate-implement-evaluate-rewrite loop with the scorer running in a separate process, on held-out tasks the generator cannot read, will improve monotonically on task performance while a co-located scorer version of the same loop will improve on the score metric while task performance plateaus or regresses.
Proof of conceptBuild two minimal loops over an identical code-synthesis task set differing only in scorer isolation, run matched iteration budgets, and log both the reported score and an independent external task metric at every iteration.
MeasurementDivergence between internal score and independent external task metric per iteration, plus rollback frequency on score regression, across isolated versus co-located scorer configurations.
Source video ↗Delta Attribution Bench
beyond-evalsFor a given single-benchmark release delta, variance attributable to scaffold and harness configuration is comparable to or larger than variance attributable to model weights, so a fixed private task-distribution eval will rank model versions differently than the public benchmark does.
Proof of conceptRun a frozen scaffold across two model versions and two scaffolds across one model version on a tool-use benchmark, then compare the resulting rankings against the team's own private task-distribution eval.
MeasurementVariance decomposition of benchmark scores by scaffold and by weights, plus rank correlation between public benchmark ordering and private eval ordering.
Source video ↗Baseline Disclosure Linter
beyond-evalsEvaluations reporting human-indistinguishability or 'beats humans' results will fail a mechanical baseline-completeness check in a majority of published cases, and requiring a declared chance level, rater count, and positive control changes the reported conclusion for a meaningful fraction of them.
Proof of conceptEncode the required fields as a schema check, run it over a collected set of human-indistinguishability claims, then recompute each result against its declared chance level and human-human control where available.
MeasurementFraction of claims missing baseline or protocol fields, and fraction whose pass/fail conclusion flips once the chance level and confidence interval are applied.
Source video ↗Actuation Gate Pilot
movement-labIn a two-stage loop where candidates must clear simulation scoring before reaching a physical execution stage, per-stage failure rates compound multiplicatively, so a pre-screening threshold tuned on simulation-to-physical concordance reduces end-to-end cost per validated hypothesis more than increasing physical stage throughput does.
Proof of conceptInstrument a pilot pipeline with simulation gating and a matched ungated pipeline, recording per-stage failure, latency, and cost, and computing how simulation error propagates into hypothesis-validation error.
MeasurementCost per validated hypothesis and end-to-end validation error rate for gated versus ungated pipelines at matched candidate counts.
Source video ↗