Conventions Are an Agent Interface: Benchmark Frameworks by Tokens-per-Correct-Change, Not Human Preference
The doctrine of convention over configuration survives in the agent era for a new reason: it lowers the context budget an agent needs to act correctly. But the framework advantage cannot be claimed until convention density is separated from training-corpus representation, so teams should adopt tokens-per-correct-change as a first-class evaluation metric alongside human onboarding speed.
AngleReinterpret convention over configuration as an agent-harness cost property and propose a controlled benchmark that isolates corpus effects by testing a conventional but minimally-pretrained framework against Rails and a low-convention baseline.
Source video ↗The Ergonomics Penalty Is Now Paid by Agents: When to Choose Rust for Its Runtime, Not Its Syntax
The historical penalty on difficult systems languages assumed human authorship. When agents write and maintain the code, the runtime-efficiency axis becomes capturable, but the decision is only sound if agent competence in that language is measured longitudinally. Otherwise teams retain an artifact their humans cannot comfortably read while the maintenance burden silently transfers to a model dependency.
AngleArgue that language selection should decouple from human readability only when paired with a maintenance-competence evaluation, using the HEY rewrite as the motivating datapoint rather than the conclusion.
Source video ↗English as a Programming Language Trades Expressiveness for Nondeterminism: Why Production Pipelines Need a Constraint Layer
Treating vagueness as an asset at the intent layer moves disambiguation into the agent, where it resolves nondeterministically and silently. Production agentic pipelines therefore need a constraint layer of acceptance criteria, invariants, and tests whose job is to bound divergence, not to replace the vague intent but to make it reproducible.
AngleExamine the failure-attribution problem when the same vague prompt yields divergent artifacts, and propose measurable divergence bounds as a required artifact in the agentic development workflow.
Source video ↗Model Choice Is Deployable Configuration: Building Seams and Regression Suites for a Multi-Source Frontier
As frontier capability arrives from multiple suppliers on a short cadence punctuated by regression intervals, any agentic stack hard-wired to one model inherits concentration risk and re-validation cost. Model identity should be a configuration guarded by a per-model regression suite, so swaps are validated rather than assumed continuous.
AngleTranslate the multi-source frontier observation into an architectural pattern of abstraction seam plus per-model evals, using the trough of disillusionment as the case for why a capability peak is not a stable foundation.
Source video ↗Tokens-to-Correct-Change Benchmark for Agent-Facing Frameworks
beyond-evalsHolding task and agent constant, a codebase with higher convention density and fewer degrees of freedom reduces tokens-to-first-correct-change and increases agent success rate at N attempts, and this effect persists for a conventional but novel framework with little pretraining representation, indicating the advantage is not solely corpus-driven.
Proof of conceptImplement the same small CRUD plus background-job application in (a) Rails, (b) a deliberately conventional but novel framework with minimal corpus presence, and (c) a low-convention Node/Express baseline; run a fixed agent on identical change tasks with retrieval access; record token consumption, retrieval volume, and success rate.
MeasurementTokens-to-first-correct-change, context tokens retrieved per task, success rate at 1, 5, and 10 attempts, and the delta between the novel-conventional and Rails conditions.
Source video ↗Longitudinal Tracker for Agent-Authored Rust Services
newThe one-time runtime gains of an agent-authored Rust rewrite (99% less CPU, 95% less memory) are sustained over successive quarters, but only if human intervention frequency and rework rate stay flat; if they rise, the maintenance burden has transferred to a model dependency that is vulnerable to model swaps.
Proof of conceptInstrument a small agent-authored Rust service (or rewrite a module in Rust) with change lead time, defect density, rework rate, and human intervention tracking over at least two quarters; compare against an equivalent service in the prior language.
MeasurementChange lead time, rework rate, defect density, human intervention frequency, and CPU, memory, and host counts per quarter.
Source video ↗Spec Constraint Layer for Bounded Agent Divergence
gatehouseAttaching machine-checkable acceptance criteria and invariants to an otherwise vague natural-language specification reduces divergence across repeated agent runs without eliminating the expressiveness advantage of high-level intent.
Proof of conceptBuild a spec wrapper that accepts a vague feature request, flags under-specified intent, and requires acceptance criteria, invariants, and tests; run N independent agent runs with and without the constraint layer on the same tasks; diff the resulting artifacts for semantic divergence.
MeasurementDivergence rate across repeated runs measured by semantic diff of produced artifacts, constraint-check pass rate, and human effort to author constraints versus effort saved in review.
Source video ↗Model-Swap Regression Harness for Agentic Stacks
beyond-evalsAn agentic stack with a model-abstraction seam and a per-model regression suite can swap frontier model suppliers with bounded re-validation cost, whereas a hard-wired stack exhibits silent behavioral regressions that are only detected after deployment.
Proof of conceptBuild a thin model-provider interface with recorded agent trajectories as regression fixtures; run the same task set across at least three frontier models, including one from a non-US lab, on each release; measure pass/fail drift and re-validation effort.
MeasurementRegression-suite pass rate per model and per release, number of silent behavioral diffs detected pre-deploy, and engineer-hours to re-validate after a swap.
Source video ↗