Secure by Construction: The Sandbox Is the Security Boundary, Not the Test Suite
Post-hoc unit tests applied to already-insecure generated code cannot remove the vulnerability; assurance has to be an environmental property of the runtime, so agent harness teams should spend on isolation and construction-time guarantees instead of test volume.
AngleContrast the 'write then test' gate with construction-time infrastructure guarantees, and argue the harness must run entirely inside the sandbox boundary it depends on.
Source video ↗Iteration Velocity: Token Maxing Optimizes the Wrong Variable
Throughput metrics such as tokens generated and PRs landed are actively misleading for agentic pipelines; without a direction-correctness signal, faster pipelines just make the wrong product faster.
AngleReframe throughput dashboards as proxies, then argue for a velocity metric that combines landed-change rate with rework, reverts, and spec-match signals.
Source video ↗Where Should the Authorization Boundary Live When Agents Write the Code?
For agent-generated applications, security invariants belong in the platform layer the agent writes against — database policies and auth — not in generated application code, so a wrong-but-compiling generation cannot silently violate them.
AngleTrace the shift of the trust boundary from hand-written server code to the data layer, and what that means for review, testing, and agent reachability.
Source video ↗The Coupling Penalty: What Decoupling Inference from the Harness Actually Costs
The value of per-agent cloud computers depends on an empirical question, not an ideological one: whether harness-relative model swapping costs less in latency, caching, and task success than the vendor lock-in it removes.
AngleTreat the open-agent-platform claim as a hypothesis to be benchmarked, not a settled benefit, and enumerate where tighter coupling pays for itself.
Source video ↗Sandbox Escape Range
gatehouseA per-agent sandbox that isolates the app kernel from the host kernel bounds supply chain attack impact but does not eradicate it; at least one non-kernel vector — credential access through build inputs, package resolution, or persistence across sandbox lifetimes — still escapes.
Proof of conceptBuild a red-team range that runs instrumented coding agents in per-agent sandboxes and systematically attempts host-kernel reach, credential exfiltration, and cross-sandbox persistence; record every successful escape and its class.
MeasurementNumber and class of successful escapes per attempt type, aggregated across N sandbox lifetimes, plus the specific vectors that remain outside the isolation boundary.
Source video ↗Direction-Correctness Instrument
beyond-evalsA velocity metric combining landed-change rate with a direction-correctness signal (rework rate, reverted changes, spec-match evaluation) diverges from token throughput and PR landing rates on real repositories, and predicts accepted outcomes better than those throughput proxies.
Proof of conceptInstrument an agentic delivery pipeline to log tokens, landed PRs, revert rate, rework, and a spec-match score per task; compare the composite metric against the throughput-only metrics across delivered changes.
MeasurementDivergence between composite velocity and token/PR metrics, and relative predictive power of each metric for post-hoc accepted changes.
Source video ↗Vuln-Bench Validity Check
beyond-evalsModel score on a vulnerability-detection benchmark correlates poorly with real-world confirmed-finding yield and precision on a mature codebase, and precision rather than recall dominates developer cost per confirmed finding.
Proof of conceptRun the scanner over a known-vulnerable corpus and a clean corpus, measuring precision, recall, false-positive ratio, and developer minutes per confirmed finding.
MeasurementPrecision/recall on both corpora, false-positive ratio per thousand lines scanned, and developer minutes per confirmed finding versus benchmark rank.
Source video ↗Generation-Proof Auth Floor
gatehouseAgent-generated applications built against a platform-layer authorization policy maintain access-control invariants at a higher rate than applications whose authorization lives in generated server code, particularly in the wrong-but-compiling case.
Proof of conceptGenerate the same application specification N times with platform-layer policy enforcement and N times with authorization in generated server code; run an automated authorization test suite against all artifacts.
MeasurementAuthorization violation rate per generated app, share of violations in apps that compile successfully, and variance across generations for each configuration.
Source video ↗