Your System Prompt Is a Model-Generation Artifact, Not an Asset
Enumerated constraints in a system prompt are rent paid on one specific model generation's weaknesses; when the model improves, those constraints stop being insurance and start being interference, so the correct engineering act at every upgrade is deletion backed by re-measurement, not preservation for backward compatibility.
AngleReframe prompt bloat as versioned technical debt with an explicit prune-and-re-measure step at every model upgrade, using the ~80% Claude Code prompt reduction as the worked example of what deletion actually looks like.
Source video ↗Benchmark the Harness, Not the Model
Capability measured in a chat interface is not predictive of capability in an execution harness, so any evaluation that fixes the interface is measuring the interface and reporting it as the model; the unit of evaluation must be the model-plus-harness pair.
AngleShow how a single interface choice (chat vs code execution with fetch and grep) flips a task from failure to trivial, and argue that benchmark reports without a harness specification are unfalsifiable.
Source video ↗The Map Is Not the Territory: Ship Elicitation Passes, Not Better Instructions
When an agent executes an abstract prompt against a real codebase, the dominant failure is not instruction-following but unstated requirements, and the fix is a mandated pass whose only job is to enumerate what the requester failed to say — a different intervention than 'ask clarifying questions'.
AngleTranslate the known/unknown matrix into concrete prompt artifacts (blindspot pass, implementation notes, deviation logging) and argue they target a failure class that generic clarification advice does not touch.
Source video ↗Browser Agent Harnesses Are Three Decisions, and You Are Making Them All Implicitly
Context storage, action-execution archetype, and auxiliary memory features are separable design axes, but most browser agent harnesses hard-code one modality stack by default; treating the axes independently lets teams choose per-task harnesses and benchmark the cross product instead of defending a single global design.
AngleDecompose the harness into its axes, show why context representation is only meaningful relative to the chosen action space, and turn the unresolved context-storage question into an explicit benchmark plan.
Source video ↗Overhang Isolator
beyond-evalsThe capability gain attributed to 'giving the model a bash tool' is mainly attributable to retrieval-and-grep access, not to code execution itself; a retrieval-only tool will recover most of the gain on parametric-recall tasks while a bare code-execution tool without data access will not.
Proof of conceptBuild a controlled task battery in three arms — (a) chat-only parametric recall, (b) chat + retrieval/grep access with no arbitrary execution, (c) chat + full script execution and fetch/grep — over datasets where the model demonstrably 'knows' the answer but cannot surface it in chat.
MeasurementTask success rate per arm, with the retrieval-only arm compared against the full-execution arm; a large gap in favor of full execution falsifies the retrieval-only hypothesis.
Source video ↗Prompt Debt Auditor
beyond-evalsFor a fixed agent task suite, a system prompt pruned by roughly 80% with minimal few-shot examples matches or beats the pre-prune prompt on a newer model generation, and any regressions are concentrated in a small, identifiable set of instruction-following cases rather than spread evenly.
Proof of conceptVersion-control two prompt tiers (full and pruned) for one agent, run both against an identical task suite on successive model generations, and produce a per-case diff report tagging each prompt clause as retained, deleted, or inverted.
MeasurementTask success delta between prompt tiers per model generation, plus the count and identity of regressed instruction-following cases; a systematic pruned-tier loss falsifies the hypothesis.
Source video ↗Harness Cross-Product Bench
movement-labContext representation and action-execution archetype interact strongly enough that no single representation dominates; the best-performing (context, action) pair changes across task families, so a per-task harness selection beats any global default.
Proof of conceptImplement a browser agent harness with pluggable context backends (full screenshots, DOM diffs, bounding-box annotations) and pluggable action interfaces, then run the cross product on one shared task suite with token and latency accounting per cell.
MeasurementTask success, tokens, and latency per matrix cell, plus whether the argmax cell differs across task families; a single cell dominating all families on all axes falsifies the hypothesis.
Source video ↗Compaction Memory-Loss Regression
gatehouseAggressive compaction causes silent, measurable memory loss on long-horizon tasks, and retention rules keyed to task-critical facts, logged deviations, and prior decisions recover most of the lost success at a fraction of the uncompacted token cost.
Proof of conceptConstruct long-horizon tasks with planted critical facts and mandated deviation logging, then run them under a sweep of compaction aggressiveness settings with and without the retention criteria enabled.
MeasurementSuccess rate and post-compaction recall of planted facts as a function of compaction aggressiveness; retention rules that do not beat naive compaction falsify the hypothesis.
Source video ↗