Your Agent Benchmark Is Measuring Parts of a System That Doesn't Exist
Component-level agent and tool benchmarks are invalid by construction for capability claims, because the properties being claimed only exist in composition; evaluations must move to the interaction level, and any component score should be reported as relative to a specific composition rather than as an intrinsic property.
AngleTake the systems claim that disassembly destroys essential properties in both the whole and the parts and turn it into an engineering indictment of the current agent-eval stack, with concrete guidance on interaction ablations.
Source video ↗Dissolve, Don't Debug: Rerouting Recurring Agent Failures to the Architecture Layer
Most agent reliability work is symptom patching at the model or prompt layer, but the highest-leverage move is redesigning the task or interface so the failure class cannot occur; teams should triage failures by recurrence and structural eliminability and route dissolution-class failures upward, accepting that the fix is a product decision, not a model decision.
AngleUse the elevator-and-mirrors case as the canonical pattern: change the environment and the incentive rather than the mechanism being complained about.
Source video ↗Data Is Cheap, Understanding Is Expensive: Budgeting Context by Value Tier
Agent context is allocated by availability when it should be allocated by value: raw retrieved data is the cheapest and least valuable tier, so context and memory pipelines should be evaluated on the tier of content they surface and should spend budget on distillation before spending it on recall.
AngleInvert the standard retrieval metric: show why a smaller, higher-tier context can beat a larger low-tier one, and reframe summarization and distillation layers as first-class architecture rather than cost-saving afterthoughts.
Source video ↗Prompting Is Teaching, and Teaching Obstructs Learning
Dense instruction is fast to author but can crowd out the experiential acquisition it is meant to produce; for open-ended capability building, feedback-rich environments and self-directed iteration should carry the load, with instructions used as scaffolding rather than prescription.
AngleReframe prompt engineering as an instruction-density tradeoff, and argue that agent improvement budgets should shift toward environment and feedback-loop design.
Source video ↗