Your Agent Evals Measure Something That Does Not Exist
Component-level evaluation of agent systems does not under-measure capability, it measures a different object: essential properties are destroyed by disassembly, so part-level scores are upper bounds on whole-system behavior rather than estimates of it. Evaluation scope must match the scope of the behavior under question.
AngleTake the automobile-disassembly claim literally and apply it to prompt suites, tool unit tests, and per-model benchmarks; show where each sits relative to the boundary of the property it claims to report.
Source video ↗Analysis Gives You Traces; Synthesis Gives You Reasons
Modern agent observability is almost entirely analytic — spans, call graphs, task DAGs — which answers how execution happened and supports local patching. Without a synthesis representation locating the system in its containing objective, the same failures recur because the explanatory layer is absent. Observability artifacts must carry role and intent, not just order.
AngleContract the two procedures by their outputs, then specify the metadata an execution record needs to serve synthesis rather than only analysis.
Source video ↗Your Agents Are Not a Machine: Purposeful Parts Break Command-and-Control
Multi-agent systems built from models pursuing their own objectives are social systems, not machines. Command dispatch plus retries assumes functional parts, so it mispredicts behavior; the correct layer is explicit representation of each component's objective and a mechanism for resolving objective conflicts.
AngleUse the machine/organism/social-system taxonomy as a classification step performed before choosing a coordination mechanism, with the coordination choice falling out of the classification.
Source video ↗The Atomization Tax: What Step Granularity Costs Agent Systems
Atomizing workflows into minimal interchangeable tool calls buys reproducibility, parallelism, and per-step measurement, and charges for it in purpose representation. This is the same move Taylor made on manual work, with the same signature: strong step-level performance, degradation on open-ended tasks.
AngleTreat granularity as a design axis with a measurable price rather than a default, and describe the failure signature that reveals when the tax has been paid.
Source video ↗