Your Agentic Pipeline's Only Honest Metric Is Exception Cost
Volume generated on the happy path is a vanity metric for generative production systems; the real scaling property is time-to-incorporate-perturbation, and pipelines that hide their exception rate are repackaging manual labor as automation.
AngleReframe evaluation of agentic pipelines around perturbation handling, using last-minute event schedule changes as the canonical test case, and argue that most 'scaled' claims are unmeasured on this axis.
Source video ↗Stop Asking the Model for the Format It's Bad At
Capability-boundary routing — generating in a modality the model handles well and converting with deterministic tooling — beats prompt iteration on output formats the model cannot reliably produce, and the converter is the testable component.
AngleUse the raster-then-vectorize workaround as the worked example and generalize it to a routing rule for any multi-tool agent system with a format-specific weak spot.
Source video ↗Design Systems Must Become Executable Contracts Once Agents Are the Primary Authors
When an agent writes the design file, a design system documented as guidance is worthless; it must be machine-enforceable constraint that is validated on rendered output, because a single unconstrained generation step contaminates every derivative artifact.
AngleArgue that prompt-level constraint specification and programmatic artifact linting are different guarantees, and that only the latter is auditable at hundreds of outputs.
Source video ↗The 100% Validation Claim Is Where Agentic Production Will Break
Self-reported near-perfect model validation accuracy on mass-produced artifacts is an unverified load-bearing assumption; without an adversarial defect benchmark, verifier failure ships silently at exactly the volume where it is least detectable.
AngleInterrogate the sponsor-logo and headshot validation claims as a methodology problem: sample size, defect taxonomy, occlusion and near-duplicate cases, and the escalation path that detects verifier failure.
Source video ↗Rendered-Artifact Validator Adversarial Benchmark
beyond-evalsVision-model verification of 'is this logo present and correct on the banner' falls well below reported accuracy on occluded, low-contrast, partially-overlapping, and near-duplicate-logo cases, and false negatives cluster by defect class rather than at random.
Proof of conceptGenerate synthetic banner layouts from a ground-truth sponsor set, inject a labeled defect taxonomy (missing logo, wrong logo, cropped, low contrast, occluded by another element, near-duplicate variants), run multiple vision models as verifiers, and record a confusion matrix per defect class.
MeasurementPer-defect-class false-negative and false-positive rates versus the reported near-100% figure, plus verifier confidence calibration and the fraction of defects that a human must adjudicate.
Source video ↗Format-Boundary Router
newFor vector-asset tasks, a raster-generate-then-deterministic-vectorize pipeline yields higher fidelity and editability than direct model-native SVG generation, and the gap widens as asset complexity increases.
Proof of conceptBuild a harness that takes the same asset spec down two paths — direct SVG emission versus PNG generation followed by vectorization — renders both at multiple scales, and scores them on shape fidelity, path count, and editability heuristics.
MeasurementFidelity score against a reference asset, number of vector paths/nodes, and human editability rating per path, evaluated across a set of asset complexity tiers.
Source video ↗Exception-Cost Instrumented Generator
movement-labThere is a measurable perturbation rate above which a scaled artifact pipeline's total human time exceeds manual production, and that threshold is dominated by per-exception handling cost rather than generation volume.
Proof of conceptInstrument a template-driven artifact generator to log every source-data change, its propagation into a regeneration, the human time each exception consumed, and the escalation path taken; sweep perturbation rates from low to high.
MeasurementHuman minutes per exception, exception rate per hundred artifacts, and total human minutes per hundred artifacts versus a manual baseline, locating the break-even perturbation rate.
Source video ↗Versioned Re-Render Pipeline for Design-Tool Runtime
gatehouseArtifacts generated by an agent inside a design tool can be made byte-reproducible and rollback-safe when source data and generation logic are both versioned and generation is pinned, contradicting the assumption that design-tool runtimes are inherently non-deterministic.
Proof of conceptPrototype a pipeline where event data-of-record and the generation spec live in version control, the agent renders artifacts from a pinned revision, and any artifact can be regenerated and diffed against the previously shipped version.
MeasurementRegeneration reproducibility rate (identical output at the same pinned revision), diff size per data change, and time-to-rollback after an intentionally corrupted upstream data commit.
Source video ↗