The Meditation Cushion Is a Harness: Why Agent Evaluation Should Happen Inside Every Workflow
Evaluation that is truly useful cannot sit apart from deployment, because an agent only becomes trustworthy by practicing every action instrumented, not by passing periodic offline tests.
AngleProcess/practice criticism of agent evaluation, using Zen work practice as the lens.
Source video ↗Difficulty Is a Training Signal: Designing Adversarial Scarcity for Autonomous Agents
Agents that see only friction-free operation develop brittle confidence; hardship should be treated as intentional curriculum, not as an anomaly to exclude.
AngleReliability engineering and adversarial curriculum design.
Source video ↗Foundation, Guardrails, Practice, Release: The Fourfold Path to Deployable Agents
Autonomy should never precede a method layer and must include an explicit unlearning phase; the fourfold progression gives deployment a sequence that today's agent stacks mostly miss.
AngleArchitecture sequencing for agent deployments.
Source video ↗Hermit Mode: Why Long-Running Agents Need Architectural Solitude
Context isolation and gated interrupts are reliability features—not social constraints—because ambient noise degrades deep reasoning faster than most teams measure.
AngleContext hygiene in persistent agents.
Source video ↗Every-Action Calibration Probe
beyond-evalsAdding a lightweight calibration probe to each production tool call of a deployed agent reduces the divergence between offline eval scores and live task success by at least 20% after 30 days, compared to an agent retrained only on periodic offline evals.
Proof of conceptInstrument a customer-support agent to emit a compact (state, action, outcome) signal per tool call; sample 1% of traces for human-in-the-loop ratings; periodically fine-tune a small adapter on these traces while continuing to use the base policy.
MeasurementCalibration divergence = |offline eval success - production task success|, measured before and after the intervention; plus raw production task success rate.
Source video ↗Scarcity Simulator
newAgents fine-tuned for 25% of episodes under injected tool failures, rate limits, and truncated context will outperform a smooth-only baseline by 30% in a deliberately degraded evaluation set, while staying within 10% of the baseline's performance on a normal evaluation set.
Proof of conceptRun an agentic coding benchmark in two regimes—full-resource and scarce-resource—and use preference optimization on trajectories from the scarce environment, then cross-evaluate the resulting policy on both regimes.
MeasurementTask completion rate in the scarred environment and in the normal environment.
Source video ↗Quiet Mode Context Router
newGating non-urgent ambient input (notifications, social feeds, low-priority email) behind a sampled context channel improves completion rate on long-horizon reasoning tasks by 20% or more, while allowing recall of 95% of high-priority interrupts.
Proof of conceptImplement a context router with two channels: a trusted high-priority stream and a sampled low-priority stream; run long-horizon document-analysis tasks with synthetic interruptions and ambient messages injected.
MeasurementTask goal completion rate, interrupt recall, context-token usage.
Source video ↗Guardrail Gate
gatehouseA deterministic method layer that checks action preconditions before tool execution will reduce irreversible-error rate by at least 50% in the first 100 production runs compared to a prompt-only guardrail, with less than 10% throughput overhead.
Proof of conceptWrap a file-management/automation agent with a guardrail service that validates each proposed tool call against a small set of preconditions (path safety, quota limits, action reversibility) before dispatch; run agentic tasks on a dev corpus.
MeasurementIrreversible-error rate and latency/throughput overhead.
Source video ↗