AI Engineer · Published 2026-10-02

Stop Rationing Tokens: Let the Harness Pick the Model — Kimchi by Cast AI

privateReview: unreviewedConfidence: high
token optimizationmulti-model routingagentic codingcloud sandboxesRemediation loopmodel-routingcost-per-taskagent-harnessremediation-loopcloud-sandboxesasync-agentscode-reviewGuardrails
Open on YouTube ↗

Summary

Overview

  • Speaker: Laurent (Co-founder and President of Cast AI) and Zilvinas (Head of Engineering for Kimchi)
  • Channel: AI Engineer
  • Main topic: Dynamic model routing and token optimization in agentic coding environments via Kimchi
  • Purpose: Introduce Kimchi Coding and its architectural capabilities for dynamic multi-model routing, cost reduction, and automated software development life cycle management. Laurent and Zilvinas discuss the challenges of runaway token costs in enterprise AI development, where companies resort to rationing developer token budgets like limiting battery life. They introduce Kimchi, a coding platform utilizing proprietary models for edge cases and open-source for the rest, along with an automated token optimization engine that routes tasks to the right model at the right time. They also introduce /ferment for hands-off execution on complex tasks and Kimchi Teleport/Studio for collaborative agentic workflows.

Topic Map

The Token Cost Problem in Enterprise AI

  • Explanation: Exploration of extreme token consumption in agentic loops, such as multi-million dollar bills in single months and companies blowing annual AI budgets in a few months.
  • Key claims:
    • Agentic loops can consume over 1,000 more tokens than a chat query.
    • Companies respond by limiting developer token budgets, which harms productivity.
  • Examples:
    • An enterprise client receiving a $500M Claude bill in one month.
    • An Uber CTO burning an entire annual AI coding budget in four months.
  • Terminology:
    • Token caps
    • Agentic loops
    • Claude bill
  • Why it matters: Token rationing stifles developer productivity by limiting access to AI tooling.

Token Optimization and Multi-Model Routing

  • Explanation: Kimchi's automated engine selects the right model for a task based on cost, capability, and outcome rather than relying on a single expensive frontier model.
  • Key claims:
    • Different LLMs have drastically different actual cost per task for the same quality output.
    • An automated harness can achieve 2.5x savings over Claude while maintaining task quality.
  • Examples:
    • Comparing listed price rank vs actual cost rank across models like Opus, GPT-4, Gemini, and MiniMax.
  • Terminology:
    • Token optimization
    • Harness
    • Multi-model routing
  • Why it matters: Managers have a responsibility to make tokens unlimited and inexpensive through smart routing.

Kimchi Coding, /ferment, and Teleport

  • Explanation: Demonstration of advanced tools built by Kimchi to handle complex coding tasks autonomously and securely in the cloud.
  • Key claims:
    • /ferment breaks agent work into milestones, executes autonomously, evaluates each step, and runs multiple passes until output is correct.
    • Kimchi Teleport offloads agent sessions to secure cloud containers so developers can close their laptops and keep working.
  • Examples:
    • /ferment process with Change, Build, Break, Ferment, Complete loop.
  • Terminology:
    • /ferment
    • Kimchi Teleport
    • Kimchi Studio
    • Remediation loop
  • Why it matters: Enables true asynchronous agentic engineering with robust error recovery and human-in-the-loop review.

Key Points

Token caps are the wrong response to high LLM costs

  • Explanation: Treating token budgets like limited laptop batteries makes engineers unproductive.
  • Evidence: Enterprise cases show budgets burning out in months due to uncontrolled agentic loops.
  • Practical implication: Instead of rationing, organizations should implement automated multi-model routing.

Models vary wildly in actual cost per task

  • Explanation: Listed token prices do not reflect the actual cost required to successfully complete a specific engineering task with equivalent quality.
  • Evidence: Academic white paper comparisons show cheaper models sometimes cost more per task due to retries or inefficiencies.
  • Practical implication: Use an automated harness to dynamically select the most cost-effective model per task.

Reading code is no longer enough in agentic engineering

  • Explanation: As generated code volume explodes, reviewing raw lines or diffs becomes unmanageable.
  • Evidence: Engineers and product managers generate massive PRs from agentic workflows.
  • Practical implication: Review intent, specs, and execution contracts rather than raw lines of code.

Frameworks, Models & Processes

Kimchi Remediation Loop (/ferment)

  • How it works: Breaks complex tasks into milestones, runs automated builds, detects breakages, diagnoses and repairs failures, and redeploys to staging.
  • Components:
    • Change
    • Build
    • Break
    • Ferment
    • Complete
  • When to use: For long-running, complex software development tasks requiring autonomous execution.

Examples & Case Studies

An enterprise client spent $500M on Anthropic in one month due to unconstrained agentic loops without usage limits.

  • Illustrates: The danger of runaway token consumption in enterprise AI deployment.
  • Lesson: Robust guardrails, routing, and usage monitoring are mandatory for enterprise AI agents.

Actionable Takeaways

  • Immediate:
    • Implement dynamic model routing to reduce token costs.
    • Use automated harnesses to select cost-effective models per task.
  • Strategic:
    • Shift code review focus from raw lines and diffs to intent and specifications.
    • Adopt cloud-offloaded agent execution environments for asynchronous development.
  • Questions to investigate:
    • How does model routing impact code quality across different programming languages?
    • What security measures are required when running automated cloud agent sandboxes?

Claims Worth Verifying

  • One enterprise client received a $500M bill from Claude for a single month of usage. (statistical claim)
  • Kimchi achieves 2.5x savings over Claude using multi-model routing. (performance metric)

Notable Quotes

"Our job as managers: make tokens unlimited and inexpensive." (at 3:24) "Reading code is not enough anymore." (at 18:21) "Offload your agent session to the cloud — close your laptop, keep working." (at 19:12)

Compressed Summary

  • Unconstrained token consumption in agentic loops causes massive enterprise bills.
  • Kimchi introduces dynamic multi-model routing to optimize token spend.
  • /ferment provides automated milestone execution and self-healing remediation loops.
  • Kimchi Teleport and Studio enable secure cloud-based agentic workflows and team collaboration.
  • Keywords: token optimization, multi-model routing, agentic coding, cloud sandboxes, remediation loop
  • Core insight: Automated multi-model routing and self-healing harnesses eliminate the need for token rationing, lowering costs while scaling agentic engineering productivity.

Core insights

6
Empirical Resulthigh noveltymoderate evidence

Listed per-token price is a poor proxy for the actual cost of completing a task: cheaper models can cost more per successful task because of retries and inefficiencies, so routing should be optimized against cost-per-completed-task rather than list price.

Why it matters

It reframes model selection as an outcome-based optimization problem; a routing policy that ranks models by sticker price will systematically pick the wrong model and may cost more than using a single frontier model.

Generalization

Any agent harness that selects among tools/models should benchmark on end-to-end task cost (including retries, loop iterations, and evaluation passes), not on unit price.

Listed token prices do not reflect the actual cost required to successfully complete a specific engineering task with equivalent quality.
Open source video
cheaper models sometimes cost more per task due to retries or inefficiencies
Open source video
Architecturehigh noveltymoderate evidence

Model selection is being moved out of the developer's hands and into the harness layer, which dynamically routes each task to the cheapest capable model instead of defaulting to one expensive frontier model.

Why it matters

It establishes a new abstraction boundary: a routing/optimization layer between the agent loop and the model APIs, making model choice a runtime policy rather than a static configuration choice.

Generalization

Treat model choice as a pluggable routing policy inside the harness, analogous to a scheduler or load balancer, so model substitution and cost policy change without touching agent logic.

Kimchi's automated engine selects the right model for a task based on cost, capability, and outcome rather than relying on a single expensive frontier model.
Open source video
An automated harness can achieve 2.5x savings over Claude while maintaining task quality.
Open source video
Mechanismmedium noveltymoderate evidence

The remediation loop treats failure as the primary driver of work: a task is decomposed into milestones, built, deliberately exercised until it breaks, diagnosed, repaired, and redeployed — repeating passes until output is correct.

Why it matters

It encodes self-healing as an explicit control loop with named phases (Change, Build, Break, Ferment, Complete), rather than relying on a single generation pass plus ad hoc human fixes.

Generalization

For long-running autonomous tasks, model the workflow as a build-break-repair cycle with milestone checkpoints and objective pass/fail evaluation at each step.

Breaks complex tasks into milestones, runs automated builds, detects breakages, diagnoses and repairs failures, and redeploys to staging.
Open source video
/ferment breaks agent work into milestones, executes autonomously, evaluates each step, and runs multiple passes until output is correct.
Open source video
Architecturemedium noveltymoderate evidence

Agent sessions are being decoupled from the developer's machine by offloading them to secure cloud containers, turning agentic coding into an asynchronous, long-running workload rather than an interactive laptop process.

Why it matters

It changes the deployment model for coding agents: session state, execution, and long-running tasks live in remote sandboxes, so developer hardware is no longer the runtime boundary.

Generalization

Treat agent sessions as durable, cloud-hosted jobs that can outlive any single client, which also surfaces sandbox security and isolation as first-class requirements.

Kimchi Teleport offloads agent sessions to secure cloud containers so developers can close their laptops and keep working.
Open source video
What security measures are required when running automated cloud agent sandboxes?
Open source video
Mental Modelmedium noveltyweak evidence

As agent-generated code volume grows, line-by-line diff review becomes unmanageable, shifting the unit of review from raw code to intent, specs, and execution contracts.

Why it matters

It changes what artifacts an agentic workflow must produce and preserve: the reviewable surface becomes the specification and contract, not the diff, which has direct implications for what the harness must log and expose.

Generalization

Agent systems that generate at scale should emit verifiable intent/spec artifacts alongside code, and tooling should review against those artifacts.

Review intent, specs, and execution contracts rather than raw lines of code.
Open source video
Reading code is not enough anymore.
Open source video
Practicemedium noveltyweak evidence

Token rationing via hard budget caps is presented as an anti-pattern that suppresses engineering productivity, with the proposed alternative being routing plus guardrails rather than caps.

Why it matters

It frames the enterprise cost problem as an optimization and observability problem rather than a budgeting problem, implying teams should invest in routing and usage monitoring before throttling developers.

Generalization

Cost control for agentic workloads should be implemented as policy at the execution layer (routing, monitoring, limits per loop) rather than as coarse per-developer quotas.

Treating token budgets like limited laptop batteries makes engineers unproductive.
Open source video
Robust guardrails, routing, and usage monitoring are mandatory for enterprise AI agents.
Open source video

Deep dives

4

Measuring Cost-per-Task in Multi-Model Agent Routing

Research question

How can we reliably measure the end-to-end cost per completed task across different models, accounting for retries and loop iterations?

Why

Without accurate per-task cost metrics, routing policies will misallocate tasks, potentially increasing cost or degrading quality. This metric is foundational for any cost-optimized routing system.

Different LLMs have drastically different actual cost per task for the same quality output.
Open source video
An automated harness can achieve 2.5x savings over Claude while maintaining task quality.
Open source video
Source video

Security Models for Cloud-Offloaded Agent Sandboxes

Research question

What isolation, credential-scoping, and network-egress constraints are necessary to securely run autonomous coding agents in cloud containers?

Why

Cloud offloading enables asynchronous agent work but introduces significant security risks if sandboxes are compromised or agents have excessive permissions. This is a prerequisite for enterprise adoption.

Kimchi Teleport offloads agent sessions to secure cloud containers so developers can close their laptops and keep working.
Open source video
Source video

Termination Criteria for Remediation Loops

Research question

Under what conditions does a build-break-repair loop converge to a correct solution, and what stopping criteria prevent infinite loops or cost divergence?

Why

Without clear termination criteria, remediation loops can consume unbounded tokens without guaranteeing correctness. Understanding convergence properties is essential for cost-effective self-healing agents.

/ferment breaks agent work into milestones, executes autonomously, evaluates each step, and runs multiple passes until output is correct.
Open source video
Enables true asynchronous agentic engineering with robust error recovery and human-in-the-loop review.
Open source video
Source video

Design of Per-Loop Guardrails as an Alternative to Token Rationing

Research question

What per-loop guardrail designs (e.g., max iterations, cost-per-task ceilings) effectively control agentic costs without limiting developer productivity?

Why

Token rationing is rejected as an anti-pattern, but concrete guardrail designs are needed to manage costs while maintaining autonomy. This research can inform enterprise policy.

Companies respond by limiting developer token budgets, which harms productivity.
Open source video
Managers have a responsibility to make tokens unlimited and inexpensive through smart routing.
Open source video
Source video

Article ideas

3

Stop Rationing Tokens: The Case for Harness-Level Cost Optimization

Token rationing via developer quotas is counterproductive; instead, enterprises should invest in harness-level routing and per-task cost optimization to reduce costs while improving productivity.

Angle

Argues that cost control should be an engineering problem solved at the execution layer, not a budgeting problem.

Source video

Beyond the Diff: Why Agent-Generated Code Requires Intent-Based Review

As agentic coding scales, line-by-line diff review becomes infeasible; the reviewable unit must shift to intent, specifications, and execution contracts, requiring harnesses to capture and surface these artifacts.

Angle

Explores the implications for tooling and workflow when code volume outstrips human review capacity.

Source video

The Self-Healing Agent: Engineering Build-Break-Repair Loops That Actually Terminate

Remediation loops can drive autonomous correctness, but without convergence criteria they risk runaway costs; we need objective termination conditions and cost-aware stopping rules.

Angle

Examines the trade-off between correctness and token cost in multi-pass agent workflows.

Source video

Project ideas

3

CostPerTask Bench

beyond-evals

A benchmark that measures end-to-end cost per successful task across models will reveal that routing based on list price is suboptimal, and that a cost-per-task router can achieve >2x savings over a single frontier model on a suite of software engineering tasks.

Proof of concept

Build a harness that runs a set of coding tasks (e.g., from SWE-bench) across multiple models, evaluates success, and computes total tokens and cost per task. Implement a simple router that selects models based on learned cost-per-task and compare against always using a frontier model.

Measurement

Average cost per successful task, success rate, and total cost for the router vs baseline. Hypothesis supported if router achieves >2x cost reduction with no drop in success rate.

Source video

Agent Sandbox Security Scanner

gatehouse

A set of adversarial tests can identify common security misconfigurations in cloud agent sandboxes (e.g., overly permissive credentials, unrestricted network egress), and applying isolation best practices significantly reduces attack surface without hindering agent functionality.

Proof of concept

Create a test suite that attempts to escape sandbox, access unauthorized resources, or exfiltrate data from within a simulated agent environment. Apply hardening measures (e.g., least-privilege credentials, network policies) and re-test.

Measurement

Number of successful exploits before and after hardening; impact on agent task completion time and success rate.

Source video

Remediation Loop Convergence Analyzer

movement-lab

Remediation loops with fixed maximum iteration counts and objective pass/fail gates will converge to correct solutions within a bounded number of passes for most tasks, but a significant fraction will either exceed the limit or oscillate, indicating the need for smarter termination criteria.

Proof of concept

Instrument a build-break-repair agent to log each pass's outcome, cost, and whether the task eventually passes. Run on a set of coding tasks and analyze pass counts and failure modes.

Measurement

Distribution of pass counts, percentage of tasks that converge within N passes, and cases of oscillation or divergence. Success if we can classify tasks into convergent vs non-convergent and identify predictive features.

Source video

Architectural implications

3

Model choice moves from a developer/config-time decision to a harness-level runtime routing policy.

Before

Developers pick a single frontier model and accept its cost and behavior for all tasks.

After

The harness evaluates each task (cost, capability, outcome) and dispatches to the most cost-effective capable model.

Consequence

Model selection, cost policy, and fallback behavior become centralized, testable components separate from agent logic — but require per-task evaluation signal to route correctly.

Source video

Agent execution is decoupled from the developer's local machine into cloud containers.

Before

Agent sessions run on the developer's laptop and terminate when the laptop closes.

After

Sessions are offloaded to secure cloud containers that continue running independently of the client.

Consequence

Long-running asynchronous agent jobs become viable, but sandbox isolation, security, and session-state durability become mandatory infrastructure concerns.

Source video

The reviewable unit in agentic engineering shifts from diffs to intent, specs, and execution contracts.

Before

Reviewers read line-by-line diffs and pull requests.

After

Reviewers evaluate intent, specifications, and contracts, treating generated code as an implementation detail.

Consequence

The harness must capture and surface intent/spec artifacts and objective pass/fail evidence, or review degrades into unreviewed bulk merges.

Source video

Tradeoffs and failure modes

4

Cost-optimized multi-model routing

Benefit

Reported 2.5x savings over a single frontier model while maintaining task quality.

Cost or risk

Routing decisions depend on accurate per-task cost and quality signals; a mis-calibrated router could silently degrade output quality or increase cost through retries.

An automated harness can achieve 2.5x savings over Claude while maintaining task quality.
Open source video
Source video

Multi-pass remediation loop

Benefit

Repeated build/break/repair passes can drive output to correctness without human intervention.

Cost or risk

Multiple passes multiply token consumption, so correctness gains trade directly against the token cost the mechanism is meant to reduce.

runs multiple passes until output is correct.
Open source video
Source video

Cloud-offloaded agent sandboxes

Benefit

Developers can close their laptops and keep working; sessions run asynchronously in the cloud.

Cost or risk

Running automated agents in cloud containers raises unresolved security questions about isolation and access.

What security measures are required when running automated cloud agent sandboxes?
Open source video
Source video

Unconstrained agentic loops

Benefit

Maximizes agent autonomy and throughput without developer throttling.

Cost or risk

Without guardrails or usage limits, loops can burn budgets at an extreme rate.

Agentic loops can consume over 1,000 more tokens than a chat query.
Open source video
Source video

Open questions

5

How do you construct a reliable end-to-end cost-per-task metric that accounts for retries and loop iterations across models?

Why unresolved

The summary asserts that list price diverges from actual cost and cites academic comparisons, but gives no method for measuring per-task cost.

Research direction

Build a benchmark harness that runs identical engineering tasks across models and records total tokens and wall-clock cost until a task passes evaluation.

Source video

Does multi-model routing change code quality or reliability as a function of language or task type?

Why unresolved

The summary explicitly lists this as an open question and offers no breakdown by language or task.

Research direction

Stratify routing evaluations by language and task category and measure pass rates, not just cost.

Source video

What security model is required for autonomous agents running in cloud sandboxes?

Why unresolved

Cloud offloading is presented as a capability, but security handling is only raised as an open question.

Research direction

Define isolation, credential-scoping, and network-egress constraints for agent sandboxes and test them adversarially.

Source video

When does a build-break-repair loop terminate instead of oscillating or diverging in cost?

Why unresolved

The loop is described as running 'multiple passes until output is correct' with no stated stopping criteria or failure threshold.

Research direction

Instrument remediation loops to measure pass counts, escape conditions, and cases where repeated passes never converge.

Source video

What per-loop guardrails replace token rationing without reintroducing productivity limits?

Why unresolved

Rationing is rejected and guardrails recommended, but no concrete guardrail design is described.

Research direction

Compare loop-level limits (max iterations, cost-per-task ceilings) against developer-level quotas for cost and productivity outcomes.

Source video

Key claims

8
factualVerification needed

An enterprise client received a $500M Claude bill for a single month of usage.

Evidence

An enterprise client receiving a $500M Claude bill in one month.

Question

Can the $500M single-month bill be corroborated by the named client or vendor, and does it include non-agentic usage?

Source video
comparativeVerification needed

Agentic loops can consume over 1,000x more tokens than a chat query.

Evidence

Agentic loops can consume over 1,000 more tokens than a chat query.

Question

What measurement basis supports the ~1,000x figure, and is it per-task or per-session?

Source video
comparativeVerification needed

An automated harness can achieve 2.5x savings over Claude while maintaining task quality.

Evidence

An automated harness can achieve 2.5x savings over Claude while maintaining task quality.

Question

How is 'task quality' measured, and on what task suite is the 2.5x savings observed?

Source video
causalVerification needed

Different LLMs have drastically different actual cost per task for the same quality output.

Evidence

Different LLMs have drastically different actual cost per task for the same quality output.

Question

What is the observed spread in cost-per-task across models holding quality constant?

Source video
factualVerification needed

An Uber CTO burned an entire annual AI coding budget in four months.

Evidence

An Uber CTO burning an entire annual AI coding budget in four months.

Question

Is this a publicly attributable statement, and what budget scope does it cover?

Source video
causalVerification needed

Cheaper models sometimes cost more per task due to retries or inefficiencies.

Evidence

cheaper models sometimes cost more per task due to retries or inefficiencies

Question

For which model/task combinations does retry overhead invert the cost ranking?

Source video
opinionVerification not requested

Reading raw code is no longer sufficient for review in agentic engineering.

Evidence

Reading code is not enough anymore.

Source video
opinionVerification not requested

Managers should make tokens unlimited and inexpensive rather than rationing them.

Evidence

Our job as managers: make tokens unlimited and inexpensive.

Source video

Connections

5