AI Engineer · Published 2026-10-17

The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI

privateReview: unreviewedConfidence: high
tokenomicsinferencecost-optimizationopen-modelsagentic-workflowsinference-costmodel-routingprompt-cachingbatchingcost-attributiondistillation
Open on YouTube ↗

Summary

Overview

  • Speaker: Eugene Cheah
  • Channel: AI Engineer
  • Main topic: Optimizing AI inference costs and tokenomics at scale
  • Purpose: To educate AI engineers and leaders on how to understand, measure, and optimize token spend through structured frameworks and cost-saving operational strategies. Eugene Cheah, CEO and founder of Featherless AI, presents a comprehensive guide to optimizing tokenomics and lowering the high AI bill for organizations spending significantly on LLM inference. He outlines a four-step framework: measuring token spend, right-sizing models, buying inference capacity efficiently, and checking internal processes. He explores the economic differences between agentic workflows and process automation, detailing strategies like model distillation, classifier patterns, and dedicated versus on-demand infrastructure.

Topic Map

Introduction & Background

  • Explanation: Introduction of the speaker, Featherless AI's mission to make AI accessible across all languages and dialects, and an overview of scale in token consumption.
  • Key claims:
    • Featherless AI provides access to over 40,000 open models via a single API.
    • Large customers serve up to 2 trillion tokens a day at scale.
  • Examples:
    • Serving 2 trillion tokens a day from a single customer at approximately $0.03 per million tokens.
  • Terminology:
    • Tokenomics
    • Open models
    • Inference provider
  • Why it matters: Establishes context on the scale of token consumption and why token optimization is critical for heavy users.

Step One: Measure It

  • Explanation: The necessity of tracking token spend by team, workload, and API key rather than using shared credentials.
  • Key claims:
    • Many organizations use a single shared API key across the entire company, obscuring where costs originate.
    • Token spend must be split by department and use case to inform workload decisions.
  • Examples:
    • Splitting shared API keys into department-specific and use-case-specific keys.
  • Terminology:
    • Token spend
    • API key
    • Workload tracking
  • Why it matters: Without granular measurement, organizations cannot make informed decisions about model selection or cost reduction.

Two Tracks: Agentic vs. Process Automation

  • Explanation: Contrasting the economics of long-running agentic workloads with high-volume, short-context process automation tasks.
  • Key claims:
    • Agentic workflows are long-running, multi-turn, context-heavy, and cash-heavy on input/caching.
    • Process automation is high-volume, predefined, task-isolated, and represents a pure volume game.
  • Examples:
    • Coding agents for agentic workflows vs. call center routing and text filtering for process automation.
  • Terminology:
    • Agentic workload
    • Process automation
    • Context caching
  • Why it matters: Different AI workloads have opposite economic profiles and require distinct optimization strategies.

Step Two: Right-Size It

  • Explanation: Selecting the smallest passing model for process automation and the best model for agentic judgment tasks.
  • Key claims:
    • Organizations often default to frontier models for trivial tasks due to the 'shiny new hammer' syndrome.
    • Open models and specialized classifiers (like Jeff or StepFun) can achieve comparable performance at a fraction of the cost.
  • Examples:
    • Using small classifiers for multi-choice categorization tasks instead of large language models.
  • Terminology:
    • Right-sizing
    • Classifiers
    • Model distillation
  • Why it matters: Right-sizing prevents overspending on expensive frontier models for tasks that smaller models can handle.

Step Three: Buy It Well

  • Explanation: Strategies for purchasing inference capacity, balancing dedicated capacity for base load with on-demand for bursts.
  • Key claims:
    • Sharing inference nodes among multiple agents lowers cost per token significantly while trading off some speed.
    • Pay for speed only where a person is actively waiting (interactive vs. background workloads).
  • Examples:
    • Sharing a node among 16 agents to reduce cost by roughly 9x.
  • Terminology:
    • Dedicated capacity
    • Base load
    • On-demand burst
    • Throughput
  • Why it matters: Purchasing inference capacity based on traffic patterns yields massive cost savings at scale.

Step Four: Check Your Processes

  • Explanation: Identifying organizational anti-patterns such as the 'slop grenade' where unreviewed AI outputs increase human review overhead.
  • Key claims:
    • Tokens are not work done; generating massive amounts of unreviewed code creates review bottlenecks.
    • Organizations must enforce ownership and measure outcomes rather than raw token usage.
  • Examples:
    • AI agents generating large unreviewed code changes that take human reviewers hours to fix.
  • Terminology:
    • Slop grenade
    • Output ownership
    • Outcome measurement
  • Why it matters: Process inefficiencies can negate token cost savings by shifting the cost to human review time.

Key Points

Granular measurement is the foundation of cost optimization

  • Explanation: Organizations cannot optimize what they do not measure. Splitting API keys by team and workload is essential.
  • Evidence: 30-40% of customers reaching out to Featherless initially used a single shared API key company-wide.
  • Practical implication: Implement separate API keys per team and workload immediately.

Agentic and process automation workloads have opposite economics

  • Explanation: Agentic workflows prioritize reasoning quality and cash-heavy context, while process automation prioritizes low cost and high volume.
  • Evidence: Agentic workflows spend over 80% of budget on re-reading context, while process automation focuses on cost per request.
  • Practical implication: Tailor model selection and caching strategies to the specific workload type.

Right-sizing prevents paying for unused intelligence

  • Explanation: Using frontier models for simple classification or extraction tasks wastes money when smaller open models or classifiers suffice.
  • Evidence: Classifiers and smaller open models achieve near-frontier quality on specialized benchmarks at 10x lower cost.
  • Practical implication: Audit existing pipelines to replace oversized models with smaller task-specific models.

Tokens do not equal work done

  • Explanation: Generating large volumes of AI content without human ownership or review creates expensive downstream bottlenecks.
  • Evidence: 40% of desk workers reported spending hours reviewing AI output, shifting costs rather than eliminating them.
  • Practical implication: Establish clear ownership and review processes for AI-generated artifacts.

Frameworks, Models & Processes

Four-Move Token Optimization Framework

  • How it works: A sequential four-step process to control and reduce AI token bills in an organization.
  • Components:
    • Measure it (track spend by team and workload)
    • Right-size it (match model capability to task requirements)
    • Buy it well (optimize infrastructure purchasing and node sharing)
    • Check your processes (eliminate organizational anti-patterns and review bottlenecks)
  • When to use: When organizational AI spending exceeds $10K/month and cost efficiency needs to be established.

Examples & Case Studies

A team of 5 developers running 8 continuous coding agents on Opus 4.8 was spending $150K/month.

  • Illustrates: The high cost of running agentic workloads continuously on top-tier frontier models.
  • Lesson: Switching to smaller optimized models or managing agent execution schedules dramatically reduces unnecessary spend.

A major telco shifted high-volume classification and extraction tasks from closed frontier models to fine-tuned open models.

  • Illustrates: Process automation cost reduction through right-sizing and fine-tuning.
  • Lesson: Inference costs dropped by 95% while maintaining required accuracy for telco-specific tasks.

Actionable Takeaways

  • Immediate:
    • Split shared API keys by department, team, and use case.
    • Audit high-spend workflows to identify oversized model usage.
    • Separate user-facing interactive traffic from background batch processing.
  • Strategic:
    • Adopt a multi-model strategy utilizing open-source and fine-tuned models for specific tasks.
    • Align team incentives around accepted outcomes rather than raw token generation.
    • Design infrastructure purchasing around base load and on-demand burst patterns.
  • Questions to investigate:
    • Can we see spend by team and by workload?
    • What test must a cheaper model pass before deployment?
    • What is our base load in a normal week?
    • Who sets the default model for each team?
    • Which workloads have a person waiting?
    • How much of what we generate gets actually used?

Claims Worth Verifying

  • Open models can match frontier model performance on specialized financial and coding benchmarks at 10x lower cost. (Benchmark claim)
  • Sharing a single inference node among multiple agents can lower token costs by approximately 9x while trading off speed. (Infrastructure economics claim)

Notable Quotes

"The AI bill is too damn high." (at 0:16) "We sell inference. This applies to any provider." (at 3:32) "Tokens are not work done." (at 57:11) "Nobody switches models to save the company money." (at 25:41)

Compressed Summary

  • Measure token spend granularly by team, workload, and API key.
  • Distinguish between agentic workflows and high-volume process automation.
  • Right-size models using classifiers and distilled open-source alternatives.
  • Optimize inference purchasing by balancing dedicated base load and shared burst capacity.
  • Fix organizational processes to avoid the 'slop grenade' review bottleneck.
  • Keywords: tokenomics, inference, cost-optimization, open-models, agentic-workflows
  • Core insight: Reducing high AI bills requires a systematic approach encompassing granular measurement, right-sized model selection, smart infrastructure purchasing, and disciplined organizational processes.

Core insights

6
Mental Modelmedium noveltymoderate evidence

Agentic and process-automation workloads have structurally opposite cost profiles, so the same optimization playbook inverts between them: agentic loops are long-running, multi-turn and dominated by re-reading context (cash-heavy on input/caching), while process automation is high-volume, task-isolated and purely a cost-per-request volume game.

Why it matters

It splits the cost-engineering roadmap into two distinct tracks before any model choice is made. For agentic systems the lever is context/cache economics per turn; for automation it is per-request unit cost and throughput. Applying one track's tactics to the other wastes effort or degrades quality.

Generalization

Classify any LLM workload by (turns per task, context reuse ratio, human-in-the-loop latency sensitivity) before selecting models or infrastructure; the cost function, not the feature description, determines the optimization strategy.

Agentic workflows are long-running, multi-turn, context-heavy, and cash-heavy on input/caching.
Open source video
Agentic workflows spend over 80% of budget on re-reading context, while process automation focuses on cost per request.
Open source video
Process automation is high-volume, predefined, task-isolated, and represents a pure volume game.
Open source video
Tradeoffmedium noveltymoderate evidence

Sharing a single inference node across many concurrent agents collapses cost per token at the price of speed — the cited configuration packs 16 agents onto one node for roughly a 9x cost reduction — reframing capacity purchasing as a multiplexing/scheduling problem rather than a per-agent procurement problem.

Why it matters

It makes concurrency scheduling a first-class architectural component: an agent runtime that queues, batches and timeshares a shared node pool can beat per-agent dedicated endpoints by nearly an order of magnitude in cost, if the workload tolerates latency variance.

Generalization

Cost per token is a function of utilization, not just model size; any inference layer that interleaves independent workloads can trade latency headroom for large unit-cost reductions.

Sharing inference nodes among multiple agents lowers cost per token significantly while trading off some speed.
Open source video
Sharing a node among 16 agents to reduce cost by roughly 9x.
Open source video
Practicemedium noveltyweak evidence

Latency is a positional purchase: speed should be paid for only where a human is actively waiting, while background workloads can absorb queueing. This splits the same system into interactive and background latency classes with different infrastructure and cost envelopes.

Why it matters

It gives a concrete decision rule for capacity allocation — dedicated/fast capacity for user-blocking calls, shared/queued capacity for agent background loops — instead of paying a latency premium uniformly across all traffic.

Generalization

Any AI system with mixed human-facing and machine-facing calls should expose at least two latency tiers and route by whether a person is blocked.

Pay for speed only where a person is actively waiting (interactive vs. background workloads).
Open source video
Practicemedium noveltymoderate evidence

Cost attribution must be built into the calling convention: a single company-wide API key is the default state (reported in 30-40% of inbound customers) and destroys the ability to reason about model choice or workload value. Per-team, per-workload keying is the minimum instrumentation for any optimization.

Why it matters

Measurement determines which downstream decisions are even possible; without per-workload attribution, right-sizing and capacity purchasing are unmade or arbitrary. This makes tenancy/credential granularity an architecture concern, not an admin task.

Generalization

In any metered AI system, the identity used for a request (key, tenant, workload tag) is the primary observability primitive and should be designed before optimization begins.

Many organizations use a single shared API key across the entire company, obscuring where costs originate.
Open source video
30-40% of customers reaching out to Featherless initially used a single shared API key company-wide.
Open source video
Practicemedium noveltyweak evidence

Right-sizing means selecting the smallest model that passes, and for narrow tasks that often means a specialized classifier or small open model rather than a frontier LLM; classifiers and smaller open models are claimed to reach near-frontier quality on specialized benchmarks at roughly 10x lower cost.

Why it matters

It converts model selection from 'default to the best model' into an evaluation-gated routing decision. The real prerequisite is a cheap pass/fail eval per task, which becomes infrastructure: a classifier-first tier with escalation to a frontier model only on failure.

Generalization

Task-specific narrow models plus a verify-and-escalate path dominate frontier-only pipelines on cost whenever the task is decomposable into classification/extraction plus a small hard tail.

Organizations often default to frontier models for trivial tasks due to the 'shiny new hammer' syndrome.
Open source video
Classifiers and smaller open models achieve near-frontier quality on specialized benchmarks at 10x lower cost.
Open source video
Using small classifiers for multi-choice categorization tasks instead of large language models.
Open source video
Failure Modemedium noveltymoderate evidence

Token volume is not work completed: unreviewed generated artifacts (the 'slop grenade') move cost downstream into human review rather than removing it, with 40% of desk workers reporting hours spent reviewing AI output. Output ownership and outcome metrics must therefore be part of the agent's contract.

Why it matters

Token savings are illusory if they increase review load; the true unit of cost is accepted-and-owned output, not tokens emitted. This puts constraints on how much an agent may emit per unit of human attention and who signs off on it.

Generalization

Any agent whose output requires human adjudication should be optimized against cost-per-accepted-output, not cost-per-token, and its blast radius (diff size, artifact count) should be bounded by the reviewer's capacity.

Tokens are not work done; generating massive amounts of unreviewed code creates review bottlenecks.
Open source video
40% of desk workers reported spending hours reviewing AI output, shifting costs rather than eliminating them.
Open source video
AI agents generating large unreviewed code changes that take human reviewers hours to fix.
Open source video

Deep dives

5

Cost-function split: agentic loops vs. process automation

Research question

What taxonomy of workload features (turns per task, context-reuse ratio, human-blocking latency sensitivity) actually predicts which cost lever dominates, and can a classifier assign a workload to the right track before any optimization is attempted?

Why

If the two tracks have opposite cost functions, then every generic 'reduce your AI bill' tactic is mis-specified for half the traffic. A predictive classifier turns cost engineering from opinion into routing: agentic services get context/cache work, automation services get unit-cost and throughput work.

Agentic workflows are long-running, multi-turn, context-heavy, and cash-heavy on input/caching.
Open source video
Agentic workflows spend over 80% of budget on re-reading context, while process automation focuses on cost per request.
Open source video
Process automation is high-volume, predefined, task-isolated, and represents a pure volume game.
Open source video
Source video

Where node multiplexing stops paying: the contention knee

Research question

At what agents-per-node density does the cost-per-completed-task curve turn upward because latency variance and queueing degrade agent task success faster than unit cost falls?

Why

The 9x figure is a cost claim with no success-rate denominator. If cost per token drops 9x but task completion drops enough that agents retry, the real optimum is well below maximum density — and any shared-pool scheduler needs an operating point, not a slogan.

Sharing inference nodes among multiple agents lowers cost per token significantly while trading off some speed.
Open source video
Sharing a node among 16 agents to reduce cost by roughly 9x.
Open source video
Source video

The pass/fail gate as the binding constraint on right-sizing

Research question

What is the cheapest gate that reliably certifies 'smallest passing model' per task, and how does cost saved versus quality lost move as a function of gate strictness and eval sample size?

Why

Right-sizing is only as good as its gate. Without a maintained per-task eval, the 10x savings on classifiers silently become regressions on the long tail. The gate is infrastructure, and its cost/latency is itself part of the token bill.

Classifiers and smaller open models achieve near-frontier quality on specialized benchmarks at 10x lower cost.
Open source video
Using small classifiers for multi-choice categorization tasks instead of large language models.
Open source video
Source video

Pricing human review into the token bill

Research question

Can a review-cost multiplier (review minutes per generated artifact, normalized by artifact size) be derived and added to token cost inside routing and rate-limit decisions so that the 'slop grenade' shows up as a first-class cost line?

Why

Token-only optimization is a local optimum that can be globally worse. If review time scales with emitted volume, then an agent that emits less but costs more per token can be strictly cheaper — which inverts the default 'cheaper model, more output' instinct.

Tokens are not work done; generating massive amounts of unreviewed code creates review bottlenecks.
Open source video
40% of desk workers reported spending hours reviewing AI output, shifting costs rather than eliminating them.
Open source video
AI agents generating large unreviewed code changes that take human reviewers hours to fix.
Open source video
Source video

Latency tiering as positional spend allocation

Research question

What is the cost delta between a single-tier deployment and a two-tier (interactive-dedicated vs. background-queued) deployment for the same mixed workload, and how often do call paths get misclassified?

Why

It is the cheapest architectural win in the talk and requires no model changes — only a classification of every call path plus a queue. But misclassification is silent: either a blocked human waits in a batch queue, or a background loop pays a premium for speed nobody consumes.

Pay for speed only where a person is actively waiting (interactive vs. background workloads).
Open source video
Sharing inference nodes among multiple agents lowers cost per token significantly while trading off some speed.
Open source video
Source video

Article ideas

4

Your Agent Isn't Expensive Because of the Model — It's Expensive Because It Re-Reads

For multi-turn agentic systems, model choice is a second-order cost lever; context re-read economics (prefix stability, cache hit rate, compaction) are first-order. Teams that switch to a cheaper model while leaving an unstable prompt prefix untouched have optimized the wrong variable and will be disappointed.

Angle

Decompose a real agent bill into fresh-input, cached-input, and output tokens per turn, then show which architectural changes (stable system prefix, scratchpad summarization, tool-result truncation) move the dominant term.

Source video

Tokens Are Not Work Done: The Slop Grenade and the Case for Cost-Per-Accepted-Output

Token-based optimization is a local optimum: agents that emit large unreviewed artifacts shift cost into human review rather than removing it, and may be net-negative versus a more expensive model producing smaller, reviewable units. The correct unit of cost is accepted-and-owned output, and agent emission should be bounded by reviewer capacity.

Angle

Derive a review-cost multiplier from review-minutes-per-accepted-artifact and add it to token cost, then show how routing decisions flip once that multiplier is included.

Source video

The Single Shared API Key Is the Real AI Cost Problem

Cost attribution isn't an admin chore, it's the observability primitive that makes every downstream optimization decidable. Organizations that can't bill token spend to a workload cannot right-size models or buy capacity rationally — so tenancy and credential granularity belong in the architecture review, not the billing portal.

Angle

Start from the shared-key default state and show the decision chain that becomes impossible without per-workload keys: right-sizing, latency tiering, and capacity purchasing all depend on an owned cost line.

Source video

Stop Paying for Speed Nobody Is Waiting For

Latency should be purchased positionally, not uniformly. A two-tier deployment that reserves dedicated capacity for human-blocking calls and queues background agent loops captures most of the cost benefit of shared capacity without degrading the experience of anyone who is actually waiting.

Angle

Show the allocation rule plus its failure mode — misclassified call paths that either block a human in a batch queue or burn dedicated capacity on a loop nobody watches.

Source video

Project ideas

4

Token Ledger

gatehouse

Replacing a company-wide key with per-team and per-workload keys plus a spend dashboard causes measurable re-routing of call sites within one month, yielding at least 20% spend reduction attributable to model/infrastructure changes, with no drop in per-task eval pass rate.

Proof of concept

A thin proxy in front of an inference API that stamps every request with tenant, workload, and interactivity class; stores tokens and cost per (workload, model); exposes a per-use-case dashboard and a model-switch log.

Measurement

Spend per workload before/after; count and dollar impact of re-routed call sites; eval pass rate per workload to confirm savings aren't hidden regressions.

Source video

Escalation Gate

beyond-evals

A classifier-first tier with an eval-gated escalation path retains at least 97% of the frontier model's pass rate on decomposable tasks at no more than 30% of its cost, and the escalation rate is the dominant predictor of realized savings.

Proof of concept

Per-task eval suite with a cheap pass/fail judge; route by default to a small open model or classifier, escalate to a larger model only on gate failure; log gate decisions with ground truth.

Measurement

Pass rate versus frontier-only baseline; cost per accepted output; escalation rate; savings as a function of gate strictness threshold.

Source video

Contention Knee

movement-lab

Cost per completed task falls monotonically only up to a contention threshold and rises beyond it, because agent retry and failure rates grow faster than per-token cost falls; the minimum of cost-per-completed-task sits strictly below maximum agents-per-node density.

Proof of concept

Shared node pool with a queueing scheduler; run the same agent benchmark suite while sweeping concurrent agents per node from 1 to 32, mixing interactive and background agents and optionally enforcing a priority class.

Measurement

Cost per completed task, task success rate, retry rate, p50/p95 wall-clock per task, as functions of agents-per-node and priority mix.

Source video

Review-Cost Multiplier

new

Human review minutes grow superlinearly with artifact size, so an agent that emits smaller bounded diffs at a higher token cost per task achieves lower total cost per accepted output than a high-emission agent, flipping the default routing decision.

Proof of concept

Instrument an agent workflow to log generated artifact size and tokens per task alongside review minutes and rework rate per accepted artifact; fit a review-cost curve and inject it into the routing objective.

Measurement

Review minutes per accepted artifact versus emitted tokens or diff size; rework rate; total cost per accepted output (token cost plus review cost) across emission-capped and uncapped configurations.

Source video

Architectural implications

4

Cost attribution is currently collapsed into shared credentials, so spend cannot be mapped to workload.

Before

One company-wide API key and a monthly invoice total.

After

Key-per-team and key-per-workload, with spend dashboards per use case that feed model-selection decisions.

Consequence

Optimization becomes decidable; every model/route change can be evaluated against an owned cost line.

Source video

Capacity is modeled per agent rather than as a shared pool with utilization economics.

Before

Each agent or service gets dedicated inference capacity sized for peak.

After

A shared node pool with a scheduling/queueing layer, priority classes for interactive vs background work, and deliberate latency relaxation for non-blocking calls.

Consequence

Large unit-cost reductions at scale, but new engineering burden: latency SLOs, fairness between agents, and head-of-line blocking risk.

Source video

Model choice is hardwired per call site rather than routed by task difficulty.

Before

Every call in the pipeline hits the same frontier model.

After

A routing tier: small classifier/models by default with an eval-gated escalation path to a larger model on failure.

Consequence

Requires a maintained per-task eval harness and failure taxonomy; the cost win depends entirely on the quality of the pass/fail gate.

Source video

Agent output is treated as free because tokens are cheap, with no accounting for downstream human review.

Before

Agents emit large diffs/artifacts with no owner; review is unbounded.

After

Bounded emission per task, explicit ownership metadata on generated artifacts, and outcome metrics alongside token metrics.

Consequence

Shifts optimization target from token spend to cost-per-accepted-output, changing agent design (smaller, reviewable units).

Source video

Tradeoffs and failure modes

4

Node sharing / multiplexing

Benefit

Roughly 9x lower cost per token by sharing one node among ~16 agents.

Cost or risk

Reduced speed and latency variance; workloads with human-blocking calls may become unacceptable without priority separation.

Sharing a node among 16 agents to reduce cost by roughly 9x.
Open source video
Source video

Right-sizing to small models and classifiers

Benefit

Near-frontier quality on specialized tasks at roughly 10x lower cost.

Cost or risk

Quality risk on judgment-heavy agentic tasks; requires an eval harness to prove the small model still passes, otherwise savings are hidden regressions.

Classifiers and smaller open models achieve near-frontier quality on specialized benchmarks at 10x lower cost.
Open source video
Source video

High-throughput generation without ownership

Benefit

More artifacts produced per dollar of tokens.

Cost or risk

The 'slop grenade': review bottlenecks and rework shift cost from tokens to human hours, possibly net-negative.

Tokens are not work done; generating massive amounts of unreviewed code creates review bottlenecks.
Open source video
Source video

Latency spend placement

Benefit

Buying speed only where a human is waiting avoids paying a premium on background throughput.

Cost or risk

Requires classifying every call path by interactivity; misclassification either wastes money or blocks a user.

Pay for speed only where a person is actively waiting (interactive vs. background workloads).
Open source video
Source video

Open questions

4

At what agent-per-node density does sharing stop paying off because latency variance degrades agent task success?

Why unresolved

The summary reports a 16-agents-per-node / ~9x figure without any success-rate or latency data, so the curve and its knee are unknown.

Research direction

Benchmark agent task success and wall-clock completion versus contention level on a shared node pool, sweeping concurrency and mixing interactive with background agents.

Source video

How much of an agent's token bill is genuinely fresh content versus re-read context, and how much of the re-read share can be removed by caching and context compaction?

Why unresolved

The summary asserts agentic workflows spend over 80% of budget on re-reading context but gives no breakdown or measured effect of caching/compaction.

Research direction

Instrument a real agent loop to log fresh-input, cached-input, and output tokens per turn; measure cost deltas from stable prompt prefixes, compaction, and scratchpad summarization.

Source video

What is the cheapest reliable gate for 'smallest passing model' selection across many heterogeneous tasks?

Why unresolved

Right-sizing is recommended, but the summary gives no evaluation method, sample sizes, or error tolerance for the pass/fail decision.

Research direction

Prototype a continuous per-task eval suite with model-escalation policies; measure cost saved versus quality lost as a function of gate strictness.

Source video

How do you price human review cost per token so the 'slop grenade' is visible in the same dashboard as inference spend?

Why unresolved

The summary quantifies reviewer burden qualitatively (40% of desk workers spending hours) but proposes no unit economics linking review time to token volume.

Research direction

Instrument review-time-per-accepted-artifact and per-generated-artifact to derive a review-cost multiplier that can be added to token cost in routing and rate-limit decisions.

Source video

Key claims

7
factualVerification needed

A single Featherless customer serves 2 trillion tokens per day at roughly $0.03 per million tokens.

Evidence

Serving 2 trillion tokens a day from a single customer at approximately $0.03 per million tokens.

Question

Does the $0.03/M figure reflect blended input/output and cached tokens, and at what concurrency and latency?

Source video
factualVerification needed

30-40% of inbound customers initially used a single shared API key company-wide.

Evidence

30-40% of customers reaching out to Featherless initially used a single shared API key company-wide.

Question

Is this measured over a defined customer cohort and time window, and does it generalize beyond customers seeking cost help?

Source video
comparativeVerification needed

Agentic workflows spend over 80% of budget on re-reading context.

Evidence

Agentic workflows spend over 80% of budget on re-reading context, while process automation focuses on cost per request.

Question

What distribution of agent workloads produces this 80% figure, and how does it shift with caching and context compaction?

Source video
causalVerification needed

Sharing one node among 16 agents yields roughly a 9x cost reduction.

Evidence

Sharing a node among 16 agents to reduce cost by roughly 9x.

Question

Relative to what baseline (dedicated per-agent endpoints?) and at what latency and throughput penalty?

Source video
comparativeVerification needed

Classifiers and smaller open models reach near-frontier quality on specialized benchmarks at about 10x lower cost.

Evidence

Classifiers and smaller open models achieve near-frontier quality on specialized benchmarks at 10x lower cost.

Question

Which benchmarks, and does the gap widen on multi-step reasoning or long-context tasks?

Source video
causalVerification needed

Tokens generated do not equal work done; unreviewed generation creates downstream review bottlenecks.

Evidence

Tokens are not work done; generating massive amounts of unreviewed code creates review bottlenecks.

Question

What is the ratio of human review minutes to generated tokens across artifact types, and where does net cost flip negative?

Source video
factualVerification needed

40% of desk workers report spending hours reviewing AI output, shifting rather than eliminating cost.

Evidence

40% of desk workers reported spending hours reviewing AI output, shifting costs rather than eliminating them.

Question

What survey population and frequency of 'hours' — and is the review work new or substituted from prior manual work?

Source video

Connections

5