Kimi AI · Published 2026-03-21

How We Scaled Kimi K2.5 | Zhilin Yang's full GTC 2026 Keynote

privateReview: unreviewedConfidence: high
muonclipkimi-linearagent-swarmsattention-residualsvision-textmuonqk-cliplinear-attentionreward-shapingmultimodal-pretrainingtoken-efficiency
Open on YouTube ↗

Summary

Overview

  • Speaker: Zhilin Yang
  • Channel: Kimi AI
  • Main topic: Scaling strategies for open-source AI models, focusing on Kimi K2.5, MuonClip optimizer, Kimi Linear attention, attention residuals, and agent swarms.
  • Purpose: To explain the architectural innovations and scaling laws employed by Moonshot AI in developing Kimi K2.5 and its associated open-source techniques. Zhilin Yang, Founder and CEO of Kimi (Moonshot AI), presents the scaling strategies behind Kimi K2.5 at NVIDIA GTC 2026. The talk covers three key scaling dimensions: token efficiency via MuonClip optimizer, long context via Kimi Linear attention, and agent swarms via multi-agent reinforcement learning. Additionally, it highlights Kimi K2.5 as the first open model with native joint vision-text pretraining, achieving superior performance across modalities.

Topic Map

Introduction to Kimi's Scaling Strategy

  • Explanation: Introduction of open models, democratizing intelligence, and the three primary scaling dimensions: token efficiency, long context, and agent swarms.
  • Key claims:
    • Open models are closing the gap with proprietary models.
    • Scaling is the primary driver of major AI developments.
    • Scaling must happen across multiple dimensions: token efficiency, context length, and agent swarms.
  • Examples:
    • Deploying open models on local servers or the cloud with full weight access.
  • Terminology:
    • open models
    • scaling law
    • token efficiency
    • long context
    • agent swarms
  • Why it matters: Establishes the foundational philosophy of making frontier intelligence accessible to developers worldwide.

Token Efficiency: MuonClip Optimizer

  • Explanation: Discusses scaling token efficiency using the Muon optimizer combined with weight decay and QK-Clip to achieve ~2x token efficiency over AdamW without training divergence.
  • Key claims:
    • Muon optimizer provides ~2x token efficiency improvement over AdamW.
    • Training instability during large-scale Muon training can be resolved using QK-Clip.
    • Token efficiency improves the upper bound of model intelligence, acting like a data multiplier.
  • Examples:
    • Scaling a 1-trillion parameter model with Muon and encountering max logit explosions solved by QK-Clip.
  • Terminology:
    • Muon optimizer
    • MuonClip
    • weight decay
    • RMS updates
    • QK-Clip
    • max logits
  • Why it matters: Overcomes data wall limitations by maximizing intelligence per training token.

Long Context: Kimi Linear Attention (KDA)

  • Explanation: Introduces Kimi Linear Attention (KDA) and Kimi Linear Architecture, combining Delta Attention with a 3:1 ratio of linear to full attention layers to achieve linear scaling and superior long-context performance.
  • Key claims:
    • Transformers outperform LSTMs asymptotically due to long-context utilization across the whole context.
    • Kimi Delta Attention improves recurrent memory management and hardware efficiency.
    • Kimi Linear architecture achieves better performance than full attention baselines while cutting memory overhead.
  • Examples:
    • Comparing transformer and LSTM token-index scaling curves and benchmarking Kimi Linear on MMLU and RULER.
  • Terminology:
    • Kimi Delta Attention (KDA)
    • Gated Delta Rule (GDR)
    • Kimi Linear
    • chunkwise formulation
    • matrix inversion
  • Why it matters: Enables efficient processing of million-token context lengths without quadratic compute penalties.

Agent Swarms and Reinforcement Learning

  • Explanation: Explains the multi-agent swarm paradigm where an orchestrator agent delegates complex tasks to specialized sub-agents working in parallel, guided by specific reinforcement learning reward functions.
  • Key claims:
    • Agent swarms scale task complexity beyond single-agent capabilities.
    • Reinforcement learning for agent swarms requires instantiation, finish, and outcome rewards to prevent serial collapse and spurious parallelism.
    • Agent swarms save significant execution time on complex tasks compared to single agents.
  • Examples:
    • Writing a 100-page literature review by spawning specialized researchers and fact-checkers in parallel.
  • Terminology:
    • agent swarms
    • orchestrator
    • sub-agents
    • instantiation reward
    • finish reward
    • outcome reward
  • Why it matters: Solves complex, multi-step workflows by orchestrating collaborative intelligence.

Vision-Text Joint Pretraining & Attention Residuals

  • Explanation: Presents Kimi K2.5 as the first open model with native joint vision-text pretraining (early fusion) and introduces Attention Residuals (AttnRes) as a rotation of LSTM residuals into attention space.
  • Key claims:
    • Early fusion vision-text joint pretraining yields better results than late fusion.
    • Vision reinforcement learning improves textual performance (cross-modal transfer).
    • Attention Residuals improve token efficiency by 24% and reduce training loss.
  • Examples:
    • Training Kimi K2.5 on NVIDIA H800 clusters and demonstrating code, vision, and text capabilities.
  • Terminology:
    • early fusion
    • vision-text joint pretraining
    • Attention Residuals (AttnRes)
    • block attention residual
  • Why it matters: Unifies vision and text natively while introducing breakthrough architectural components.

Key Points

MuonClip doubles token efficiency over AdamW

  • Explanation: By applying the second-order Muon optimizer with weight decay and consistent RMS updates, model training achieves equivalent intelligence with half the tokens.
  • Evidence: Scaling law graphs showing ~2x token efficiency vs AdamW.
  • Practical implication: Reduces training compute requirements and effectively expands limited high-quality data budgets.

QK-Clip stabilizes large-scale Muon training

  • Explanation: Scaling Muon to 1-trillion parameter models causes max logits to explode; QK-Clip constrains query and key projections in attention heads to prevent divergence.
  • Evidence: Validation loss and max logits training curves before and after QK-Clip.
  • Practical implication: Allows stable training of large transformer models with advanced second-order optimizers.

Kimi Linear Attention balances efficiency and expressivity

  • Explanation: Using a 3:1 ratio of linear attention (Kimi Delta Attention) to full attention layers with chunkwise formulation cuts memory overhead while outperforming full attention.
  • Evidence: Performance benchmarks on RULER and MMLU across 128k to 1M token lengths.
  • Practical implication: Drastically reduces time-per-output-token (TPOT) at long context lengths.

Agent swarms accelerate complex task execution

  • Explanation: Orchestrator-driven agent swarms decompose complex tasks into parallel subtasks, using specialized RL reward functions to ensure successful task completion.
  • Evidence: Execution time vs task complexity graphs showing up to 4.5x speedups.
  • Practical implication: Enables automated execution of complex workflows like software engineering and comprehensive research.

Attention Residuals generalize residual connections

  • Explanation: Treating attention as an LSTM residual rotated by 90 degrees and applying attention residuals across layers improves pretraining token efficiency by 24%.
  • Evidence: Scaling law and benchmark comparison tables for Attention Residuals (AttnRes).
  • Practical implication: Provides a drop-in architectural upgrade over standard residual connections for transformer models.

Frameworks, Models & Processes

Kimi's Scaling Strategy Framework

  • How it works: Scales model capabilities across three orthogonal dimensions: Token Efficiency (MuonClip), Long Context (Kimi Linear Attention), and Agent Swarms.
  • Components:
    • Token Efficiency (MuonClip Optimizer)
    • Long Context (Kimi Delta Attention)
    • Agent Swarms (Orchestrator and Sub-agents)
  • When to use: When building and scaling frontier open-source foundation models.

Agent Swarms RL Reward Function

  • How it works: Guides multi-agent training using three weighted reward components to encourage effective task decomposition and completion.
  • Components:
    • Instantiation Reward (incentivizes sub-agent creation, prevents serial collapse)
    • Finish Reward (encourages completion of assigned subtasks)
    • Outcome Reward (measures whether the overall task achieves the end goal)
  • When to use: When training collaborative multi-agent systems via reinforcement learning.

Examples & Case Studies

Training a 1-trillion parameter model with Muon optimizer without QK-Clip.

  • Illustrates: Training instability and max logits explosion leading to divergence.
  • Lesson: Advanced second-order optimizers require stability mechanisms like QK-Clip for large-scale training.

Executing a 100+ word systematic literature review using K2.5 Agent Swarm.

  • Illustrates: Parallel task execution, document downloading, data analysis, and PDF compilation across specialized sub-agents.
  • Lesson: Agent swarms dramatically reduce execution time for complex, multi-source research tasks.

Actionable Takeaways

  • Immediate:
    • Adopt MuonClip optimizer for ~2x token efficiency in LLM pretraining.
    • Implement QK-Clip to prevent attention logit explosions in large models.
    • Use Attention Residuals as a drop-in replacement for standard residuals to boost efficiency.
  • Strategic:
    • Scale models across compute efficiency, context length, and agentic workflows simultaneously.
    • Embrace open models to democratize frontier AI capabilities globally.
    • Use early fusion for native multimodal (vision-text) joint pretraining.
  • Questions to investigate:
    • How do Attention Residuals scale beyond 100B parameters?
    • What are the optimal reward weights for scaling agent swarms to thousands of agents?

Claims Worth Verifying

  • Muon optimizer provides ~2x token efficiency improvement over AdamW. (empirical)
  • Kimi Linear architecture outperforms full attention baselines while cutting memory overhead. (benchmark)
  • Attention Residuals improve pretraining token efficiency by 24%. (architectural)

Notable Quotes

"We believe in democratizing intelligence." "Scaling is a primary driver of for a lot of progress." "Token efficiency is not just about efficiency. It's actually also about improving the upper bound of intelligence." "Muon optimizer is... ~2x token efficiency." "An LSTM is a ResNet rotated 90 degrees."

Compressed Summary

  • MuonClip optimizer delivers ~2x token efficiency over AdamW with QK-Clip stability.
  • Kimi Linear Attention (KDA) combines Delta Attention and 3:1 linear-to-full attention ratio for efficient long-context scaling.
  • Agent swarms use orchestrator-subagent hierarchies and 3-part RL rewards to solve complex multi-step tasks efficiently.
  • Attention Residuals rotate LSTM residuals into attention space, yielding a 24% boost in pretraining token efficiency.
  • Kimi K2.5 features native early-fusion vision-text joint pretraining and cross-modal RL transfer.
  • Keywords: muonclip, kimi-linear, agent-swarms, attention-residuals, vision-text
  • Core insight: Scaling frontier AI requires simultaneous advancement in token efficiency optimizers, linear attention architectures, and multi-agent reinforcement learning.

Core insights

6
Mechanismhigh noveltymoderate evidence

Reward shaping for orchestrator-driven agent swarms requires at least three distinct reward signals — instantiation, finish, and outcome — specifically to prevent two degenerate behaviors: serial collapse (the orchestrator does everything itself) and spurious parallelism (spawning agents that contribute nothing).

Why it matters

Multi-agent speedups are not automatic; without explicit rewards for when to spawn and when to terminate sub-agents, the orchestrator degenerates into either a single serial agent or wasteful fan-out. This tells builders that the decomposition policy itself is a learned, reward-shaped artifact rather than a static planner rule, and that harnesses need per-behavior reward instrumentation, not just a final task-success signal.

Generalization

Any orchestrator/worker architecture (including non-RL planners) needs explicit incentives or heuristics for spawn justification and termination, otherwise fan-out cost is paid without parallelism benefit.

Reinforcement learning for agent swarms requires instantiation, finish, and outcome rewards to prevent serial collapse and spurious parallelism.
Open source video
Agent swarms save significant execution time on complex tasks compared to single agents.
Open source video
Architecturehigh noveltymoderate evidence

Kimi Linear uses a hybrid stack with a 3:1 ratio of Kimi Delta Attention (linear) layers to full attention layers, with a chunkwise formulation, and reportedly beats full-attention baselines while cutting memory overhead.

Why it matters

The design claim is that you do not need pure linear attention or pure full attention: a fixed ratio mixture preserves expressivity while getting linear-scaling economics. For serving stacks this shifts the KV-cache/memory budget and the time-per-output-token profile at long context, so the scheduler, paging, and batching assumptions of a standard transformer runtime no longer hold unchanged.

Generalization

Hybrid layer-ratio architectures are a tunable knob between expressivity and memory/compute cost, applicable wherever a recurrent or compressed-state layer can substitute for some full-attention layers.

combining Delta Attention with a 3:1 ratio of linear to full attention layers to achieve linear scaling and superior long-context performance
Open source video
Kimi Linear architecture achieves better performance than full attention baselines while cutting memory overhead.
Open source video
Drastically reduces time-per-output-token (TPOT) at long context lengths.
Open source video
Failure Modehigh noveltymoderate evidence

Scaling the Muon optimizer to a 1-trillion-parameter model caused max logits to explode; QK-Clip constrains query and key projections in attention heads to restore stability, enabling ~2x token efficiency over AdamW without divergence.

Why it matters

Second-order/orthogonalized optimizers buy token efficiency but introduce a specific, diagnosable failure mode (attention logit blowup) that must be handled architecturally rather than by lowering learning rate. This gives practitioners a concrete instability signature to watch (max logits) and a targeted intervention (clipping Q/K projections) rather than a generic 'training is unstable' heuristic.

Generalization

Aggressive optimizers can be stabilized by constraining the specific parameter subspace that produces the blowup, instead of globally reducing step size and giving back the efficiency gain.

Scaling a 1-trillion parameter model with Muon and encountering max logit explosions solved by QK-Clip.
Open source video
Training instability during large-scale Muon training can be resolved using QK-Clip.
Open source video
Mechanismhigh noveltymoderate evidence

Attention Residuals (AttnRes) are framed as an LSTM residual rotated into attention space and applied as block attention residuals; they are reported to improve token efficiency by 24% and reduce training loss.

Why it matters

It reframes residual connections as a form of recurrence that can be rotated/generalized rather than a fixed identity skip. If a residual-connection variant alone yields double-digit token efficiency gains, the residual stream is an under-explored design surface, not settled infrastructure — worth prototyping before adding more data or parameters.

Generalization

Cross-pollinating recurrence structure into attention topology (and vice versa) is a productive source of architectural gain, and residual-path redesign is measurable in token efficiency, not just in loss curves.

Attention Residuals improve token efficiency by 24% and reduce training loss.
Open source video
introduces Attention Residuals (AttnRes) as a rotation of LSTM residuals into attention space
Open source video
Empirical Resultmedium noveltymoderate evidence

Early-fusion (native joint) vision-text pretraining outperforms late fusion, and vision reinforcement learning improves textual performance — i.e., cross-modal transfer flows from vision training into text capability.

Why it matters

This challenges the common practice of bolting a vision encoder onto a frozen or lightly tuned text model with adapters. It implies multimodal capability is a pretraining-pipeline decision, not a post-hoc integration layer, and that multimodal RL is not merely additive to text quality.

Generalization

Modality fusion at pretraining time can act as a regularizer/capability multiplier for the primary modality, so modality investment should be budgeted as core training rather than as a downstream feature.

Early fusion vision-text joint pretraining yields better results than late fusion.
Open source video
Vision reinforcement learning improves textual performance (cross-modal transfer).
Open source video
first open model with native joint vision-text pretraining
Open source video
Mental Modelmedium noveltymoderate evidence

Kimi frames scaling as three parallel dimensions — token efficiency, context length, and agent swarms — rather than a single axis of parameters or FLOPs, with token efficiency treated as a data multiplier that raises the ceiling of achievable intelligence under a fixed high-quality data budget.

Why it matters

It reframes the binding constraint as data quality/quantity rather than compute, which changes investment priority: optimizer and architecture work that halves tokens-per-unit-intelligence is equivalent to doubling the usable corpus, and is therefore cheaper than acquiring new data. It also gives a portfolio framing for roadmap decisions across training, serving, and agent layers.

Generalization

When a resource is capped (data, budget, latency), efficiency gains in that resource should be evaluated as multipliers on the capped quantity, not as incremental speedups.

Token efficiency improves the upper bound of model intelligence, acting like a data multiplier.
Open source video
Scaling must happen across multiple dimensions: token efficiency, context length, and agent swarms.
Open source video

Deep dives

5

Cross-domain transfer and reward hacking of the three swarm reward signals

Research question

Do the instantiation/finish/outcome reward signals produce spawn-efficient orchestration on task families other than the reported parallel-research workloads, or do they overfit and invite reward hacking?

Why

Spawn/terminate behavior is a learned policy with its own degenerate regimes; if the reward mix only works on one task family, teams copying the recipe will pay fan-out cost without parallelism benefit, and their task-success metric will not reveal it.

Reinforcement learning for agent swarms requires instantiation, finish, and outcome rewards to prevent serial collapse and spurious parallelism.
Open source video
Execution time vs task complexity graphs showing up to 4.5x speedups.
Open source video
Source video

Attribution of Muon's claimed ~2x token efficiency

Research question

How much of the ~2x token efficiency over AdamW comes from Muon's orthogonalized updates versus weight decay and the RMS-update scheme, and does the ranking survive changes in scale and data distribution?

Why

If most of the gain comes from regularization rather than the optimizer, teams can capture it on a standard optimizer without inheriting the max-logit stability tooling; if not, the efficiency claim justifies the full MuonClip stack and its operational cost.

Muon optimizer provides ~2x token efficiency improvement over AdamW.
Open source video
Token efficiency improves the upper bound of model intelligence, acting like a data multiplier.
Open source video
Source video

QK-Clip mechanism, clip-threshold sensitivity, and its cost to attention expressivity

Research question

What subspace does QK-Clip constrain, and does clipping query/key projections degrade attention sharpness, long-context retrieval, or interact with hybrid linear-attention layers?

Why

QK-Clip is what makes the Muon token-efficiency gain usable at trillion-parameter scale; if it silently refunds that gain as reduced long-context retrieval quality, the two headline results cannot be treated as independent wins.

Scaling a 1-trillion parameter model with Muon and encountering max logit explosions solved by QK-Clip.
Open source video
Training instability during large-scale Muon training can be resolved using QK-Clip.
Open source video
Source video

Attention Residuals composed with hybrid 3:1 KDA/full-attention stacks

Research question

Do AttnRes's 24% token-efficiency gains persist when residual topology is placed over recurrent-state (KDA) layers, or do the gains assume homogeneous full-attention blocks?

Why

Both AttnRes and Kimi Linear are recurrence-flavored modifications to the same stack; if they are reported separately with no interaction ablation, an architecture team cannot know whether to adopt one, both, or neither.

Attention Residuals improve token efficiency by 24% and reduce training loss.
Open source video
Kimi Linear architecture achieves better performance than full attention baselines while cutting memory overhead.
Open source video
Source video

Isolating cross-modal transfer from vision RL into text capability

Research question

Does vision RL on a shared backbone improve text benchmarks beyond an equal-compute text-only RL control, or is the gain simply additional RL compute applied to the same parameters?

Why

If the transfer is real it justifies budgeting modality work as core training rather than as a downstream feature; if it is a compute artifact, teams that add vision RL expecting text gains will misallocate training budget.

Early fusion vision-text joint pretraining yields better results than late fusion.
Open source video
Vision reinforcement learning improves textual performance (cross-modal transfer).
Open source video
Source video

Article ideas

4

Your Orchestrator Is a Learned Policy, Not a Planner

Once task decomposition is trained with spawn and terminate rewards, the decomposition policy acquires its own failure modes — serial collapse and spurious parallelism — so agent harnesses need per-agent lifecycle telemetry and cost attribution at the same priority as the final task score; teams that measure only task success will pay for parallelism they never received.

Angle

Translate the reward-shaping result into concrete harness instrumentation requirements: spawn/terminate accounting, marginal-contribution logging, and cost-per-quality dashboards.

Source video

Token Efficiency Is a Data-Acquisition Strategy, Not an Optimizer Tuning Detail

If halving tokens-per-unit-intelligence is equivalent to doubling the usable high-quality corpus, then optimizer and architecture work should be budgeted against data-acquisition economics — and headline efficiency multipliers must ship with ablations separating the optimizer from its weight-decay and update-normalization components.

Angle

Reframe efficiency research as a multiplier on a capped resource and demand matched-baseline ablations before treating any headline number as portable.

Source video

Hybrid 3:1 Attention Breaks Your Serving Stack's Assumptions

A mixture of linear-to-full attention layers turns time-per-output-token and memory overhead into configurable properties, which means schedulers, paging, batching, and chunk-boundary kernels must be re-derived per layer class instead of inherited from a uniform-transformer runtime.

Angle

Walk through which runtime assumptions (single KV-growth model, homogeneous kernels, fixed TPOT profile) stop holding and what must replace them.

Source video

The Residual Stream Is Not Settled Infrastructure

If reshaping residuals buys double-digit token efficiency, residual topology deserves the ablation budget currently spent on layer count and width — but AttnRes must first be validated on hybrid recurrent stacks, since a gain proven only on full-attention blocks is not yet a general-purpose gain.

Angle

Argue the residual path is an under-explored design surface and specify the ablation matrix needed to promote it from curiosity to default.

Source video

Project ideas

4

Spawn Ledger

beyond-evals

Instrumenting multi-agent runs with per-agent marginal-contribution accounting will expose redundant-agent rates and true cost per unit quality that final-task-success scoring hides, and these figures will differ systematically across reward configurations.

Proof of concept

Wrap an orchestrator/worker harness so every spawn, termination, sub-agent token cost, and each sub-agent's marginal effect on the outcome is logged; replay one task suite under outcome-only reward versus instantiation+finish+outcome reward.

Measurement

Spawn efficiency (contributing agents divided by spawned agents), redundant-agent rate, cost per unit of task quality, and speedup versus a single agent at matched total cost.

Source video

Logit Sentinel

movement-lab

Max-logit trajectory is a measurable early-warning signal of large-scale optimizer divergence, and constraining query/key projections restores stability across seeds without degrading long-context retrieval beyond a bounded clip-threshold window.

Proof of concept

Instrument per-step max logits during Muon-style training runs at multiple scales, sweep Q/K clip thresholds, and evaluate long-context retrieval and downstream loss at each threshold.

Measurement

Steps-to-divergence and max-logit slope per configuration; retrieval and loss deltas as a function of clip threshold and random seed.

Source video

Efficiency Attribution Harness

movement-lab

A measurable share of the reported ~2x token efficiency over AdamW is attributable to weight decay and RMS update normalization rather than to orthogonalized updates, distinguishable as tokens-to-target-loss at matched settings.

Proof of concept

Run controlled ablations at several model scales comparing AdamW with matched weight decay and RMS normalization against Muon, with tokens-to-target-loss as the dependent variable.

Measurement

tokens-to-target-loss ratio per arm per scale, divergence rate per arm, and the portion of the reported multiplier retained once regularization is matched.

Source video

Hybrid-KV Planner

new

A runtime that models a 3:1 linear-to-full attention stack as two layer classes with distinct memory and state-update behavior predicts long-context time-per-output-token and peak memory within a small error margin, where a uniform-transformer model does not.

Proof of concept

Build a scheduler/paging simulator with separate KV-cache growth for full-attention layers and fixed-size recurrent state for linear layers, then validate predictions against measured TPOT and memory across context lengths.

Measurement

Predicted versus observed TPOT and peak memory at long context, and the deviation of the uniform-transformer baseline's predictions from the same measurements.

Source video

Architectural implications

4

Attention layer composition is no longer homogeneous across the stack.

Before

Uniform full self-attention in every layer, with a single KV-cache growth model and quadratic long-context cost.

After

A 3:1 interleaving of Kimi Delta Attention (chunkwise, recurrent-state) with full attention layers.

Consequence

Serving runtimes must account for mixed layer types: memory budgeting, chunk boundaries, matrix-inversion/state-update kernels, and batching/parallelism strategies differ per layer class, and long-context TPOT becomes a configurable rather than fixed property.

Source video

Multi-agent swarms are trained end-to-end with reward shaping over orchestration behavior, not assembled from a static planner.

Before

A single agent with a long context and hand-written decomposition logic.

After

An orchestrator that spawns specialist sub-agents in parallel, optimized with instantiation, finish, and outcome rewards.

Consequence

Task decomposition becomes a learned policy with its own failure modes (serial collapse, spurious parallelism), so harnesses need per-agent lifecycle telemetry, spawn/terminate accounting, and cost attribution to detect useless parallelism.

Source video

The residual stream itself becomes a design surface.

Before

Standard identity residual (x + f(x)) repeated uniformly across layers.

After

Attention residuals / block attention residuals, motivated as an LSTM residual rotated into attention space.

Consequence

Layer-level memory and gradient-flow properties change, so architecture search and ablation budgets shift from layer count/width toward residual topology.

Source video

Multimodal integration moves from the post-training/integration layer into pretraining.

Before

Late fusion: adapters or projection heads attached to a text-pretrained backbone.

After

Early fusion: native joint vision-text pretraining.

Consequence

Data pipeline, tokenizer, sequence packing, and training-time modality mixing become coupled concerns; vision RL can feed back into text quality, so modality experiments stop being isolated feature work.

Source video

Tradeoffs and failure modes

4

Muon-style second-order optimization at scale

Benefit

~2x token efficiency over AdamW, effectively multiplying a limited high-quality data budget.

Cost or risk

Max logit explosion / training divergence at trillion-parameter scale, requiring QK-Clip intervention and presumably extra stability tooling and monitoring.

Training instability during large-scale Muon training can be resolved using QK-Clip.
Open source video
Source video

Linear attention hybridization

Benefit

Memory overhead reduction and superior long-context performance versus full-attention baselines, plus lower TPOT at long context.

Cost or risk

Reduced expressivity of recurrent-state layers plus implementation complexity (chunkwise formulation, matrix inversion), and the 3:1 ratio is an added tuning dimension that must be validated per scale.

combining Delta Attention with a 3:1 ratio of linear to full attention layers to achieve linear scaling and superior long-context performance
Open source video
Source video

Parallel agent swarms

Benefit

Up to 4.5x speedups on complex tasks, e.g. a 100-page literature review via parallel researchers and fact-checkers.

Cost or risk

Reward hacking toward degenerate regimes — serial collapse (no real parallelism) or spurious parallelism (agents spawned without contributing) — which inflates cost and latency without quality gain.

Reinforcement learning for agent swarms requires instantiation, finish, and outcome rewards to prevent serial collapse and spurious parallelism.
Open source video
Source video

Early-fusion multimodal pretraining

Benefit

Better results than late fusion, with cross-modal transfer from vision RL into textual performance.

Cost or risk

Commitment to joint modality data pipeline and pretraining from the start instead of cheaper adapter-based retrofitting; abandoning the single-modality backbone path limits reuse of existing text-only checkpoints.

Early fusion vision-text joint pretraining yields better results than late fusion.
Open source video
Source video

Open questions

5

How much of the reported ~2x token efficiency comes from Muon itself versus weight decay and the consistent RMS-update scheme, and does the gain hold as model scale and data distribution change?

Why unresolved

The summary reports a headline efficiency figure and scaling-law graph but no ablation separating the optimizer from its regularization and update-normalization components.

Research direction

Run controlled ablations at multiple scales comparing Muon vs AdamW with matched weight decay and RMS normalization, measuring tokens-to-target-loss as the dependent variable.

Source video

What is QK-Clip actually constraining, and does clipping Q/K projections degrade attention expressivity, long-context retrieval, or interact with Kimi Linear's hybrid layers?

Why unresolved

The summary states that QK-Clip prevents max-logit divergence but gives no mechanism detail, no hyperparameter sensitivity, and no quality-cost measurement.

Research direction

Instrument max-logit trajectories and downstream benchmark deltas as a function of clip threshold, including long-context retrieval (RULER-style) tasks where attention sharpness matters.

Source video

Do the instantiation/finish/outcome rewards transfer to other swarm workloads, or do they overfit to the reported research-style tasks and invite reward hacking?

Why unresolved

Only one task family (parallel research/fact-checking) is described, with no cross-domain reward ablations or adversarial task analysis.

Research direction

Build a benchmark suite of orchestrator tasks spanning software engineering, data analysis, and adversarial prompts, measuring spawn efficiency, redundant-agent rate, and reward-hack frequency per reward configuration.

Source video

Can Attention Residuals compose with linear/recurrent attention layers, or do the 24% token-efficiency gains assume full-attention blocks?

Why unresolved

Attention Residuals and Kimi Linear are presented as separate innovations with no reported interaction or combined configuration.

Research direction

Ablate AttnRes on hybrid 3:1 KDA/full stacks versus pure full-attention stacks, tracking token efficiency, loss, and long-context benchmarks.

Source video

Does vision RL improving textual performance replicate beyond this model, and is it a genuine capability transfer or a benchmark artifact?

Why unresolved

The transfer claim is stated without mechanism or control experiments, and the summary provides no quantitative cross-modal transfer measurements.

Research direction

Run text-only evaluations before and after vision RL on identical checkpoints, with modality-matched and modality-mismatched RL baselines to isolate the transfer effect.

Source video

Key claims

8
comparativeVerification needed

The Muon optimizer provides roughly 2x token efficiency improvement over AdamW.

Evidence

Muon optimizer provides ~2x token efficiency improvement over AdamW.

Question

At what model scales and data budgets was the 2x figure measured, and against what exact AdamW configuration and learning-rate schedule?

Source video
causalVerification needed

QK-Clip resolves the max-logit explosion that occurs when scaling Muon to trillion-parameter models.

Evidence

Training instability during large-scale Muon training can be resolved using QK-Clip.

Question

Does QK-Clip eliminate divergence across scales and seeds without degrading downstream quality, or merely delay it?

Source video
comparativeVerification needed

A 3:1 ratio of Kimi Delta Attention layers to full attention layers outperforms full-attention baselines while reducing memory overhead.

Evidence

Kimi Linear architecture achieves better performance than full attention baselines while cutting memory overhead.

Question

On which benchmarks and context lengths was superiority demonstrated, and is the 3:1 ratio optimal or simply chosen?

Source video
causalVerification needed

Attention Residuals improve token efficiency by 24% and reduce training loss.

Evidence

Attention Residuals improve token efficiency by 24% and reduce training loss.

Question

Is the 24% token-efficiency gain measured against a matched baseline with identical parameter count, data, and compute, and does it persist at larger scale?

Source video
comparativeVerification needed

Early-fusion vision-text joint pretraining yields better results than late fusion.

Evidence

Early fusion vision-text joint pretraining yields better results than late fusion.

Question

On which tasks and with what parameter/compute-matched late-fusion baseline was this comparison made?

Source video
causalVerification needed

Vision reinforcement learning improves textual performance via cross-modal transfer.

Evidence

Vision reinforcement learning improves textual performance (cross-modal transfer).

Question

Is the text improvement attributable to vision RL specifically, or to additional RL compute applied to a shared backbone?

Source video
opinionVerification needed

Agent swarms achieve up to 4.5x speedups on complex tasks relative to single agents.

Evidence

Execution time vs task complexity graphs showing up to 4.5x speedups.

Question

At what task complexity does the speedup appear, and what is the total token/cost overhead including redundant sub-agents?

Source video
factualVerification needed

Kimi K2.5 is the first open model with native joint vision-text pretraining.

Evidence

first open model with native joint vision-text pretraining

Question

What counts as 'native joint pretraining' versus prior open multimodal models, and is the 'first' claim verifiable against public model cards?

Source video

Connections

5