Imagination in Action · Published 2026-09-21

What Comes After Transformers? | Conversations in Action

privateReview: unreviewedConfidence: high
transformerssemanticsscalingagentscomplexitysemantic-representationstate-space-modelspost-traininglong-contextmulti-agentevaluation-metrics
Open on YouTube ↗

Summary

Overview

  • Speaker: Yoav Shoham
  • Channel: Imagination in Action
  • Main topic: The future of artificial intelligence beyond transformers and large language models
  • Purpose: To explore deep foundational AI research, limitations of LLMs, semantic knowledge representation, and the future evolution of AI architectures. In this conversation, AI pioneer Yoav Shoham discusses the limitations of current large language models, the role of semantic representation, the evolution of AI index metrics, and what architectures might eventually succeed transformers.

Topic Map

Limitations of Large Language Models and Semantic Representation

  • Explanation: Yoav Shoham argues that current LLMs lack true semantic representation, leading to brilliant outputs mixed with garbage due to a lack of underlying structural grounding.
  • Key claims:
    • LLMs are missing semantic representation.
    • Current systems are totally brilliant yet total garbage some of the time.
    • Knowledge representation and semantic depth are fundamental roadblocks to general intelligence.
  • Examples:
    • Historical parallels to 1990s knowledge representation research.
    • Automatic completion on steroids producing impressive text without deep reasoning guarantees.
  • Terminology:
    • LLMs
    • semantic representation
    • knowledge representation
    • general intelligence
  • Why it matters: Understanding these architectural limitations helps determine whether current scaling trends are sufficient to reach artificial general intelligence or if structural paradigms must change.

Industry Metrics and Measuring ROI vs. Token Costs

  • Explanation: The AI industry frequently measures token costs and model scale rather than actual return on investment and intellectual rigor.
  • Key claims:
    • The AI industry measures the cost of tokens rather than ROI.
    • The hardest thing is knowing what to want.
    • Scaling parameters and tokens is insufficient without addressing fundamental knowledge representation.
  • Examples:
    • Comparing token consumption metrics to real business value delivery.
  • Terminology:
    • ROI
    • tokens
    • scaling
    • parameters
  • Why it matters: Misaligned industry metrics lead to inflated hype and misallocated resources in AI development.

The AI Index and Tracking Progress

  • Explanation: Discussion on the AI Index project, its origins at Stanford with Eric Horvitz, and the challenges of creating reliable macro indices for artificial intelligence.
  • Key claims:
    • The AI Index tracks global AI activity and progress.
    • Indices necessarily lose information to provide macro clarity.
    • Comparing global AI metrics is difficult due to varying data reliability.
  • Examples:
    • Parallels to GDP as an imperfect yet useful macroeconomic index.
    • Global vibrancy index comparing parameter counts, compute, and investment.
  • Terminology:
    • AI Index
    • global vibrancy index
    • macro metrics
    • benchmarking
  • Why it matters: Robust tracking mechanisms are essential to understand the pace and global distribution of AI capabilities.

What Comes After Transformers?

  • Explanation: An exploration of whether transformers are an off-ramp or stepping stone to AGI, and what architectural improvements or alternatives might follow.
  • Key claims:
    • Transformers have quadratic complexity with context length.
    • Instruction tuning and post-training delivered massive capability gains without parameter increases.
    • Hybrid architectures and state-space models represent potential future directions.
  • Examples:
    • Mamba and state-space models as linear complexity alternatives.
    • Vanilla transformers versus decoder-only architectures.
  • Terminology:
    • transformers
    • state-space models
    • Mamba
    • instruction tuning
    • quadratic complexity
  • Why it matters: Overcoming the quadratic scaling bottlenecks of transformers is critical for scaling long-context reasoning and agentic systems.

Key Points

Lack of Semantic Grounding

  • Explanation: LLMs operate primarily on statistical token prediction without rigorous semantic grounding.
  • Evidence: Models generate plausible-sounding falsehoods or hallucinations due to lacking underlying semantic checks.
  • Practical implication: Developers must supplement LLMs with symbolic reasoning or retrieval systems to ensure reliability.

Scaling Laws vs. Fundamental Architecture

  • Explanation: Simply scaling parameters and training data may hit fundamental barriers without structural breakthroughs.
  • Evidence: Historical shifts from encoder-decoder models to decoder-only and alternative architectures.
  • Practical implication: Research investment should balance compute scaling with architectural innovation in memory and reasoning.

Multi-Agent Systems and Human Economics

  • Explanation: As millions of AI agents interact, economic lessons from multi-agent systems and game theory become vital.
  • Evidence: Game theory and multi-agent simulation provide frameworks for understanding agent incentives and safety.
  • Practical implication: Designing robust multi-agent systems requires anticipating emergent economic behaviors and strategic dynamics.

Frameworks, Models & Processes

AI Indexing Framework

  • How it works: Aggregates diverse parameters like compute, investment, publications, and performance into macro indicators.
  • Components:
    • Compute metrics
    • Publication volume
    • Investment tracking
    • Performance benchmarks
  • When to use: Evaluating macro-level progress and global standing in artificial intelligence.

Examples & Case Studies

The transition from early word embeddings (Word2Vec) to modern transformer architectures.

  • Illustrates: The historical evolution of NLP representation learning.
  • Lesson: Architectural shifts happen when existing paradigms hit performance and scaling plateaus.

Actionable Takeaways

  • Immediate:
    • Look beyond token cost and measure true ROI for AI implementations.
    • Recognize the distinction between statistical prediction and semantic understanding in LLMs.
  • Strategic:
    • Invest in alternative architectures addressing transformer quadratic complexity limitations.
    • Incorporate game theory and multi-agent economics into AI system design.
  • Questions to investigate:

Claims Worth Verifying

  • LLMs are missing semantic representation and rely purely on statistical token correlations. (theoretical)
  • State-space models like Mamba offer linear complexity alternatives to transformer quadratic scaling. (technical)

Notable Quotes

"Even thinking about us versus machines is missing the point. Us and machines will be one!" (at 0:06) "We're missing semantic representation. They're totally brilliant, and then not just a little wrong, but total garbage some of the time." (at 0:16) "Measuring the cost of tokens rather than ROI." (at 0:29)

Compressed Summary

  • LLMs lack deep semantic representation and structural grounding.
  • Industry metrics often focus on token costs rather than true ROI.
  • Transformer quadratic complexity necessitates exploration of alternative architectures like state-space models.
  • Multi-agent dynamics and game theory will become critical as autonomous AI agents proliferate.
  • Keywords: transformers, semantics, scaling, agents, complexity
  • Core insight: While large language models have achieved unprecedented scaling success, overcoming their foundational limitations in semantic representation requires exploring new architectural paradigms beyond standard transformers.

Core insights

6
Failure Modemedium noveltymoderate evidence

LLM failure is bimodal rather than graceful: the same statistical mechanism that produces brilliant output also produces 'total garbage' because no semantic representation exists to check the output. Reliability cannot be inferred from fluency or model confidence; it must be supplied by an external structural component (symbolic reasoning, retrieval, or explicit checks).

Why it matters

If errors are catastrophic rather than graded, then guardrails keyed to model confidence, logprobs, or self-verification are architecturally unsound; correctness has to live outside the generator as a separate verifiable layer.

Generalization

Any system whose competence is statistical rather than grounded needs independently verifiable intermediate representations between generation and action.

We're missing semantic representation. They're totally brilliant, and then not just a little wrong, but total garbage some of the time.
Open source video
Models generate plausible-sounding falsehoods or hallucinations due to lacking underlying semantic checks.
Open source video
Empirical Resultmedium noveltymoderate evidence

Instruction tuning and post-training delivered massive capability gains without any increase in parameters, meaning a large share of capability budget can be spent in the post-training pipeline rather than in pretraining scale. This relocates engineering investment from model size toward tuning data, objectives, and post-training harnesses.

Why it matters

It reframes the build-vs-scale decision: teams can capture capability through post-training assets (data, tuning loops, evaluators) that they can own, instead of competing on parameter count.

Generalization

When capability gains come from a post-hoc transformation stage, the differentiator is the transformation pipeline and its evaluation, not the base artifact.

Instruction tuning and post-training delivered massive capability gains without parameter increases.
Open source video
Architecturemedium noveltymoderate evidence

Transformer attention cost grows quadratically with context length, which makes context the binding architectural constraint for long-context reasoning and agentic systems. Linear-complexity state-space alternatives (Mamba) and hybrid architectures are framed as the candidate successors, not as drop-in upgrades.

Why it matters

Memory/context in an agent runtime should be treated as a selectable component with a cost curve, not as an ever-growing buffer; the choice of memory primitive changes the feasible context regime and the cost model.

Generalization

When a primitive's cost grows superlinearly in the resource agents most need (context), the memory abstraction becomes a first-class architecture decision rather than a configuration parameter.

Transformers have quadratic complexity with context length.
Open source video
Mamba and state-space models as linear complexity alternatives.
Open source video
Hybrid architectures and state-space models represent potential future directions.
Open source video
Mental Modelhigh noveltymoderate evidence

The dominant industry metric is token cost, not return on investment, and the harder unsolved problem is specifying what to want in the first place. Cost-per-token optimizations can therefore be locally optimal while misallocating effort relative to delivered value.

Why it matters

Observability and cost accounting should be keyed to task-level outcomes rather than consumption metrics; otherwise optimization pressure flows toward cheaper tokens instead of more valuable workflows.

Generalization

Any measurement layer that tracks resource consumption instead of value delivered will bias system design toward the measured quantity.

The AI industry measures the cost of tokens rather than ROI.
Open source video
Measuring the cost of tokens rather than ROI.
Open source video
The hardest thing is knowing what to want.
Open source video
Predictionmedium noveltyweak evidence

As agent populations scale into the millions, design problems shift from single-agent correctness to emergent strategic and economic dynamics, with game theory and multi-agent simulation proposed as the frameworks for anticipating incentives and safety risks.

Why it matters

At agent scale, verification moves from per-output checks to incentive analysis and simulation of emergent behavior, which is a different discipline than testing a single agent loop.

Generalization

Once many autonomous actors share an environment, system properties are emergent at the population level and cannot be validated by inspecting individual actors.

As millions of AI agents interact, economic lessons from multi-agent systems and game theory become vital.
Open source video
Designing robust multi-agent systems requires anticipating emergent economic behaviors and strategic dynamics.
Open source video
Tradeoffmedium noveltymoderate evidence

Aggregate AI indices (compute, investment, publications, benchmark performance) necessarily discard information to achieve macro clarity, and cross-region comparison is further limited by uneven data reliability — an explicit trade-off in measurement design rather than a fixable data problem.

Why it matters

Evaluation and benchmarking infrastructure should be designed with the index granularity chosen deliberately; a leaderboard-style macro score cannot retain the per-dimension detail needed for engineering decisions.

Generalization

Every aggregation step in an evaluation pipeline trades diagnostic resolution for comparability, and the loss is structural rather than incidental.

Indices necessarily lose information to provide macro clarity.
Open source video
Comparing global AI metrics is difficult due to varying data reliability.
Open source video
Global vibrancy index comparing parameter counts, compute, and investment.
Open source video

Deep dives

5

External semantic verification as the reliability layer for statistical generators

Research question

Does routing probabilistic generation through an external structural check (retrieval-grounded or symbolic) reduce catastrophic error rates more cheaply than scaling or confidence-based self-verification?

Why

If errors are bimodal rather than graded, then guardrails keyed to logprobs or model self-critique are architecturally unsound; correctness must be owned by a separate verifiable layer, which turns the generator/checker interface into a first-class design boundary.

LLMs are missing semantic representation.
Open source video
Current systems are totally brilliant yet total garbage some of the time.
Open source video
Knowledge representation and semantic depth are fundamental roadblocks to general intelligence.
Open source video
Automatic completion on steroids producing impressive text without deep reasoning guarantees.
Open source video
Source video

Post-training saturation curves at fixed parameter count

Research question

At fixed base-model size, what is the shape of the capability-vs-post-training-data/compute curve, and where does the post-training lever flatten?

Why

Post-training relocated a large share of the capability budget out of pretraining scale and into an owned pipeline; without a saturation curve, teams cannot decide when to invest in tuning data versus when to abandon architectural work that scaling cannot reach.

Instruction tuning and post-training delivered massive capability gains without parameter increases.
Open source video
Scaling parameters and tokens is insufficient without addressing fundamental knowledge representation.
Open source video
Source video

Memory as a pluggable component with an explicit cost curve

Research question

At matched capability, how do linear-complexity state-space and hybrid memory backends compare to quadratic transformer context on long-horizon agentic reasoning quality, not just asymptotics?

Why

If attention cost grows superlinearly in the resource agents need most, memory abstraction boundaries must be designed before model choice is fixed, and long-horizon agent design becomes a cost-modeling problem rather than a context-window configuration.

Transformers have quadratic complexity with context length.
Open source video
Mamba and state-space models as linear complexity alternatives.
Open source video
Hybrid architectures and state-space models represent potential future directions.
Open source video
Source video

Outcome-keyed cost accounting versus consumption metrics

Research question

What task-level outcome metric (cost per completed workflow, cost per accepted output) changes engineering decisions relative to token-cost accounting, and does it produce better delivered value?

Why

Measurement layers that track resource consumption rather than value delivered bias system design toward the measured quantity, so token-cost optimization can be locally optimal while misallocating effort; the harder unsolved problem remains specifying what to want.

The AI industry measures the cost of tokens rather than ROI.
Open source video
The hardest thing is knowing what to want.
Open source video
Comparing token consumption metrics to real business value delivery.
Open source video
Source video

Pre-deployment verification of emergent multi-agent dynamics

Research question

Do population-scale agent deployments exhibit strategic or economic behaviors that single-agent and small-group testing fails to predict, and can incentive simulation surface them in advance?

Why

Once many autonomous actors share an environment, system properties are emergent at the population level, so verification shifts from per-output checks to incentive analysis and simulation — a different discipline than testing one agent loop.

As millions of AI agents interact, economic lessons from multi-agent systems and game theory become vital.
Open source video
Designing robust multi-agent systems requires anticipating emergent economic behaviors and strategic dynamics.
Open source video
Source video

Article ideas

4

Fluency Is Not a Correctness Signal: Why Confidence-Keyed Guardrails Are Architecturally Unsound

Because model failure is bimodal rather than graceful, any guardrail keyed to model confidence, logprobs, or self-verification is checking the wrong object; correctness must be supplied by an external structural layer that the generator is forced to pass through.

Angle

Take the 'totally brilliant yet total garbage' observation to its engineering conclusion: prove that confidence does not bound error, then specify the interface between probabilistic generation and deterministic checking as an explicit architectural boundary.

Source video

Your Dashboard Optimizes Tokens; Your Business Buys Outcomes

Token-cost accounting is a proxy that reshapes design pressure toward cheaper consumption rather than greater delivered value, so AI systems should be instrumented at task-level outcomes — and the unmeasured harder problem is deciding what to want.

Angle

Contrast a consumption-metric stack with an outcome-metric stack on the same workload, showing where the two diverge and which decisions each one drives.

Source video

Memory Is an Architecture Decision, Not a Context-Window Setting

When the dominant primitive's cost grows superlinearly in the resource agents need most, memory stops being a configuration parameter and becomes a first-class component with a cost curve that must be chosen before the model is fixed.

Angle

Reframe agent memory as a selectable backend (quadratic attention, linear state-space, hybrid) with distinct feasible context regimes and cost models, and show why the abstraction boundary comes first.

Source video

The Post-Training Moat: Where Capability Budget Actually Goes

Large capability gains at fixed parameter count mean the differentiator is the post-hoc transformation pipeline and its evaluation harness, not the base artifact — which changes build-versus-scale strategy from buying model size to owning tuning data, objectives, and evaluators.

Angle

Argue that teams should invest in owned post-training assets and the evaluation infrastructure to measure instruction-following gains, while flagging that the lever's saturation point is unmeasured.

Source video

Project ideas

4

Semantic Gatehouse

gatehouse

Inserting an external structural verification layer (retrieval-grounded or symbolic checks) between generation and action reduces catastrophic error rate on held-out tasks more than equal-cost investment in a larger or better-prompted generator.

Proof of concept

Wrap a fixed base model in a checker that must validate claims against a retrieval/symbolic intermediate representation before any output or tool call is emitted; run against a bare baseline on a task suite with known catastrophic failure cases.

Measurement

Catastrophic error rate (as distinct from graded error), cost per accepted output, and false-rejection rate at matched task throughput.

Source video

Post-Training Saturation Lab

movement-lab

At fixed base-model size and fixed evaluation, capability gains from post-training follow a saturating curve in tuning data/compute, and the flattening point is measurable and predictable.

Proof of concept

Hold the base model and eval suite constant while sweeping instruction-tuning data volume and post-training compute across matched runs; plot capability against post-training investment instead of parameter count.

Measurement

Capability slope versus post-training data/compute at fixed parameters; identification of the investment level at which marginal gain approaches zero.

Source video

Outcome-Cost Accounting Harness

beyond-evals

Teams instrumented with task-level outcome metrics (cost per completed workflow, cost per accepted output) make different architecture and routing decisions than teams instrumented with token-cost metrics, and achieve lower cost per accepted outcome.

Proof of concept

Run two parallel instrumentation modes — token consumption versus outcome accounting — over the same agent workflows, log the design decisions each metric set induces, and compare resulting cost per accepted outcome.

Measurement

Cost per completed workflow and cost per accepted output under each instrumentation mode, plus divergence count in induced design decisions.

Source video

Multi-Agent Incentive Simulator

new

Population-scale agent deployments exhibit emergent strategic and economic behaviors that are not predicted by single-agent or small-group testing, and incentive modeling in simulation surfaces those behaviors before production rollout.

Proof of concept

Build a simulated environment with many interacting agents under explicit incentive structures, then compare predicted outcomes against per-agent test extrapolations and flag divergences.

Measurement

Divergence between simulated emergent population behavior and single-agent test predictions; count of strategic failure modes surfaced prior to deployment.

Source video

Architectural implications

3

Semantic grounding is treated as a missing structural layer rather than a property that emerges from scale.

Before

Correctness is assumed to be a property of the generation model, checked by confidence or self-review.

After

Correctness is owned by an external semantic/symbolic or retrieval layer that the generator must pass through.

Consequence

The LLM becomes one component in a verification pipeline, and the interface between probabilistic generation and deterministic checking becomes an explicit design boundary.

Source video

Context cost scales quadratically with transformer context length, while state-space and hybrid models offer linear alternatives.

Before

Agent memory is implemented as ever-larger context windows on a single transformer.

After

Memory is a pluggable component with an explicit cost curve, potentially a non-transformer or hybrid module.

Consequence

Long-context and long-horizon agent design becomes a cost-modeling problem, and memory abstraction boundaries must be designed before model choice is fixed.

Source video

Post-training, not parameter scaling, produced the largest capability gains in the discussed trajectory.

Before

Capability investment concentrates in pretraining scale and parameter count.

After

Capability investment concentrates in instruction tuning, post-training data, and the tuning/eval harness around a fixed base model.

Consequence

Teams can differentiate on owned post-training pipelines, but must build evaluation infrastructure capable of measuring instruction-following gains.

Source video

Tradeoffs and failure modes

4

Statistical fluency versus correctness

Benefit

Broad, fluent, plausible generation across domains without task-specific engineering.

Cost or risk

Errors are catastrophic rather than graceful, and no internal semantic check exists to bound them.

They're totally brilliant, and then not just a little wrong, but total garbage some of the time.
Open source video
Source video

Parameter and token scaling versus architectural change

Benefit

Scaling gives predictable, measurable capability improvements in the near term.

Cost or risk

Without structural work on knowledge representation, scaling may hit fundamental barriers and misallocate resources.

Scaling parameters and tokens is insufficient without addressing fundamental knowledge representation.
Open source video
Source video

Macro-level AI indices

Benefit

Provide comparable, communicable signals about global AI activity and progress.

Cost or risk

Aggregation discards the detail needed for engineering decisions, and source data reliability varies across regions.

Indices necessarily lose information to provide macro clarity.
Open source video
Source video

Large-scale multi-agent deployment

Benefit

Millions of interacting agents create new capability and economic dynamics.

Cost or risk

Emergent strategic and economic behaviors must be anticipated; per-agent testing does not cover them.

Designing robust multi-agent systems requires anticipating emergent economic behaviors and strategic dynamics.
Open source video
Source video

Open questions

5

How far can instruction tuning and post-training gains extend without increasing parameter count before they saturate?

Why unresolved

The summary reports massive post-training gains without parameter increases but gives no saturation curve or scaling relationship for post-training effort.

Research direction

Measure capability versus post-training data/compute at fixed model size to find where the post-training lever flattens.

Source video

What measurement design captures ROI for an AI system without the information loss inherent in a macro index?

Why unresolved

The summary asserts industry measures token cost rather than ROI and that indices necessarily lose information, but describes no concrete ROI metric.

Research direction

Prototype task-level outcome accounting (cost per completed workflow, per accepted output) alongside token metrics and compare which drives better design decisions.

Source video

What architecture actually supplies semantic representation without reintroducing the brittleness of 1990s knowledge representation systems?

Why unresolved

The summary identifies the gap and the historical parallel but offers no mechanism that resolves it.

Research direction

Evaluate hybrid designs where retrieval or symbolic structures provide verifiable intermediate representations that a statistical generator must satisfy.

Source video

Do state-space and hybrid architectures preserve long-context reasoning quality relative to transformers at comparable capability?

Why unresolved

They are framed only as potential future directions with linear-complexity benefits; no quality evidence is presented.

Research direction

Benchmark quadratic versus linear-complexity and hybrid memory backends on long-context reasoning and agentic task suites at matched capability.

Source video

How can emergent multi-agent economic and strategic behavior be verified before deployment?

Why unresolved

Game theory and multi-agent simulation are proposed as lenses, not as validated verification methods.

Research direction

Build multi-agent simulations with incentive modeling to surface strategic dynamics and failure modes prior to production rollout.

Source video

Key claims

8
factualVerification needed

LLMs lack semantic representation and rely purely on statistical token correlations.

Evidence

LLMs are missing semantic representation and rely purely on statistical token correlations.

Question

Can probing, intervention, or controlled reasoning tests show a capability gap attributable to missing semantic structure rather than to data or evaluation artifacts?

Source video
causalVerification needed

Instruction tuning and post-training delivered massive capability gains without increases in parameter count.

Evidence

Instruction tuning and post-training delivered massive capability gains without parameter increases.

Question

Holding base model and evaluation fixed, what share of observed capability gain is attributable to post-training versus other changes?

Source video
factualVerification not requested

Transformer attention has quadratic complexity with context length.

Evidence

Transformers have quadratic complexity with context length.

Source video
comparativeVerification needed

State-space models such as Mamba offer linear-complexity alternatives to transformer quadratic scaling.

Evidence

State-space models like Mamba offer linear complexity alternatives to transformer quadratic scaling.

Question

At matched capability, how do linear-complexity state-space and hybrid models compare on long-context reasoning quality, not just asymptotics?

Source video
comparativeVerification needed

The AI industry measures the cost of tokens rather than ROI.

Evidence

The AI industry measures the cost of tokens rather than ROI.

Question

What share of published AI system reporting uses consumption metrics versus task-level value or outcome metrics?

Source video
predictionVerification needed

Scaling parameters and tokens is insufficient without addressing fundamental knowledge representation.

Evidence

Scaling parameters and tokens is insufficient without addressing fundamental knowledge representation.

Question

Do scaling curves for grounded reasoning tasks diverge from those for fluency tasks at large parameter and token budgets?

Source video
opinionVerification not requested

Indices necessarily lose information in order to provide macro clarity.

Evidence

Indices necessarily lose information to provide macro clarity.

Source video
predictionVerification needed

As millions of AI agents interact, game theory and multi-agent economics become vital to system design.

Evidence

As millions of AI agents interact, economic lessons from multi-agent systems and game theory become vital.

Question

Do population-scale agent deployments exhibit emergent strategic behaviors not predicted by single-agent or small-group testing?

Source video

Connections

5