The Deep View · Published 2026-08-09

The new architecture challenging AI’s dominant idea

privateReview: unreviewedConfidence: high
transformersbdhmemoryreasoningsparsitynative-memoryabstract-reasoningcontinuous-learningcatastrophic-forgettingagent-architecture
Open on YouTube ↗

Summary

Overview

  • Speaker: Zuzanna Stamirowska
  • Channel: The Deep View
  • Main topic: Post-transformer AI architecture and the limitations of Large Language Models
  • Purpose: To explain the fundamental limitations of transformer-based LLMs and introduce a new post-transformer architecture designed for autonomous, continuous, and efficient AI systems. Jason Hiner interviews Zuzanna Stamirowska, CEO and co-founder of Pathway, discussing the next generation of AI architectures beyond large language models (LLMs). They explore the limitations of current transformer models, such as memory bottlenecks, ephemeral context windows, lack of continuous learning, and high computational costs. Zuzanna introduces Pathway's post-transformer architecture called BDH (Dragon Hatchling), which incorporates native memory, structured reasoning in abstract space, and sparse localized neural interactions, drawing inspiration from biological systems and complex systems theory.

Topic Map

Introduction to Pathway and AI Limitations

  • Explanation: Zuzanna Stamirowska introduces Pathway and discusses the limitations of current LLMs regarding memory and context.
  • Key claims:
    • Current LLMs suffer from ephemeral context windows and lack of internalised knowledge persistence.
    • There is no link between queries and the world evolving once models are trained.
  • Examples:
    • Writing something on a piece of paper and storing it in a notebook without ever internalizing the knowledge.
  • Terminology:
    • Large Language Models (LLMs)
    • Context window
    • Ephemeral context
    • Parametric memory
  • Why it matters: Understanding LLM limitations highlights why new architectures are required for truly autonomous and scalable AI.

The Dogma of Differentiation and Transformer Limits

  • Explanation: Zuzanna explains the historical context of transformer models and why treating language as the sole medium for thought creates bottlenecks.
  • Key claims:
    • All modern models are built on top of transformers and attention mechanisms.
    • Transformers are sequential sequence models with fundamental limitations in memory and continual learning.
    • Thinking and reasoning should not be strictly limited to language.
  • Examples:
    • Chess players using intuition and shortcuts rather than verbalizing every step.
  • Terminology:
    • Transformer
    • Attention mechanism
    • Sequence models
    • Continual learning
    • Catastrophic forgetting
  • Why it matters: Recognizing transformer limitations pushes the industry to explore non-language-bound reasoning models.

BDH (Dragon Hatchling) Architecture

  • Explanation: Introduction of Pathway's post-transformer architecture called BDH (Dragon Hatchling), inspired by biological neural systems.
  • Key claims:
    • BDH provides native memory built directly into the learning process.
    • It applies sparsity and localized neuron interactions similar to biological brains.
    • It achieves computational efficiency and reduces hallucinations through structured reasoning in abstract space.
  • Examples:
    • Biological neural networks with sparse, localized interactions rather than global dense activations.
  • Terminology:
    • BDH (Dragon Hatchling)
    • Native memory
    • Sparsity
    • Neural interactions
    • Interpretability
  • Why it matters: BDH offers a pathway to more interpretable, memory-efficient, and continuously learning AI systems.

Key Points

Memory and Context Window Bottlenecks

  • Explanation: LLMs rely on ephemeral context windows and lack true parametric memory update capabilities over time.
  • Evidence: Comparison to writing notes in a notebook without internalizing the knowledge.
  • Practical implication: Requires expensive retraining or external vector databases to maintain persistent state.

Language as a Constraint on Reasoning

  • Explanation: Chain-of-thought in language forces models to process sequentially and waste compute on explicit verbalization.
  • Evidence: Expert chess players do not verbalize every calculation; they use abstract intuition.
  • Practical implication: Moving to abstract-space reasoning reduces token explosion and improves computational efficiency.

Biological Inspiration for Efficiency

  • Explanation: Biological brains use sparse, localized interactions rather than dense global computations.
  • Evidence: Neuroscience principles applied to artificial neural network architectures.
  • Practical implication: Significantly lowers training and inference costs while improving scalability.

Frameworks, Models & Processes

BDH (Dragon Hatchling) Architecture

  • How it works: Replaces dense transformer layers with sparse, localized neural interactions and native parametric memory in abstract space.
  • Components:
    • Native memory
    • Sparse localized interactions
    • Abstract-space reasoning
    • Continuous learning mechanisms
  • When to use: Building autonomous AI agents requiring continuous learning, low latency, and efficient long-term memory.

Examples & Case Studies

Swapping the auditory and visual nerve in a mouse experiment.

  • Illustrates: Function shapes the network and neural plasticity.
  • Lesson: Neural architectures adapt to the structural patterns of information processing.

Chess players making intuitive moves without explicit verbal calculation.

  • Illustrates: Reasoning occurs in abstract space without language bottlenecks.
  • Lesson: Advanced reasoning does not require verbalization at every step.

Actionable Takeaways

  • Immediate:
    • Understand the memory and scaling limits of transformer models.
    • Explore sparse neural architectures for cost efficiency.
  • Strategic:
    • Shift focus from language-only generation to abstract-space reasoning systems.
    • Incorporate native memory structures into AI model design.
  • Questions to investigate:
    • How does BDH handle fine-tuning compared to standard transformers?
    • What are the hardware optimization requirements for sparse neural architectures?

Claims Worth Verifying

  • BDH architecture eliminates catastrophic forgetting and enables native memory. (Technical claim)
  • Pathway published proof-of-concept papers in October 2025. (Chronological claim)

Notable Quotes

"it's a bit like a groundhog day. I mean, then you have that head, let's say, it's created and then as you use it, you feed in information that goes into context." (at 0:00) "At Pathway we are building AI that autonomously and continually learns, evolves and reasons."

Compressed Summary

  • Transformers suffer from memory limits and ephemeral context windows.
  • BDH architecture introduces native memory and abstract-space reasoning.
  • Biological inspiration leads to sparse, efficient neural interactions.
  • Moving beyond language-bound reasoning reduces compute overhead.
  • Keywords: transformers, bdh, memory, reasoning, sparsity
  • Core insight: Overcoming the memory and reasoning limits of transformers requires moving from language-bound sequence processing to sparse, biologically inspired architectures with native memory.

Core insights

5
Mechanismhigh noveltystrong evidence

Language-bound reasoning is an architectural bottleneck: requiring chain-of-thought to be verbalized forces token-by-token sequential computation and wastes compute on explicit verbalization. Abstract-space reasoning can represent the same steps more compactly, reducing token explosion and computational cost.

Why it matters

Engineers currently rely on prompting models with chain-of-thought to improve reasoning. If reasoning can happen in abstract space, the dominant cost and latency of verbose CoT traces may be avoidable, changing how reasoning agents are designed and priced.

Generalization

Any AI system that uses human-readable intermediate steps as a proxy for reasoning may be paying a token and latency tax that can be removed when the reasoning itself is not intended for human consumption.

Chain-of-thought in language forces models to process sequentially and waste compute on explicit verbalization.
Open source video
Moving to abstract-space reasoning reduces token explosion and improves computational efficiency.
Open source video
Advanced reasoning does not require verbalization at every step.
Open source video
Mental Modelmedium noveltystrong evidence

Context windows are not memory: they are ephemeral scratchpads. Persistent agent state requires native parametric memory that is updated as part of the learning process, rather than external vector databases or context stuffing, which are workarounds for a missing capability.

Why it matters

Agent systems today treat context windows and RAG as long-term memory. This hides a fundamental architectural gap: true persistence must be a model capability, not an input buffer. Recognizing this changes where memory responsibility belongs in an agent stack.

Generalization

For any system that must persist state across interactions, the state update mechanism should be part of the core computation, not an appendage that gets injected and discarded.

Current LLMs rely on ephemeral context windows and lack true parametric memory update capabilities over time.
Open source video
Requires expensive retraining or external vector databases to maintain persistent state.
Open source video
BDH provides native memory built directly into the learning process.
Open source video
Architecturehigh noveltymoderate evidence

Sparse, localized neural interactions — inspired by biological brains — can replace dense global attention, significantly lowering training and inference costs while improving scalability.

Why it matters

Dense attention computes global pairwise interactions, which is a scalability wall. Sparse localized interactions are a concrete alternative that changes compute scaling, hardware requirements, and the cost curve for large models.

Generalization

In any domain where interactions are inherently local or hierarchical, replacing dense global computation with sparse, structured computation can yield large efficiency gains.

Biological neural networks with sparse, localized interactions rather than global dense activations.
Open source video
Significantly lowers training and inference costs while improving scalability.
Open source video
Failure Modemedium noveltystrong evidence

A trained model is frozen relative to a changing world: there is no link between queries and the world evolving after training. Autonomous agents need architectures that continuously learn from live data, not just retrieve from stale parameters.

Why it matters

Production agents operating in evolving environments will silently act on stale knowledge. Continuous learning is not an optional enhancement but a requirement for autonomy, and it introduces new operational concerns around drift, safety, and evaluation.

Generalization

Any deployed AI system that interacts with a changing environment needs a closed loop for updating its world model, otherwise its behavior degrades over time.

There is no link between queries and the world evolving once models are trained.
Open source video
At Pathway we are building AI that autonomously and continually learns, evolves and reasons.
Open source video
Predictionmedium noveltymoderate evidence

The claim that BDH eliminates catastrophic forgetting through native memory is a high-stakes, testable hypothesis, not an established fact. If true, it changes the retraining/fine-tuning lifecycle; if false, continuous learning remains an unsolved problem.

Why it matters

Engineers evaluating post-transformer architectures should treat vendor claims about catastrophic forgetting as hypotheses needing verification before building production systems around them.

Generalization

Architectural claims that would change the model lifecycle should be met with targeted experiments rather than accepted at face value.

BDH architecture eliminates catastrophic forgetting and enables native memory.
Open source video
How does BDH handle fine-tuning compared to standard transformers?
Open source video

Deep dives

5

Abstract-space reasoning vs language-bound chain-of-thought

Research question

Can post-transformer architectures perform multi-step reasoning in abstract space with lower token cost and latency while preserving or improving accuracy compared to verbalized chain-of-thought in LLMs?

Why

Engineers currently rely on prompting models with chain-of-thought to improve reasoning; if reasoning can happen in abstract space, the dominant cost and latency of verbose CoT traces may be avoidable, changing how reasoning agents are designed and priced.

Thinking and reasoning should not be strictly limited to language.
Open source video
It achieves computational efficiency and reduces hallucinations through structured reasoning in abstract space.
Open source video
Source video

Native parametric memory as a replacement for context-window/RAG persistence

Research question

How does native parametric memory (updated during learning) compare with context-window stuffing and vector databases for long-horizon agent memory?

Why

Agent systems today treat context windows and RAG as long-term memory, hiding a fundamental architectural gap: true persistence must be a model capability, not an input buffer. This dictates where memory responsibility belongs in an agent stack.

Current LLMs suffer from ephemeral context windows and lack internalised knowledge persistence.
Open source video
BDH provides native memory built directly into the learning process.
Open source video
Source video

Validating the catastrophic forgetting elimination claim

Research question

Under what task sequences and scales does a sparse-memory architecture like BDH avoid catastrophic forgetting compared to standard fine-tuned transformers?

Why

Architectural claims that would change the model lifecycle should be met with targeted experiments rather than accepted at face value. If true, it changes the retraining/fine-tuning lifecycle; if false, continuous learning remains unsolved.

BDH provides native memory built directly into the learning process.
Open source video
Transformers are sequential sequence models with fundamental limitations in memory and continual learning.
Open source video
Source video

Hardware and systems implications of sparse localized neural interactions

Research question

What are the compute and memory bottlenecks for sparse-localized architectures on current dense-optimized hardware?

Why

Dense attention computes global pairwise interactions, which is a scalability wall; sparse localized interactions are a concrete alternative that changes compute scaling, hardware requirements, and the cost curve for large models.

Biological neural networks with sparse, localized interactions rather than global dense activations.
Open source video
It applies sparsity and localized neuron interactions similar to biological brains.
Open source video
Source video

Operational guardrails for continuously learning models

Research question

How can continuously updating models be safely deployed without drift, memory corruption, or loss of alignment?

Why

Production agents operating in evolving environments will silently act on stale knowledge. Continuous learning is not optional for autonomy, but it introduces new operational concerns around drift, safety, and evaluation.

There is no link between queries and the world evolving once models are trained.
Open source video
Source video

Article ideas

4

Stop Making AI Think in English

Verbalized chain-of-thought is a costly concession to human interpretability, not a requirement for machine reasoning; architectures that reason in abstract space will be cheaper and faster, and we must design new verification methods for them.

Angle

Argument from economics and biology to a post-language reasoning paradigm.

Source video

Your Agent Doesn't Have Memory—It Has a Scratchpad

Context windows and RAG are stopgap mechanisms for missing native memory; true autonomous agents require parametric memory updated during learning, which changes the entire agent stack.

Angle

Critical engineering analysis of agent memory patterns.

Source video

The Next AI Hardware Shift Will Be Driven by Sparsity

Dense attention is a scaling dead end; sparse, localized interaction patterns, inspired by biology, will force a redesign of AI accelerators and kernels—and the first companies to build for sparsity will own the next infrastructure wave.

Angle

Forward-looking infrastructure and hardware thesis.

Source video

Vendor Claims Are Hypotheses: How to Stress-Test 'Catastrophic Forgetting Solved'

Architectural breakthroughs that would change model lifecycles should be adopted only after independent, reproduced benchmarks; the BDH catastrophic-forgetting claim is a case study in verification.

Angle

Evidence-driven engineering management.

Source video

Project ideas

4

AbstractReasoningBench

beyond-evals

An LLM solution using compact/abstract internal reasoning (without forced verbalization) can match or exceed chain-of-thought accuracy on multi-step reasoning tasks while using <50% of the tokens and <50% latency.

Proof of concept

Create a reasoning task suite (e.g., math word problems, logical deductions) and compare three conditions: standard CoT, 'private' hidden reasoning (via model output distribution or a custom abstraction), and an external black-box reasoner; instrument token counts and latency.

Measurement

Accuracy, token count, latency, and cost per task

Source video

MemoryPersistenceBench

movement-lab

An agent using native memory updates will outperform a RAG/context-packing agent on long-horizon tasks with >10 turns or evolving facts, where the RAG baseline shows progressive accuracy degradation.

Proof of concept

Build a small agent with two memory backends: a vector store/RAG and a simulated native memory (or a fine-tuned memory model). Run tasks that require retention of facts across many turns and measure success rates as horizon increases.

Measurement

Task success rate vs horizon length; fact-retention accuracy

Source video

ForgettingStress

gatehouse

A continual-learning model with native memory (e.g., simulated with a replay buffer) will have a significantly lower forgetting rate (e.g., <10% drop in prior-task accuracy) compared to a fine-tuned transformer baseline on a standard sequential task benchmark.

Proof of concept

Use a continual learning benchmark (e.g., permuted MNIST, sequential CIFAR, or a language version) with a transformer baseline and a memory-augmented model; train sequentially and evaluate prior tasks.

Measurement

Forgetting rate (accuracy drop), average accuracy, stability

Source video

SparseKernelProbe

new

On current NVIDIA GPUs, a sparse-local interaction layer (e.g., block-local attention) will show lower MFU and throughput than dense attention at identical parameter count and sequence length, confirming a hardware/kernel gap.

Proof of concept

Implement a dense attention layer and a sparse localized attention layer in PyTorch/OpenAI Triton; benchmark FLOPs, wall time, and memory bandwidth across sequence lengths.

Measurement

Achieved TFLOPS, wall-clock time, memory bandwidth

Source video

Architectural implications

4

Agent memory is typically implemented as context window stuffing or external vector database retrieval.

Before

Context injection and RAG are the default memory mechanisms for LLM-based agents.

After

Memory becomes native parametric state, updated online as part of the learning process, with no need for external stores or expensive retraining for new facts.

Consequence

Agent harnesses can stop replicating memory via prompt injection; evaluation shifts to measuring memory persistence and continual learning rather than retrieval quality.

Source video

Reasoning is exposed as natural-language chain-of-thought, forcing token-by-token verbalization.

Before

Systems prompt for step-by-step explanations, paying for every token and adding latency.

After

Reasoning happens in abstract space; language is used only at input/output boundaries.

Consequence

Token cost and latency drop significantly, but reasoning becomes less human-auditable, requiring new interpretability and verification methods.

Source video

Dense attention computes global pairwise interactions across the entire sequence.

Before

Scaling model size and context length is bounded by dense attention's quadratic cost.

After

Sparse, localized interactions reduce compute and scale more gracefully.

Consequence

Hardware and kernels must be designed for sparsity, and models may trade long-range global context for locality.

Source video

Models are trained once and deployed statically; the world moves on.

Before

Periodic retraining or manual fine-tuning is required to update knowledge.

After

Architectures with native memory and continuous learning update from data streams autonomously.

Consequence

Operations shift from train-deploy to living systems with guardrails for drift, safety, and memory corruption.

Source video

Tradeoffs and failure modes

4

Abstract-space reasoning vs language-bound reasoning

Benefit

Avoids token explosion and sequential compute; more efficient and scalable reasoning.

Cost or risk

Reasoning becomes opaque and difficult for humans to audit or debug; existing CoT-based interpretability tools lose value.

Chess players making intuitive moves without explicit verbal calculation.
Open source video
Source video

Native memory vs external vector databases

Benefit

True persistent state without expensive retraining; enables continuous learning and autonomy.

Cost or risk

Requires a fundamentally new architecture; memory contents are harder to inspect or control than explicit database rows.

Requires expensive retraining or external vector databases to maintain persistent state.
Open source video
Source video

Sparse localized interactions vs dense global attention

Benefit

Lower training and inference costs; improves scalability.

Cost or risk

Potential loss of long-range dependencies; hardware is currently optimized for dense matrix multiplication, not sparse neural interactions.

Biological neural networks with sparse, localized interactions rather than global dense activations.
Open source video
Source video

Continuous learning vs static training

Benefit

Model stays aligned with a changing world and can operate autonomously.

Cost or risk

Risk of catastrophic forgetting or memory corruption; requires new safeguards and evaluation methods for online learning.

At Pathway we are building AI that autonomously and continually learns, evolves and reasons.
Open source video
Source video

Open questions

5

Does BDH actually eliminate catastrophic forgetting at production scale, and under what conditions?

Why unresolved

The claim is high-stakes and currently supported only by vendor statements and proof-of-concept papers.

Research direction

Design a continual learning benchmark that measures forgetting over long horizons and varied task distributions.

Source video

How does BDH handle fine-tuning compared to standard transformers?

Why unresolved

The summary explicitly identifies this as an open question; the architecture's native memory may change transfer learning dynamics.

Research direction

Run controlled fine-tuning experiments on downstream tasks and compare sample efficiency, stability, and forgetting.

Source video

What are the hardware optimization requirements for sparse neural architectures like BDH?

Why unresolved

Current hardware and ML kernels are designed for dense transformers, not sparse localized interactions.

Research direction

Profile sparse architectures on existing GPUs and explore custom sparse kernels, memory layouts, and interconnect patterns.

Source video

How can we evaluate the correctness of reasoning performed in abstract space when it is not verbalized?

Why unresolved

Evaluation today relies on human-readable text; abstract representations are not directly inspectable.

Research direction

Develop probing methods and behavioral tests that verify abstract reasoning outputs without requiring language traces.

Source video

Does structured reasoning in abstract space actually reduce hallucinations as claimed?

Why unresolved

The summary claims BDH 'reduces hallucinations through structured reasoning in abstract space' but provides no experimental evidence.

Research direction

Create adversarial question sets that trigger hallucination in current LLMs and measure BDH or similar architectures on the same set.

Source video

Key claims

7
factualVerification needed

Current LLMs suffer from ephemeral context windows and lack internalised knowledge persistence.

Evidence

Current LLMs rely on ephemeral context windows and lack true parametric memory update capabilities over time.

Question

Can this be established by measuring how LLM performance degrades when context is truncated or world state changes?

Source video
factualVerification needed

There is no link between queries and the world evolving once models are trained.

Evidence

There is no link between queries and the world evolving once models are trained.

Question

Does this hold for all deployed LLMs, or can external retrieval/grounding effectively create such a link?

Source video
causalVerification needed

BDH architecture eliminates catastrophic forgetting and enables native memory.

Evidence

BDH architecture eliminates catastrophic forgetting and enables native memory.

Question

What experiments would demonstrate absence of catastrophic forgetting in BDH over long sequences of tasks?

Source video
factualVerification needed

Pathway published proof-of-concept papers in October 2025.

Evidence

Pathway published proof-of-concept papers in October 2025.

Question

Where were these papers published and what are their titles?

Source video
causalVerification needed

Moving to abstract-space reasoning reduces token explosion and improves computational efficiency.

Evidence

Moving to abstract-space reasoning reduces token explosion and improves computational efficiency.

Question

Can a controlled study show the same reasoning task solved with less compute when the model is not forced to verbalize intermediate steps?

Source video
opinionVerification not requested

Thinking and reasoning should not be strictly limited to language.

Evidence

Thinking and reasoning should not be strictly limited to language.

Source video
factualVerification not requested

Biological neural networks use sparse, localized interactions rather than global dense activations.

Evidence

Biological neural networks with sparse, localized interactions rather than global dense activations.

Source video

Connections

5