Andrew Gordon Wilson · Published 2026-09-07

The Foundations of Modern AI: Generalization, Data Selection, and Epiplexity

privateReview: unreviewedConfidence: high
epiplexitygeneralizationcompressioninductive-biasoverparameterizationsoft-inductive-biasdata-selectionmdlmodel-selection
Open on YouTube ↗

Summary

Overview

  • Speaker: Andrew Gordon Wilson
  • Channel: Andrew Gordon Wilson
  • Main topic: Foundations of Modern AI, Generalization, Data Selection, and Epiplexity
  • Purpose: To re-evaluate and re-program how machine learning researchers and practitioners think about model construction, data selection, overparameterization, and generalization through rigorous theoretical frameworks and intuitive examples. Andrew Gordon Wilson presents a deep dive into the foundations of modern artificial intelligence, challenging conventional wisdom around model selection, generalization, overparameterization, and learning theory. He argues that traditional intuitions regarding model size, data scarcity, and inductive biases are often flawed or incomplete. By introducing frameworks like soft inductive biases, epiplexity (epistemic complexity), and sequential coding, Wilson demonstrates how massive models can achieve extraordinary generalization and data efficiency without suffering from catastrophic overfitting.

Topic Map

Introduction and Model Selection Warm-up

  • Explanation: An introductory discussion on model selection using airline passenger data, challenging the room on choosing between linear, cubic, and 10,000th order polynomials.
  • Key claims:
    • Real-world data is complicated, and true models require high parameter capacity.
    • Most practitioners implicitly embrace expressive models despite conventional warnings against overparameterization.
  • Examples:
    • Airline passenger data fitted with linear, cubic, and 10,000th order polynomials.
  • Terminology:
    • model selection
    • overparameterization
    • inductive bias
  • Why it matters: Sets the stage for rethinking foundational assumptions in machine learning.

Soft Inductive Biases

  • Explanation: Explaining how models balance expressiveness and simplicity using soft inductive biases rather than hard structural constraints.
  • Key claims:
    • Inductive bias is essential for learning by example (induction).
    • Soft inductive biases allow models to be expressive while strongly preferring simpler or structured solutions.
  • Examples:
    • High-order polynomials with order-dependent regularization.
  • Terminology:
    • soft inductive bias
    • restriction bias
    • regularization
    • Occam's razor
  • Why it matters: Provides a unifying intuition for how modern neural networks achieve generalization.

Generalization Guarantees and Compression

  • Explanation: Connecting generalization bounds to model compressibility using Kolmogorov complexity and PAC-Bayes theory.
  • Key claims:
    • Evaluating generalization bounds is easy when combining expressiveness with a compression bias.
    • Larger models achieve better compression, shrinking the generalization gap as a power law.
  • Examples:
    • Solomonoff prior and prefix-free Kolmogorov complexity applied to LLMs.
  • Terminology:
    • PAC-Bayes
    • Kolmogorov complexity
    • expected risk
    • empirical risk
    • compression
  • Why it matters: Explains why massive models generalize so well in practice.

The Three Paradoxes of Information

  • Explanation: Highlighting tensions between traditional information theory and modern machine learning practice.
  • Key claims:
    • Paradox 1: Information cannot be created by deterministic processes, yet synthetic data and pseudorandom number generators enable superhuman systems like AlphaZero.
    • Paradox 2: Information is independent of factorization order, yet sequence direction (e.g., left-to-right text) matters immensely.
    • Paradox 3: Likelihood modeling is merely distribution matching, yet emergent phenomena occur where models go beyond data-generating processes.
  • Examples:
    • AlphaZero playing chess, elementary cellular automata, and next-word prediction.
  • Terminology:
    • epiplexity
    • epistemic complexity
    • time-bounded entropy
    • data processing inequality
  • Why it matters: Identifies fundamental flaws in classical information theory assumptions when applied to bounded computation.

Epiplexity and Sequential Coding

  • Explanation: Introducing epiplexity as a new measure of information that accounts for computational bounds and structured data.
  • Key claims:
    • Epiplexity measures the structural information content extracted by a computationally bounded observer.
    • Sequential coding pushes compression to absolute limits, showing large models achieve better than 1 bit per parameter compression.
  • Examples:
    • Evaluating epiplexity on natural language data, time series, and code.
  • Terminology:
    • epiplexity
    • sequential coding
    • minimum description length
    • time-bounded MDL
  • Why it matters: Provides a rigorous framework for data selection, generation, and understanding out-of-distribution generalization.

Key Points

Overparameterization is not inherently harmful

  • Explanation: Larger models with massive parameter counts often generalize better because they provide stronger soft inductive biases toward simpler, compressible solutions.
  • Evidence: Observed across polynomial regression, Gaussian processes, ResNets, and large language models.
  • Practical implication: Do not fear overparameterization; instead, design architectures and regularization that favor compressible solutions.

Epiplexity resolves classical information theory paradoxes

  • Explanation: Traditional information theory assumes unlimited computation, whereas epiplexity incorporates computational bounds to measure predictable structure in data.
  • Evidence: Analysis of elementary cellular automata (ECA rules 15, 30, and 54) and AlphaZero.
  • Practical implication: Use epiplexity for advanced data selection and curating high-quality training curricula.

Language data exhibits exceptionally high epiplexity

  • Explanation: Natural language, code, and math are highly structured and compressible representations of information compared to raw pixels.
  • Evidence: Empirical measurements of information content across modalities like images, time series, and text.
  • Practical implication: Pre-training on text and structured data yields powerful transferable representations.

Frameworks, Models & Processes

Soft Inductive Bias Framework

  • How it works: Combines an expressive hypothesis space with soft, continuous preferences for simpler or more structured solutions.
  • Components:
    • Hypothesis space
    • Restriction bias
    • Soft preference (regularization, priors, optimization)
  • When to use: When designing model architectures and training objectives for complex real-world data.

Epiplexity (Epistemic Complexity)

  • How it works: Measures the predictable structural information extractable by a computationally bounded observer, defined via time-bounded minimum description length (MDL).
  • Components:
    • Model bits (description length)
    • Data-given-model bits (cross-entropy loss)
    • Time-bounded entropy
  • When to use: For data selection, understanding out-of-distribution generalization, and evaluating model compressibility.

Examples & Case Studies

Fitting airline passenger data with polynomial regression of varying degrees.

  • Illustrates: Model selection and the role of regularization in preventing overfitting.
  • Lesson: Representing beliefs honestly through regularization prevents disastrous overfitting while maintaining expressiveness.

AlphaZero mastering chess starting from minimal rules and self-play.

  • Illustrates: Paradox 1: Information cannot be created deterministically, yet computation generates emergent structure.
  • Lesson: Bounded computation allows observers to extract massive amounts of structural information from deterministic environments.

Evaluating elementary cellular automata rules (Rule 15, 30, 54) under compute-bounded evolution.

  • Illustrates: How time-bounded MDL and epiplexity evolve as compute increases.
  • Lesson: Complex, predictable patterns (like gliders in Rule 54) emerge and become compressible with sufficient compute.

Actionable Takeaways

  • Immediate:
    • Embrace expressive hypothesis spaces combined with a compression bias.
    • Recognize that overparameterization strengthens soft inductive biases rather than weakening them.
    • Use epiplexity metrics for intelligent data curation and selection.
  • Strategic:
    • Re-evaluate classical learning theory assumptions that ignore computational bounds.
    • Focus on representation learning and similarity metrics tailored to high-dimensional natural signals.
    • Leverage scaling laws driven by model compressibility and sequential coding.
  • Questions to investigate:
    • How can epiplexity be efficiently estimated for billion-parameter foundation models?
    • What are the precise mathematical links between deep learning optimization trajectories and epiplexity?
    • How do different training curricula impact the epiplexity of downstream representations?

Claims Worth Verifying

  • Larger models achieve compression better than 1 bit per parameter post-training. (empirical)
  • Sequential coding bounds on generalization gaps shrink as a power law with scale. (theoretical)

Notable Quotes

"The real world is a complicated place and there's going to be some setting of those parameters, the AJs, which will get us closer to the ground truth description of this or any other problem." "Induction is actually impossible without making assumptions." "Accounting for computational constraints is understanding modern deep learning."

Compressed Summary

  • Soft inductive biases bridge expressiveness and generalization.
  • Overparameterization enhances implicit regularization toward simple, compressible solutions.
  • Epiplexity measures computationally bounded structural information in data.
  • Sequential coding explains why scaling laws hold for massive neural networks.
  • Keywords: epiplexity, generalization, compression, inductive-bias, overparameterization
  • Core insight: Modern deep learning succeeds by combining maximally expressive models with soft simplicity biases, where scale reduces the generalization gap through extreme data compression.

Core insights

6
Mental Modelmedium noveltymoderate evidence

Generalization is achieved not by restricting model capacity but by pairing an expressive hypothesis space with a *soft* preference for compressible solutions — a continuous bias (regularization, priors, optimization dynamics) rather than a hard structural restriction. This reframes overparameterization as a feature that strengthens the compression bias.

Why it matters

Directly challenges the default engineering instinct to cap model size or hand-design restrictive architectures to prevent overfitting. It implies architecture and regularization choices should be evaluated by whether they make compressible solutions more likely, not by parameter count.

Generalization

Applies to agentic systems design: prefer high-capacity components governed by soft priors/objectives (e.g., length/description penalties, preference losses) over narrow hard-coded capability boundaries.

Soft inductive biases allow models to be expressive while strongly preferring simpler or structured solutions.
Open source video
Real-world data is complicated, and true models require high parameter capacity.
Open source video
Mechanismhigh noveltymoderate evidence

Epiplexity measures the structural information a *computationally bounded* observer can extract from data, defined via time-bounded minimum description length. Classical information theory assumes unlimited computation, which is why it produces paradoxes when applied to real learning systems.

Why it matters

Provides a concrete, compute-aware metric for data selection rather than heuristics like dedup or perplexity. Because the metric is observer- and budget-relative, it implies data quality is not an intrinsic property of data but of the (model, compute) pair consuming it.

Generalization

Any pipeline that curates training or context data should measure value relative to the compute/model that will consume it; the same dataset may be high- or low-signal depending on the learner's budget.

Epiplexity measures the structural information content extracted by a computationally bounded observer.
Open source video
Traditional information theory assumes unlimited computation, whereas epiplexity incorporates computational bounds to measure predictable structure in data.
Open source video
Empirical Resulthigh noveltymoderate evidence

Larger models achieve better compression, and the generalization gap shrinks as a power law; sequential coding can push compression past 1 bit per parameter. This links model size to compressibility as a measurable scaling relationship.

Why it matters

If generalization gap shrinks predictably with compression, compression ratio becomes a candidate proxy metric for expected generalization — useful for model selection and scaling decisions without held-out overfitting-paranoia.

Generalization

Suggests using achieved compression (e.g., bits of description length over data) as a model-selection signal across domains, not just NLP.

Larger models achieve better compression, shrinking the generalization gap as a power law.
Open source video
Sequential coding pushes compression to absolute limits, showing large models achieve better than 1 bit per parameter compression.
Open source video
Tradeoffhigh noveltymoderate evidence

Sequence direction / factorization order matters empirically even though classical information theory treats information as independent of factorization order. Left-to-right text ordering is doing real work.

Why it matters

Token/sequence ordering is an architectural decision with measurable consequences, not an arbitrary convention. Choices about how a task is serialized (order of steps, tool calls, context) affect what the model can compress and learn.

Generalization

For agentic systems, the order in which context, tool outputs, and reasoning steps are serialized into a trajectory may materially change learnability and inference quality.

Information is independent of factorization order, yet sequence direction (e.g., left-to-right text) matters immensely.
Open source video
Empirical Resultmedium noveltymoderate evidence

Natural language, code, and math exhibit exceptionally high epiplexity relative to raw pixels, i.e., they are highly structured and compressible representations. This makes them high-value pretraining substrate.

Why it matters

Gives a principled reason (structural compressibility) for why text/code/math pretraining yields strong transferable representations, informing modality prioritization for training and context selection.

Generalization

When choosing what data or context to feed a system, prefer sources with high extractable structure relative to a bounded learner rather than raw low-structure inputs.

Natural language, code, and math are highly structured and compressible representations of information compared to raw pixels.
Open source video
Pre-training on text and structured data yields powerful transferable representations.
Open source video
Mental Modelhigh noveltymoderate evidence

The first information paradox: classical theory says deterministic processes cannot create information (data processing inequality), yet synthetic data generation and pseudorandom generators enable superhuman systems like AlphaZero.

Why it matters

Legitimizes self-play and synthetic data generation as genuine information-creating mechanisms under bounded computation, undermining the intuition that augmenting data cannot add signal beyond the source distribution.

Generalization

Synthetic/self-generated trajectories can carry extractable structure that no original dataset contained, provided the generator and learner are computationally bounded in a structured search space.

Information cannot be created by deterministic processes, yet synthetic data and pseudorandom number generators enable superhuman systems like AlphaZero.
Open source video

Deep dives

5

Compute-Bounded Epiplexity for Data Selection

Research question

How can epiplexity be estimated tractably on large corpora to drive data selection and curriculum learning, and how sensitive are selection decisions to the chosen compute bound?

Why

Data quality is currently treated as an intrinsic property, but epiplexity makes it observer- and budget-relative; an operational estimator would enable model-and-budget-coupled data pipelines that select for extractable structure rather than heuristics like dedup or perplexity.

Epiplexity measures the structural information content extracted by a computationally bounded observer.
Open source video
Traditional information theory assumes unlimited computation, whereas epiplexity incorporates computational bounds to measure predictable structure in data.
Open source video
Source video

Soft Inductive Biases vs. Hard Capacity Limits

Research question

Does pairing an expressive hypothesis space with a soft preference for compressible solutions (e.g., regularization, MDL penalties, optimization dynamics) yield better generalization than hard structural restrictions at equal data and compute?

Why

It challenges the default engineering instinct to cap model size or hand-design restrictive architectures; if soft biases are the mechanism, architecture and regularization reviews should evaluate compressibility, not parameter count.

Soft inductive biases allow models to be expressive while strongly preferring simpler or structured solutions.
Open source video
Real-world data is complicated, and true models require high parameter capacity.
Open source video
Source video

Compression as a Proxy for Generalization

Research question

Is achieved compression (e.g., bits of description length over data) a reliable model-selection signal, and does the generalization gap shrink as a power law with model size across architectures and tasks?

Why

If compression ratio predicts expected generalization, it could replace held-out overfitting paranoia with a measurable scaling relationship, informing model sizing and scaling decisions.

Larger models achieve better compression, shrinking the generalization gap as a power law.
Open source video
Sequential coding pushes compression to absolute limits, showing large models achieve better than 1 bit per parameter compression.
Open source video
Source video

Factorization Order as a Learnability Variable

Research question

What determines whether sequence direction or factorization order helps or hurts learning, and how large is the measurable effect on learned representations and downstream performance?

Why

If order matters immensely even though classical information theory says it shouldn't, then serialization choices in agent trajectories, tool-call ordering, and context layout are optimizable design surfaces rather than neutral implementation details.

Information is independent of factorization order, yet sequence direction (e.g., left-to-right text) matters immensely.
Open source video
Source video

Synthetic Data and the Data Processing Inequality

Research question

Under what bounded-computation conditions does deterministic generation (self-play, pseudorandomness) produce usable new signal that surpasses the original data-generating process?

Why

It legitimizes self-play and synthetic data as genuine information-creating mechanisms under bounded computation, undermining the intuition that augmentation cannot add signal beyond the source distribution.

Information cannot be created by deterministic processes, yet synthetic data and pseudorandom number generators enable superhuman systems like AlphaZero.
Open source video
Source video

Article ideas

4

Epiplexity: Why Data Quality Is Relative to the Learner

Data quality is not an intrinsic property of a corpus; it is the structural information a bounded learner can extract, so data selection must be model- and compute-coupled rather than based on heuristics like dedup or perplexity.

Angle

Reframe data pipelines as compute-aware estimators of extractable structure, and show how this changes curation decisions across model scales.

Source video

Stop Capping Model Size: Overparameterization Is a Compression Strategy

The instinct to limit parameters to avoid overfitting is backwards; generalization comes from soft inductive biases that make compressible solutions likely, so scaling capacity with the right regularization improves generalization.

Angle

Argue that architecture reviews should shift from parameter budgets to compressibility and bias-softness as governing criteria.

Source video

Factorization Order Is a First-Class Design Variable in Agentic Systems

Sequence serialization order materially affects learnability and inference quality, so the order of context, tool outputs, and reasoning steps in agent trajectories should be optimized, not treated as neutral.

Angle

Show how agent trajectory formats and context layouts become optimizable design surfaces under bounded computation.

Source video

Synthetic Data Breaks Classical Information Theory — Here's Why That Matters

Deterministic generation can yield superhuman systems because bounded computation allows new structure to emerge, legitimizing self-play and synthetic data as genuine information-creating mechanisms.

Angle

Use the first information paradox to argue that classical guarantees about information content no longer bound what synthetic pipelines can produce.

Source video

Project ideas

4

Epiplexity-Guided Data Curation

new

Selecting training data by epiplexity (time-bounded MDL) yields better out-of-distribution generalization than dedup or perplexity filtering at the same compute budget.

Proof of concept

Implement a tractable epiplexity estimator during training; use it to curate subsets of a text/code corpus; train a small model on each subset and compare OOD transfer.

Measurement

OOD accuracy and achieved compression (bits per token) versus baseline curation methods.

Source video

Soft Bias Scaling Law

movement-lab

Increasing model capacity while adding a soft compression penalty (e.g., MDL regularization) shrinks the generalization gap more than capacity increase alone at equal data and compute.

Proof of concept

Train models of varying parameter counts with and without a soft compression bias; measure held-out generalization gap and achieved compression.

Measurement

Generalization gap (test minus train loss) as a function of model size, with and without soft bias.

Source video

Trajectory Ordering Optimization

gatehouse

The order in which tool outputs and reasoning steps are serialized into an agent trajectory significantly affects downstream task success and learnability.

Proof of concept

Generate agent trajectories with different orderings (e.g., tool calls before vs. after reasoning, context placement) and evaluate task success and training efficiency when fine-tuning on each ordering.

Measurement

Task success rate and sample efficiency (steps to converge) across orderings.

Source video

Self-Play Structure Emergence

new

Self-play with bounded compute can produce novel extractable structure (strategies, compressed representations) not present in the seed data, contradicting the data processing inequality intuition.

Proof of concept

Run self-play in a game (e.g., AlphaZero-style) and measure new strategies or compressed representations that outperform the seed data's best.

Measurement

Performance beyond seed data, and information-theoretic novelty of generated trajectories relative to seed.

Source video

Architectural implications

3

Data selection currently relies on heuristic filtering (dedup, quality classifiers, perplexity) rather than a compute-aware measure of extractable structure.

Before

Curate data by intrinsic statistical properties (cleanliness, dedup, perplexity) independent of the consuming model and its compute budget.

After

Curate data by epiplexity: measure structural information extractable by the specific bounded learner, then select for curriculum and out-of-distribution value.

Consequence

Data pipelines become model- and budget-coupled; the same corpus yields different selection decisions across model scales or compute tiers.

Source video

Model sizing decisions are often constrained by classical overfitting fears rather than by compression behavior.

Before

Cap parameters / restrict capacity to avoid overfitting under data scarcity.

After

Scale capacity while ensuring soft inductive biases push toward compressible solutions, since larger models compress better and shrink the generalization gap.

Consequence

Architecture reviews shift from parameter budgets to compressibility and bias-softness as the governing criteria.

Source video

Sequence serialization (order of steps, tokens, tool outputs) is treated as a neutral implementation detail.

Before

Serialize context/steps in arbitrary order, assuming information content is order-invariant.

After

Treat ordering as a first-class design variable, since factorization order materially affects learning under bounded computation.

Consequence

Agent trajectory formats, tool-call ordering, and context layout become optimizable design surfaces.

Source video

Tradeoffs and failure modes

3

Overparameterization without a compression bias

Benefit

An expressive hypothesis space can represent the complexity of real-world data.

Cost or risk

Expressiveness alone (without a soft preference for compressible/simple solutions) risks selecting non-compressible hypotheses and losing the generalization benefit.

Soft inductive biases allow models to be expressive while strongly preferring simpler or structured solutions.
Open source video
Source video

Compute-bounded information measures are observer-relative

Benefit

Epiplexity accounts for finite compute, resolving classical information-theory paradoxes.

Cost or risk

Because the measure depends on the bounded observer, data 'value' is not fixed — curation decisions may not transfer across model scales or compute budgets.

Epiplexity measures the structural information content extracted by a computationally bounded observer.
Open source video
Source video

Reliance on synthetic/self-generated data

Benefit

Deterministic generation (self-play, pseudo-randomness) can yield systems that surpass the original data-generating process.

Cost or risk

This contradicts the classical data processing inequality, so classical guarantees about information content no longer bound what such pipelines can produce or fail to produce.

Information cannot be created by deterministic processes, yet synthetic data and pseudorandom number generators enable superhuman systems like AlphaZero.
Open source video
Source video

Open questions

5

How is epiplexity computed tractably on large corpora so it can drive practical data selection?

Why unresolved

The summary defines epiplexity via time-bounded MDL (model bits + data-given-model bits) but does not describe an operational estimation procedure or cost profile.

Research direction

Prototype estimators that approximate time-bounded MDL during training and benchmark them against dedup/perplexity filtering for downstream transfer.

Source video

How does the compute bound in time-bounded MDL get chosen, and how sensitive are selection decisions to it?

Why unresolved

Epiplexity is defined relative to a computationally bounded observer, but the summary does not specify how the budget is set or how results vary with it.

Research direction

Run sensitivity studies measuring epiplexity and downstream performance across different compute bounds.

Source video

Does high epiplexity data predict better out-of-distribution generalization, or only better in-distribution compression?

Why unresolved

The summary claims epiplexity aids data selection and understanding OOD generalization but provides no direct evidence linking measured epiplexity to OOD outcomes.

Research direction

Build benchmarks correlating corpus-level epiplexity with OOD transfer metrics.

Source video

What determines whether sequence direction (factorization order) helps or hurts learning?

Why unresolved

The paradox notes order matters immensely but does not explain the mechanism or give rules for choosing an order.

Research direction

Controlled experiments varying serialization order in generative modeling and agent trajectories to characterize when order affects learnability.

Source video

Where do emergent 'beyond data-generating process' behaviors come from under bounded computation?

Why unresolved

The third paradox observes emergence under likelihood modeling but does not explain the mechanism.

Research direction

Use tools like elementary cellular automata to probe when bounded learners exceed their training distribution.

Source video

Key claims

7
causalVerification needed

Larger models achieve better compression, shrinking the generalization gap as a power law.

Evidence

Larger models achieve better compression, shrinking the generalization gap as a power law.

Question

Is the generalization gap vs. model size relationship empirically a power law across architectures and tasks?

Source video
factualVerification needed

Large models can achieve better than 1 bit per parameter compression under sequential coding.

Evidence

Sequential coding pushes compression to absolute limits, showing large models achieve better than 1 bit per parameter compression.

Question

What data and coding setup produce sub-1-bit-per-parameter compression, and is it general?

Source video
factualVerification needed

Epiplexity measures structural information extractable by a computationally bounded observer, defined via time-bounded MDL.

Evidence

Epiplexity measures the structural information content extracted by a computationally bounded observer.

Question

Is epiplexity formally well-defined and reproducible as a measurable quantity?

Source video
comparativeVerification needed

Natural language, code, and math are more structured and compressible than raw pixels.

Evidence

Natural language, code, and math are highly structured and compressible representations of information compared to raw pixels.

Question

Do empirical epiplexity measurements confirm a modality ranking with text/code above pixels?

Source video
causalVerification needed

Deterministic processes (synthetic data, pseudorandom generators) can enable superhuman systems like AlphaZero, contradicting the data processing inequality intuition.

Evidence

Information cannot be created by deterministic processes, yet synthetic data and pseudorandom number generators enable superhuman systems like AlphaZero.

Question

Under what bounded-computation conditions does deterministic generation produce usable new signal?

Source video
comparativeVerification needed

Sequence direction (e.g., left-to-right text) matters immensely even though information is independent of factorization order.

Evidence

Information is independent of factorization order, yet sequence direction (e.g., left-to-right text) matters immensely.

Question

How large is the measurable effect of factorization order on learned representations and performance?

Source video
factualVerification not requested

Traditional information theory assumes unlimited computation, whereas epiplexity incorporates computational bounds.

Evidence

Traditional information theory assumes unlimited computation, whereas epiplexity incorporates computational bounds to measure predictable structure in data.

Source video

Connections

5