Yann LeCun: World Models: Enabling the next AI revolution

Overview

• Speaker: Yann LeCun
• Channel: Computer Vision and Geometry Group, ETH Zurich
• Main topic: World Models, Joint-Embedding Predictive Architectures (JEPA), and Energy-Based Models for Artificial Intelligence
• Purpose: To explain the limitations of current AI architectures, particularly LLMs and generative video models, and present world models and JEPA as the necessary architectural foundation for achieving human-level AI. Yann LeCun argues that current machine learning paradigms like LLMs and purely generative video models fall short of true animal- and human-level intelligence because they lack grounded world models, common sense, and the ability to reason and plan in high-dimensional continuous spaces. He introduces JEPA (Joint Embedding Predictive Architecture) and energy-based models as the path forward for autonomous machine intelligence, emphasizing hierarchical planning, representation-space prediction, and non-generative self-supervised learning.

Topic Map

Why AI Sucks Compared to Animals

• Explanation: Comparing human and animal learning capabilities against current AI systems reveals a massive gap in zero-shot learning, common sense, and handling continuous high-dimensional data.
• Key claims:
  • Machine learning struggles with high-dimensional continuous data like video and audio.
  • Humans and animals learn common sense and physical intuition rapidly with very few samples.
  • Languages are simple, but the real world is messy and continuous.
• Examples:
  • A 10-year-old can clean a table and load a dishwasher zero-shot, while robots cannot.
  • A teenager learns to drive in 20 hours, whereas self-driving cars need millions of training hours.
• Terminology:
  • zero-shot
  • common sense
  • Moravec's paradox
  • declarative knowledge
• Why it matters: Highlights why current AI approaches cannot achieve general intelligence without grounding in the physical world.

What is Intelligence, Really?

• Explanation: Drawing on Jean Piaget and developmental psychology, intelligence is defined not by accumulated knowledge or static skills, but by adaptability and the ability to handle new situations with little or no prior training.
• Key claims:
  • Intelligence is what you do when you don't know.
  • Intelligence is not an accumulation of declarative knowledge or a collection of skills.
  • Human intelligence is highly specialized, and AGI as a general unspecialized phrase is nonsensical.
• Examples:
  • Jean Piaget's views on cognitive development and adaptation.
• Terminology:
  • declarative knowledge
  • adaptive intelligence
  • AGI
• Why it matters: Redefines how we should measure and build artificial intelligence.

How Humans and Animals Learn

• Explanation: Cognitive science and developmental psychology show that infants build mental models of the world primarily through passive observation in their early months.
• Key claims:
  • Humans and animals learn mental models of the world driven by objectives.
  • Infants learn physical concepts like object permanence and gravity through passive observation of video.
  • The world is three-dimensional, and distance from the observer is the best way to explain changes.
• Examples:
  • An 8-month-old baby observing toy drop experiments and recognizing physical violations.
• Terminology:
  • object permanence
  • intuitive physics
  • mental models
  • violation of expectation
• Why it matters: Demonstrates that world models can be learned through passive video observation without explicit action.

Inference: Feed-Forward vs. Search/Optimization

• Explanation: Contrasting system 1 action prediction (LLM-style feed-forward token generation) with system 2 planning (world model search and optimization).
• Key claims:
  • Reasoning and planning require search and optimization.
  • Feed-forward propagation in autoregressive models is computationally limited for complex reasoning.
  • Planning involves imagining action sequences, predicting outcomes with a world model, and optimizing against an objective.
• Examples:
  • Robot arm object manipulation tasks using model predictive control (MPC).
• Terminology:
  • model predictive control
  • system 1
  • system 2
  • optimization
  • cost function
• Why it matters: Explains why autoregressive LLMs struggle with true logical reasoning and planning.

World Model Architectures: Generative vs. JEPA

• Explanation: Comparing generative architectures (like LLMs and diffusion models) with Joint Embedding Predictive Architectures (JEPA) for world modeling.
• Key claims:
  • JEPAs learn abstract representations of data and ignore unpredictable details like noise.
  • Generative models predict every detail at the pixel level, making them blurry and unsuitable for high-dimensional continuous data.
  • JEPA performs prediction in abstract representation space rather than pixel space.
• Examples:
  • Block tower prediction models and video prediction experiments.
• Terminology:
  • JEPA
  • joint embedding
  • predictive architecture
  • pixel space
  • representation space
• Why it matters: Establishes JEPA as the correct architectural paradigm for building world models.

Training JEPA and Preventing Collapse

• Explanation: Addressing the collapse problem in joint-embedding architectures using information maximization (SIGReg) and distillation techniques.
• Key claims:
  • Unregularized joint-embedding architectures suffer from collapse where encoders ignore inputs.
  • SIGReg (Sketched Isotropic Gaussian Regularization) maximizes information content by making embedding distributions isotropic Gaussian along random projections.
  • Contrastive methods and regularized methods prevent collapse in energy-based models.
• Examples:
  • LeWorldModel trained with SIGReg on robotic tasks.
  • I-JEPA and V-JEPA architectures.
• Terminology:
  • SIGReg
  • collapse problem
  • isotropic Gaussian
  • information maximization
  • contrastive methods
• Why it matters: Provides a mathematically sound way to train non-generative self-supervised representation models.

Learning a Hierarchy of Abstractions

• Explanation: Complex systems require hierarchical planning across multiple levels of abstraction and time scales.
• Key claims:
  • Lower levels make short-range predictions with details; higher levels make long-range predictions with fewer details.
  • Science and modeling are about finding the right representation at the right level of abstraction.
  • Hierarchical planning allows complex long-term goal realization by generating subgoals.
• Examples:
  • Planning a trip from New York to Paris using hierarchical subgoals.
• Terminology:
  • hierarchical planning
  • level of abstraction
  • subgoals
  • quantum field to ecosystem hierarchy
• Why it matters: Essential for scaling world models to real-world control and reasoning tasks.

Key Points

AI must abandon generative pixel-level prediction

• Explanation: Predicting every pixel or token in high-dimensional continuous data leads to blurry, intractable models that fail to capture underlying semantics.
• Evidence: Early video prediction experiments showed blurry outputs; JEPAs predict in abstract feature space instead.
• Practical implication: Researchers should focus on representation-space prediction rather than video generation for building world models.

Planning requires world models and optimization

• Explanation: True agentic AI requires the ability to imagine future states, evaluate them against objectives and guardrails, and optimize action sequences.
• Evidence: Model Predictive Control (MPC) and hierarchical planning frameworks.
• Practical implication: Build agentic systems using System 2 planning loops rather than pure autoregressive token generation.

SIGReg prevents collapse in JEPA

• Explanation: To train self-supervised joint embedding models without collapse, regularizers like SIGReg ensure variance and independence across embedding dimensions.
• Evidence: LeWorldModel success across multiple robotic simulation benchmarks.
• Practical implication: Use SIGReg or similar information maximization regularizers when training non-generative representation models.

Frameworks, Models & Processes

Joint-Embedding Predictive Architecture (JEPA)

• How it works: Encodes two inputs (x and y) into abstract representations and uses a predictor to forecast y's representation from x's representation conditioned on an action.
• Components:
  • Encoder (Enc)
  • Predictor (Pred)
  • Distance Metric (D)
  • Action Conditioning (a)
• When to use: For self-supervised representation learning and world modeling on high-dimensional continuous data like video and sensor outputs.

Energy-Based Models (EBM)

• How it works: Assigns low energy to compatible pairs of variables and higher energy to incompatible pairs using an implicit energy function optimized via energy minimization.
• Components:
  • Energy Landscape
  • Contrastive or Regularized Loss
  • Argmin Inference
• When to use: For modeling complex dependencies and multimodal prediction tasks where multiple valid answers exist.

Examples & Case Studies

An 8-month-old infant observing a car pushed off a platform floating in mid-air shows surprise.

• Illustrates: Infants possess intuitive physics and object permanence learned through passive observation.
• Lesson: World models can be acquired passively through observation before active interaction.

LeWorldModel applied to robotic pushing and manipulation tasks (Push-T, Reacher).

• Illustrates: Action-conditioned JEPA world models enable zero-shot goal reaching and MPC planning in simulation.
• Lesson: World models trained with SIGReg successfully support planning and control.

Actionable Takeaways

• Immediate:
  • Stop relying solely on LLMs for physical world reasoning.
  • Explore JEPA and non-generative self-supervised representation learning.
  • Implement SIGReg for training stable joint-embedding architectures.
• Strategic:
  • Shift AI research from video generation to abstract world modeling.
  • Develop hierarchical planning systems combining perception, memory, world models, and actor-critics.
  • Focus on physical AI and robotics through grounded world models.
• Questions to investigate:
  • How to scale SIGReg and JEPA to real-world multi-modal data efficiently?
  • How to automatically learn hierarchical abstractions for complex control tasks?
  • What are the precise mathematical bounds of information maximization in high-dimensional representations?

Claims Worth Verifying

• A four-year-old child has seen more data (1.1E14 bytes) through sensory input than an LLM trained on text. (analytical estimate)
• SIGReg recovers independent variables if they are Gaussian. (theoretical theorem)

Notable Quotes

"Intelligence is not what you know, it's what you do when you don't know." (at 6:56) "The real world is messy, language is simple." (at 1:51) "IF YOU ARE INTERESTED IN HUMAN-LEVEL AI, DON'T WORK ON LLMS" (at 80:28)

Compressed Summary

• LLMs and generative models lack grounded physical world models.
• JEPA enables non-generative representation learning in abstract space.
• SIGReg prevents representation collapse by enforcing isotropic Gaussian distributions.
• Hierarchical planning via world models is essential for complex reasoning and robotics.
• Keywords: world models, jepa, sigreg, energy-based models, planning, representation learning
• Core insight: Building human-level AI requires action-conditioned world models trained via non-generative self-supervised representation learning (JEPA) rather than autoregressive text or pixel generation.