Dwarkesh Patel · Published 2025-11-25

Ilya Sutskever – We're moving from the age of scaling to the age of research

privateReview: unreviewedConfidence: high
scalingReinforcement learningsigmoidentropyalignmentreinforcement-learningscaling-lawssigmoid-scalingkl-divergenceinformation-theoryrlvrsample-efficiency
Open on YouTube ↗

Summary

Overview

  • Speaker: Ilya Sutskever and Dwarkesh Patel
  • Channel: Dwarkesh Patel
  • Main topic: The transition from the age of scaling to the age of research in AI, reinforcement learning scaling limits, and AGI alignment.
  • Purpose: To provide an in-depth exploration of frontier AI scaling dynamics, reinforcement learning limits, and philosophical implications of artificial general intelligence. Ilya Sutskever discusses the shift in AI research from simple scaling laws to a new era of fundamental research, exploring why reinforcement learning is information inefficient compared to supervised learning, the sigmoid shape of RL scaling curves, and the deep analogies between human neural evolution and machine learning.

Topic Map

The Shift from Scaling to Research

  • Explanation: Discussion on how the industry has moved from simply scaling compute and data to needing deeper algorithmic research breakthroughs.
  • Key claims:
    • Pretraining data is finite and scaling laws are hitting new regimes.
    • We are moving back to an age of research because compute alone is no longer enough to bridge the gap.
  • Examples:
    • Transition from the 2012-2020 research era to the 2020-2025 scaling era and back.
  • Terminology:
    • scaling laws
    • pretraining
    • compute
  • Why it matters: It explains why future AI progress requires fundamental algorithmic innovations rather than just larger clusters.

RL Scaling and Information Inefficiency

  • Explanation: Analysis of why reinforcement learning requires vastly more compute samples than supervised learning.
  • Key claims:
    • RL requires unrolling entire trajectories to get a single binary reward signal.
    • The learning curve of RL follows a sigmoid shape rather than a pure power law.
  • Examples:
    • Using Gemini 3 to plot binary entropy and DKL divergence equations in Google Colab.
  • Terminology:
    • RLVR
    • sigmoid scaling
    • bits per sample
    • KL divergence
  • Why it matters: It reveals the fundamental bottlenecks in current alignment and RLHF pipelines.

Key Points

RL scaling is sigmoid-shaped

  • Explanation: Unlike pretraining power laws where models learn immediately, RL learns very little for a long time, then rapidly improves, and finally plateaus.
  • Evidence: Analysis from recent papers on RL compute scaling and Gemini 3 generated Python scripts.
  • Practical implication: Researchers must understand the pass-rate Goldilocks zone to effectively allocate RL compute.

Human vs Machine Evolution

  • Explanation: Humans evolved via RL over billions of years with pre-trained biological priors, whereas deep learning models rely heavily on pretraining data distributions.
  • Evidence: Discussion on brain regions, innate desires, and genetic priors.
  • Practical implication: Understanding biological learning priors can inform better artificial learning architectures.

Frameworks, Models & Processes

RL Information Gain Framework

  • How it works: Models information gain in RL as the binary entropy of outcome based on pass rates and KL divergence.
  • Components:
    • Trajectory length
    • Pass rate
    • KL divergence
    • Binary entropy calculation
  • When to use: When evaluating compute efficiency in reinforcement learning versus supervised learning.

Examples & Case Studies

Dwarkesh uses Gemini 3 to plot RL vs Supervised Learning information gain curves.

  • Illustrates: How AI tools can verify theoretical math and generate experimental code without bugs.
  • Lesson: AI assistants can bridge the gap between abstract mathematical hypotheses and empirical code verification.

Actionable Takeaways

  • Immediate:
    • Understand the sigmoid nature of RL scaling.
    • Recognize the data inefficiency of trajectory rollouts.
  • Strategic:
    • Prepare for the transition back to an era of fundamental AI research.
    • Re-evaluate alignment strategies in light of sentient AI emergence.
  • Questions to investigate:
    • How do we bridge the gap between RL efficiency and human learning priors?
    • What are the long-term economic impacts of broad AI deployment?

Claims Worth Verifying

  • RL scaling curves for LLMs follow a sigmoid shape. (empirical)
  • The number of samples needed to get a gradient update in RL scales exponentially with KL divergence. (theoretical)

Notable Quotes

"You know what's crazy? That all of this is real." (at 0:00) "The information gained is the binary entropy of the outcome." (at 56:49)

Compressed Summary

  • Shift from the age of scaling back to the age of research.
  • RL scaling follows a sigmoid curve rather than a power law.
  • RL is significantly more information inefficient than supervised learning.
  • Biological evolution provides interesting analogies for machine learning priors.
  • Keywords: scaling, reinforcement learning, sigmoid, entropy, alignment
  • Core insight: Reinforcement learning scaling is fundamentally sigmoid-shaped and information inefficient, requiring a transition from brute-force scaling to deep algorithmic research.

Core insights

5
Empirical Resulthigh noveltymoderate evidence

RL post-training scaling for LLMs follows a sigmoid curve — long flat region with almost no learning, then rapid improvement, then plateau — rather than the power law seen in pretraining; the pass rate of the task determines where on that curve you land, creating a narrow 'Goldilocks zone' for RL compute allocation.

Why it matters

If RL gains are sigmoid, then the marginal return on RL compute is highly non-linear and budget-dependent: pouring compute into a task whose pass rate sits in the flat region produces almost nothing, while the same compute near the inflection point produces large jumps. This changes how teams should schedule and gate RL training runs, and makes pass-rate instrumentation a first-class metric rather than an afterthought.

Generalization

Any learning loop where reward is sparse and binary (RLHF, RLVR, agentic task success, verifier-gated code generation) should be expected to show sigmoid rather than power-law returns, and should be scheduled by proximity to the inflection point, not by raw compute volume.

Unlike pretraining power laws where models learn immediately, RL learns very little for a long time, then rapidly improves, and finally plateaus.
Open source video
RL scaling curves for LLMs follow a sigmoid shape.
Open source video
Researchers must understand the pass-rate Goldilocks zone to effectively allocate RL compute.
Open source video
Mechanismhigh noveltymoderate evidence

RL's information gain per sample is bounded by the binary entropy of the outcome and degrades as a function of KL divergence from the policy: unrolling an entire trajectory collapses to a single binary reward bit, and the sample count needed for a gradient update scales exponentially with KL divergence.

Why it matters

This reframes RL inefficiency as a measurable information-theoretic quantity rather than a vague complaint. It gives engineers a concrete diagnostic — compute outcome entropy from pass rate, and project the number of rollouts needed as policy diverges — which can be used to decide when a trajectory-based reward is simply not worth the compute versus when a denser reward signal or supervised objective would be cheaper.

Generalization

For any system where a long rollout yields a low-bit reward (binary success/failure, single thumbs-up/down, single test pass), the information-per-token ratio of the training signal is the governing cost constraint; loss functions with dense per-step signal should dominate whenever they are constructible.

RL requires unrolling entire trajectories to get a single binary reward signal.
Open source video
The information gained is the binary entropy of the outcome.
Open source video
The number of samples needed to get a gradient update in RL scales exponentially with KL divergence.
Open source video
Predictionmedium noveltymoderate evidence

The frontier is shifting back from 'age of scaling' to 'age of research' because pretraining data is finite and scaling laws are entering new regimes, so further capability gains must come from algorithmic breakthroughs rather than larger clusters.

Why it matters

This is an architectural-budget claim: infrastructure planning that assumes monotone returns from more data/compute is betting against a finite-resource constraint. Teams should expect the relative value of research, harness design, and data quality work to rise versus cluster size, and should reorganize roadmaps accordingly.

Generalization

Any compute-scaling program that depends on a finite input (web text, human demonstrations, verifier coverage) eventually hits a regime change where the bottleneck migrates from resources to algorithm design — capacity planning should track the scarce input, not the accelerator count.

Pretraining data is finite and scaling laws are hitting new regimes.
Open source video
We are moving back to an age of research because compute alone is no longer enough to bridge the gap.
Open source video
Mental Modelmedium noveltyweak evidence

Biological intelligence is framed as an RL process running on top of a pre-trained set of evolutionary/genetic priors (brain regions, innate desires), which is structurally analogous to how LLMs depend on a pre-training distribution before RL post-training — implying priors, not RL mechanics, are the differentiator.

Why it matters

If the human advantage comes from inherited priors rather than from the RL loop itself, then the highest-leverage engineering question is what to bake into the base model or harness before RL rather than how to run more RL. This shifts investment toward architecture, initialization, and environment/curriculum design.

Generalization

In any learning system, the choice of start state determines the achievable region of the RL search; improvements to initialization and environment shaping can dominate improvements to the optimizer.

Humans evolved via RL over billions of years with pre-trained biological priors, whereas deep learning models rely heavily on pretraining data distributions.
Open source video
Practicemedium noveltymoderate evidence

Frontier models were used to generate and execute the mathematical/plotting code that produced the empirical RL-vs-supervised information-gain curves, closing the loop between a theoretical hypothesis and verified experimental output within a single tool session.

Why it matters

This is a concrete example of a theory-to-experiment harness where the model writes and runs the code that tests the model's own framing. It suggests a practical research workflow: keep hypotheses expressed as runnable equations so an assistant can generate, execute, and falsify them immediately rather than after a manual coding cycle.

Generalization

Agent harnesses that can produce and execute the measurement code for their own training dynamics — entropy curves, pass-rate sweeps, KL tracking — compress the research iteration loop and make claims like 'sigmoid scaling' empirically checkable on demand.

Dwarkesh uses Gemini 3 to plot RL vs Supervised Learning information gain curves.
Open source video
AI assistants can bridge the gap between abstract mathematical hypotheses and empirical code verification.
Open source video

Deep dives

4

Sigmoid RL scaling and pass-rate Goldilocks zone

Research question

How does the pass rate of a task determine the position of the inflection point on the RL sigmoid curve, and can we derive a scheduling rule to allocate RL compute optimally?

Why

If RL gains follow a sigmoid, marginal returns are highly non-linear and pass-rate dependent; teams need to know when additional compute will yield gains versus when it is wasted, and how to adjust task difficulty or reward shaping to stay near the inflection.

Unlike pretraining power laws where models learn immediately, RL learns very little for a long time, then rapidly improves, and finally plateaus.
Open source video
Researchers must understand the pass-rate Goldilocks zone to effectively allocate RL compute.
Open source video
Source video

Information-theoretic bounds on RL sample efficiency

Research question

How does the required number of rollouts for a gradient update scale with KL divergence in practice, and at what divergence threshold does a denser reward signal become more compute-efficient than trajectory-based RL?

Why

This provides a concrete diagnostic for deciding when to abandon trajectory-based RL in favor of supervised or dense-reward alternatives, which is crucial for efficient pipeline design.

RL requires unrolling entire trajectories to get a single binary reward signal.
Open source video
Models information gain in RL as the binary entropy of outcome based on pass rates and KL divergence.
Open source video
Source video

Priors vs RL: The role of initialization and environment shaping

Research question

Can engineered priors (initialization, curriculum, environment shaping) significantly improve RL sample efficiency compared to standard pretraining, and how much of the human-machine gap is attributable to priors versus the RL algorithm?

Why

If priors are the differentiator, then investment should shift toward architecture and environment design rather than scaling RL compute, changing research priorities.

Humans evolved via RL over billions of years with pre-trained biological priors, whereas deep learning models rely heavily on pretraining data distributions.
Open source video
Source video

The shift to the age of research: infrastructure and evaluation implications

Research question

What are the leading indicators that a scaling program has entered the regime where algorithmic innovation dominates, and how should evaluation infrastructure and research roadmaps be restructured to track the scarce input?

Why

Teams betting on continued scaling may be misallocating resources; understanding the transition point helps reorient investment toward research and harness design.

Pretraining data is finite and scaling laws are hitting new regimes.
Open source video
We are moving back to an age of research because compute alone is no longer enough to bridge the gap.
Open source video
Source video

Article ideas

4

The End of Scaling Is a Budget Problem, Not a Capability Problem

The shift from the age of scaling to the age of research is driven by finite pretraining data and diminishing returns, not by a sudden loss of capability; therefore, infrastructure roadmaps that anchor on cluster size are betting against a finite resource, and the rate-limiting investment must shift to algorithmic and harness-level innovation.

Angle

Economic and strategic analysis of scaling constraints, with a call to reorganize R&D budgets around scarce inputs rather than accelerator counts.

Source video

Your RLHF Pipeline Is Probably in the Flat Part of the Sigmoid

Most alignment pipelines waste compute because they ignore pass-rate and KL divergence, landing in the flat region of the RL sigmoid; instrumenting outcome entropy and policy divergence is a first-class scheduling requirement, not an afterthought.

Angle

Practical engineering argument with a diagnostic framework for when to adjust task difficulty or switch objectives.

Source video

The Information-Theoretic Case Against Trajectory-Based RL

Because a full trajectory collapses to a single binary reward bit, and required samples grow exponentially with KL divergence, dense-reward and supervised objectives will dominate for most tasks; trajectory-based RL should be reserved for cases where pass rate is near the inflection and KL is tightly constrained.

Angle

Information-theoretic critique of current RLHF/RLVR methods, with concrete cost formulas and decision criteria.

Source video

Priors Are the Product: Why Initialization and Environment Shaping Matter More Than RL Mechanics

The human-machine learning gap is primarily due to inherited biological priors, not the RL algorithm; therefore, the highest-leverage engineering effort is in base model design, curriculum, and environment shaping, with RL as refinement rather than the main event.

Angle

Architectural argument shifting focus from RL loops to start-state design, supported by the evolutionary analogy.

Source video

Project ideas

4

Sigmoid Scheduler

movement-lab

For a given task, the RL learning curve inflection point can be predicted from initial pass rate, and scheduling compute to focus near that inflection yields higher final performance per FLOP than uniform compute allocation.

Proof of concept

Instrument an RLHF/RLVR pipeline to measure pass rate and KL divergence, then run experiments with uniform vs. pass-rate-targeted compute allocation on tasks with varying difficulty.

Measurement

Compare final reward/accuracy per unit compute between the two allocation strategies across multiple tasks.

Source video

KL-Budget Calculator

gatehouse

The number of rollouts required for a gradient update grows exponentially with KL divergence, and can be predicted from measured KL to within a factor of 2, enabling an automatic switch to supervised or dense-reward training when projected cost exceeds a threshold.

Proof of concept

Build a tool that measures KL divergence during RL training and estimates required rollouts using the exponential model; validate on multiple tasks by comparing predicted vs. actual rollouts to reach a target reward.

Measurement

Relative error between predicted and actual rollout counts; frequency of correct switch decisions.

Source video

Prior Engineering Benchmark

movement-lab

Adding engineered priors (curriculum, environment shaping, initialization) significantly improves RL sample efficiency on sparse-reward tasks, closing a measurable fraction of the gap to human-like learning.

Proof of concept

Create a suite of sparse-reward tasks, train models with and without engineered priors, and measure sample efficiency (rollouts to threshold).

Measurement

Rollouts to reach target success rate, compared across conditions.

Source video

Theory-to-Experiment Harness

beyond-evals

Model-generated experimental code for testing RL scaling hypotheses (e.g., plotting entropy curves) is accurate enough to replace manual coding for rapid iteration, with error rates below 5% for standard mathematical plots.

Proof of concept

Have a LLM generate code to compute and plot binary entropy and KL divergence curves from pass-rate data, then compare its outputs to manually verified calculations.

Measurement

Error rate in calculated values, human evaluation of code correctness, time saved vs. manual.

Source video

Architectural implications

3

RL post-training yields one binary signal per full trajectory rollout.

Before

Training budget allocated in proportion to rollouts generated, on the assumption that more trajectories linearly improve the model.

After

Budget allocation must be conditioned on outcome entropy (derived from pass rate) and on policy KL divergence, since rollout count needed scales exponentially with divergence.

Consequence

Harnesses need per-task pass-rate and KL instrumentation as scheduling signals; tasks in the flat sigmoid region should be deprioritized or re-authored rather than scaled.

Source video

Pretraining data is finite and scaling laws are in new regimes.

Before

Capability roadmaps anchored to cluster size and data volume as the primary lever.

After

Roadmaps anchored to algorithmic and harness-level innovation, with data/compute as necessary but insufficient inputs.

Consequence

Research and system-design headcount and evaluation infrastructure become the rate-limiting investment rather than raw compute procurement.

Source video

Human capability is attributed to pre-trained biological priors on top of RL, not to RL alone.

Before

RL post-training treated as the primary place to add capability and alignment.

After

Where priors live (base model, system prompt, harness constraints, tool interfaces) becomes the primary design surface, with RL as refinement.

Consequence

Agent scaffolding, environment curricula, and initialization move earlier in the pipeline; RL becomes the last-mile rather than the whole story.

Source video

Tradeoffs and failure modes

4

Trajectory-based RL reward sparsity

Benefit

Binary outcome rewards generalize to any task with a checkable success condition and require no reward model or dense human annotation.

Cost or risk

A full trajectory collapses into one bit, making the scheme information-poor per unit compute and structurally dependent on pass rate being near the sigmoid inflection.

RL requires unrolling entire trajectories to get a single binary reward signal.
Open source video
Source video

KL divergence between policy and behavior

Benefit

Staying close to the reference policy stabilizes RL updates and bounds reward hacking.

Cost or risk

Sample requirement grows exponentially with KL divergence, so accepting larger policy movement to gain capability imposes super-linear compute cost.

The number of samples needed to get a gradient update in RL scales exponentially with KL divergence.
Open source video
Source video

Scaling-era continuation

Benefit

Extending compute and data keeps near-term capability gains flowing without requiring new algorithmic insight.

Cost or risk

Pretraining data is finite and scaling laws are entering new regimes, so continued reliance on scaling risks stalling progress once the finite input is exhausted.

Pretraining data is finite and scaling laws are hitting new regimes.
Open source video
Source video

RL plateau as capability ceiling

Benefit

Post-plateau behavior is stable and cheap to maintain, with little further drift.

Cost or risk

If the sigmoid plateaus before the target capability is reached, additional RL compute is wasted; without per-task inflection tracking, teams may not detect that they are in the flat tail.

RL learns very little for a long time, then rapidly improves, and finally plateaus.
Open source video
Source video

Open questions

4

Can RL sample efficiency be raised to close the gap with human learning priors, or is the gap structural to how priors are inherited biologically?

Why unresolved

The summary draws the analogy between human evolutionary priors and model pretraining but provides no mechanism for transferring such priors or measuring the efficiency delta beyond the qualitative comparison.

Research direction

Build controlled comparisons of RL sample efficiency with and without engineered priors (initialization, curriculum, environment shaping) to quantify how much of the gap is attributable to priors rather than to the RL algorithm.

Source video

How does the exponential dependence of required samples on KL divergence change the practical design of RLHF/RLVR pipelines?

Why unresolved

The relationship is stated as a theoretical claim in the summary but no measurement or threshold is given for where it becomes prohibitive in real pipelines.

Research direction

Measure rollout cost versus achieved KL divergence across several tasks with varying pass rates, and derive a scheduling rule for when to switch to a denser or supervised signal.

Source video

What determines where the inflection point of the RL sigmoid sits for a given task, and can it be moved deliberately?

Why unresolved

The summary identifies the pass-rate Goldilocks zone as the controlling variable but does not specify how pass rate should be manipulated or what range is optimal.

Research direction

Run pass-rate sweeps on fixed tasks by varying task difficulty and reward shaping, then plot compute-to-inflection as a function of pass rate to define an actionable target band.

Source video

What are the long-term economic and alignment impacts of broad AI deployment as the field re-enters a research-driven era?

Why unresolved

The summary lists economic impact and sentient-AI alignment as strategic concerns but provides no evidence, mechanism, or metric for either.

Research direction

Establish evaluation criteria for alignment under capability regimes that are not reachable by scaling alone, and instrument deployment economics against capability inflection points.

Source video

Key claims

7
opinionVerification needed

RL scaling curves for LLMs follow a sigmoid shape rather than a power law.

Evidence

RL scaling curves for LLMs follow a sigmoid shape.

Question

Do published RL compute-scaling curves for LLMs consistently show a sigmoid (flat, rise, plateau) shape rather than a power law across tasks and model sizes?

Source video
causalVerification needed

The number of samples needed to get a gradient update in RL scales exponentially with KL divergence.

Evidence

The number of samples needed to get a gradient update in RL scales exponentially with KL divergence.

Question

Does the required sample count for a gradient update empirically grow exponentially in KL divergence across a range of simulated or real RL settings?

Source video
factualVerification needed

RL information gain per update is the binary entropy of the outcome, computed from pass rate.

Evidence

The information gained is the binary entropy of the outcome.

Question

Is per-update RL information gain well-modeled as the binary entropy of the outcome given pass rate, and how well does that model predict measured learning curves?

Source video
factualVerification needed

A full RL trajectory yields only a single binary reward signal.

Evidence

RL requires unrolling entire trajectories to get a single binary reward signal.

Question

In current RLVR/RLHF pipelines, is the reward signal genuinely one bit per trajectory, or do intermediate steps or value estimates contribute additional signal?

Source video
causalVerification needed

Pretraining data is finite and scaling laws are hitting new regimes, so compute alone is no longer sufficient.

Evidence

Pretraining data is finite and scaling laws are hitting new regimes.

Question

What evidence indicates scaling laws have entered a new regime rather than a smooth continuation, and how much usable pretraining data remains?

Source video
comparativeVerification needed

Humans evolved via RL over billions of years with pre-trained biological priors, while deep learning models rely heavily on pretraining data distributions.

Evidence

Humans evolved via RL over billions of years with pre-trained biological priors, whereas deep learning models rely heavily on pretraining data distributions.

Question

Is the evolutionary-prior analogy load-bearing for engineering decisions, and what measurable prediction does it make about model sample efficiency?

Source video
opinionVerification needed

AI assistants can generate and verify the experimental code for theoretical math hypotheses, bridging theory and empirical verification.

Evidence

AI assistants can bridge the gap between abstract mathematical hypotheses and empirical code verification.

Question

How often do model-generated theoretical plots and derivations survive independent scrutiny, and what error rate applies to assistant-generated verification code?

Source video

Connections

5