ethan · Published 2026-09-19

Diogo Almeida (ex-OpenAI) reveals JEV: 200x faster, 400x cheaper, zero hallucinations.

privateReview: unreviewedConfidence: high
RLHFmode collapseassistanceautomationscaling lawsmode-collapseHuman-in-the-loophallucinationagent-reliabilitypost-trainingtask-selection
Open on YouTube ↗

Summary

Overview

  • Speaker: Diogo Almeida
  • Channel: ethan
  • Main topic: The divide between LLM overpromise and underdeliver, driven by optimization choices in RLHF.
  • Purpose: To explain the technical and economic disconnect between LLM capabilities and practical automation utility through the lens of optimization and RLHF. Diogo Almeida, former OpenAI researcher and co-author of GPT-4 and ChatGPT, examines the fundamental tension in current AI: why LLMs feel magical in demonstration yet fail at reliable automation. He argues that modern LLMs are optimized entirely for assistance (human-in-the-loop tasks) via RLHF, which inherently prioritizes plausibility over precision, leading to mode collapse and an inability to handle tasks with high-stakes decision-making without human oversight.

Topic Map

The Divide Between Overpromise and Underdeliver

  • Explanation: There is a massive chasm between the narrative of LLMs as an exponential 'machine god' and the reality that 95% of generative AI pilots fail or generate no economic value.
  • Key claims:
    • LLMs have saturated benchmarks while failing to trigger a broad economic revolution.
    • The AI community has shifted goalposts from automating most economically valuable work to vague goals like 'renaissance' or revenue targets.
  • Examples:
    • Microsoft CEO stating AI is generating basically no value.
    • MIT report showing 95% of generative AI pilots at companies are failing.
  • Terminology:
    • LLM overpromise
    • LLM underdeliver
    • Generative AI pilots
    • Economic revolution
  • Why it matters: Understanding this divide prevents misplaced investments and clarifies where AI can actually be applied effectively.

Assistance vs. Automation

  • Explanation: Tasks where LLMs succeed are 'human-in-the-loop assistance tasks' (e.g., ChatGPT, copilots), while tasks where they fail are 'machine-in-the-loop automation tasks' requiring high reliability without human oversight.
  • Key claims:
    • Today's LLMs are optimized entirely for assistance, not automation.
    • Customer service chatbots without decision-making power work well, but end-to-end automated customer service fails because models are not reliable enough.
  • Examples:
    • Customer service bots giving documentation vs. executing actions.
    • Coding agents operating without version control.
  • Terminology:
    • Human-in-the-loop assistance
    • Machine-in-the-loop automation
    • Reliability
    • Customer service
  • Why it matters: Matching the right task type to LLM capabilities determines whether a product succeeds or fails.

Optimization and RLHF Trade-offs

  • Explanation: RLHF (Reinforcement Learning from Human Feedback) optimizes models for human preference (HF), which leads to mode collapse and plausible-looking incorrect answers.
  • Key claims:
    • You get what you optimize for; optimizing for human preference (HF) sacrifices logical precision.
    • RLHF acts like a GAN, exhibiting extreme mode dropping where models favor safe, plausible-looking outputs over rarer correct modes.
  • Examples:
    • Yann LeCun's argument about auto-regressive models diverging exponentially with sequence length.
    • The contrast between pre-training (mode covering, creative) and RLHF (mode dropping, plausible).
  • Terminology:
    • RLHF
    • Mode collapse
    • Mode dropping
    • Mode covering
    • Auto-regressive models
    • Human preference
  • Why it matters: Explains why LLMs hallucinate plausibly and why they struggle with long multi-step reasoning tasks.

The Bitter Lesson of Data and Tasks

  • Explanation: Expanding on Rich Sutton's bitter lesson, data matters more than compute, and doing the right task matters more than data.
  • Key claims:
    • Compute matters more than algorithms, but data matters more than compute, and doing the right task matters more than data.
    • The success of Anthropic vs. OpenAI comes down to having a clearer north star on task optimization.
  • Examples:
    • The FLAN paper showing that fine-tuning with instruction data had an optimal amount of absolute zero for certain benchmarks.
  • Terminology:
    • Bitter lesson
    • Compute
    • Data
    • Doing the right task
    • Instruction tuning
  • Why it matters: Highlights that architectural scale alone cannot fix misaligned optimization objectives.

Key Points

LLMs optimize for assistance, not automation

  • Explanation: Because current models are trained via RLHF to please human raters, they excel at assisting humans who can catch errors, but fail when asked to automate high-stakes tasks autonomously.
  • Evidence: Studies showing developers using AI tools take longer or misjudge speedup, alongside widespread failures in autonomous enterprise automation.
  • Practical implication: Do not deploy LLMs for automated tasks requiring high reliability without a human in the loop.

RLHF causes mode dropping

  • Explanation: Optimizing for human preference incentivizes models to produce safe, plausible-sounding text, dropping low-probability correct paths in multi-step execution.
  • Evidence: Yann LeCun's theoretical arguments on error compounding in auto-regressive generation and observed hallucination patterns.
  • Practical implication: Expect plausible hallucinations when asking models to perform long chains of logical decisions.

The illusion of software engineering automation

  • Explanation: While coding assistants are popular, using them without rigorous safeguards like version control is akin to trusting high-stakes autonomous workflows without oversight.
  • Evidence: Taco Bell drive-thru automation failures and developer productivity studies.
  • Practical implication: Treat coding agents as assistants with version control safety nets rather than autonomous developers.

Frameworks, Models & Processes

Assistance vs. Automation Task Matrix

  • How it works: Classifies AI use cases based on whether they require human-in-the-loop assistance (where errors are cheap and caught by humans) or machine-in-the-loop automation (where errors carry financial or operational stakes).
  • Components:
    • Human-in-the-loop assistance tasks
    • Machine-in-the-loop automation tasks
    • Stake level assessment
    • Reliability threshold
  • When to use: When evaluating whether an LLM product is suitable for assistive workflows or autonomous deployment.

Examples & Case Studies

Taco Bell AI drive-thru ordering errors resulting in massive incorrect orders.

  • Illustrates: The failure of autonomous automation in high-stakes consumer-facing environments without proper reliability.
  • Lesson: You cannot trust LLMs to make decisions with stakes that you pay for when reliability is required.

Google's FLAN paper demonstrating instruction tuning.

  • Illustrates: How benchmark-driven tuning can sometimes show optimal performance at unexpected data scales.
  • Lesson: Doing the right task matters more than simply accumulating more data or compute.

Actionable Takeaways

  • Immediate:
    • Never deploy LLMs for automated decision-making in high-stakes environments without a human in the loop.
    • Recognize that coding agents and chatbots are assistance tools, not autonomous automation engines.
    • Evaluate AI tools based on whether they optimize for assistance or end-to-end reliability.
  • Strategic:
    • Understand that RLHF inherently introduces mode dropping and plausible hallucinations.
    • Focus on aligning data and task selection rather than relying solely on compute scaling.
    • Prepare for engagement-maximized AI systems as consumer applications evolve.
  • Questions to investigate:
    • How can future post-training methods overcome RLHF mode dropping?
    • What architectural shifts are needed to move from assistance to true reliable automation?
    • How will enterprise AI adoption change as companies realize the limitations of current LLM optimization?

Claims Worth Verifying

  • 95% of generative AI pilots at companies are failing according to an MIT report. (Statistical claim)
  • RLHF functions similarly to GAN mode dropping by favoring safe, plausible outputs over rare correct modes. (Technical claim)
  • The optimal amount of instruction-tuning data in early FLAN experiments was zero on certain benchmarks. (Empirical claim)

Notable Quotes

"AI: too good to be true, too bad to be useful" (at 0:07) "The bitterest lesson that can be simply explained, which is data matters way more than compute, and doing the right task matters way more than data." (at 19:04) "LLMs are more like software: unpredictable, error prone, malicious... it's best to use the same interfaces / controls we use for humans." (at 12:10)

Compressed Summary

  • Modern LLMs suffer from a divide between overpromise and underdeliver due to RLHF optimization.
  • RLHF optimizes models for assistance (human preference) rather than automation reliability.
  • Assistance tasks succeed because humans catch errors; automation tasks fail because models lack reliability and exhibit mode dropping.
  • Data and doing the right task matter more than raw compute scaling.
  • Keywords: rlhf, mode collapse, assistance, automation, scaling laws
  • Core insight: Current LLMs feel magical because RLHF optimizes them for human-in-the-loop assistance, but this same optimization makes them unreliable for autonomous automation.

Core insights

6
Mechanismmedium noveltymoderate evidence

RLHF behaves like a GAN: optimizing for human preference drives extreme mode dropping, so models prefer safe, plausible-looking outputs over rarer correct ones. This reframes hallucination not as a knowledge gap but as an artifact of the post-training objective, and explains why more pretraining scale alone cannot fix it.

Why it matters

If hallucination is an optimization artifact rather than a capability gap, then agent reliability work should target the post-training objective and decoding/task design, not just bigger models. It also predicts that failure concentrates in low-probability-but-correct reasoning paths, i.e. exactly the multi-step tasks automation needs.

Generalization

Any system trained against a preference proxy (human raters, reward models, judges) that is cheap to evaluate but only correlated with correctness will drop the rare modes that carry the highest task value. Applies to RLHF, RLAIF, reward-model selection, and LLM-as-judge fine-tuning loops.

RLHF acts like a GAN, exhibiting extreme mode dropping where models favor safe, plausible-looking outputs over rarer correct modes.
Open source video
You get what you optimize for; optimizing for human preference (HF) sacrifices logical precision.
Open source video
which inherently prioritizes plausibility over precision, leading to mode collapse and an inability to handle tasks with high-stakes decision-making without human oversight
Open source video
Mental Modelmedium noveltystrong evidence

The assistance vs. automation distinction is a deployability boundary, not a marketing distinction: models are optimized for human-in-the-loop assistance where errors are cheap and caught by a human, and systematically fail machine-in-the-loop automation where errors carry financial or operational stakes. The classification key is the stakes/reliability threshold of the task, not the task's domain.

Why it matters

It gives a concrete pre-deployment screen for agent projects: any task whose errors are not cheaply caught by a human falls outside current LLM competence unless a harness supplies the catch. This directly changes what you should promise stakeholders and where you spend reliability engineering.

Generalization

Scope any autonomous agent by the cost of an undetected error multiplied by the probability the model commits it; if that product exceeds the human's catch rate, the system must be re-architected as assistive or given a verifier.

Today's LLMs are optimized entirely for assistance, not automation.
Open source video
Classifies AI use cases based on whether they require human-in-the-loop assistance (where errors are cheap and caught by humans) or machine-in-the-loop automation (where errors carry financial or operational stakes).
Open source video
Do not deploy LLMs for automated tasks requiring high reliability without a human in the loop.
Open source video
Failure Modemedium noveltymoderate evidence

Autoregressive generation compounds error exponentially with sequence length, so long chains of logical decisions are structurally where plausibility-over-precision bites. The engineering corollary is to break long-horizon automation into short verifiable segments with checkpoints rather than trusting a single long generation.

Why it matters

It predicts that agent reliability decays with trajectory length and step count, meaning harness design (step granularity, intermediate verification, re-entry points) matters more than raw model quality for long-horizon tasks.

Generalization

Reliability of any chained generative pipeline is a multiplicative function of per-step correctness; halving steps or inserting verifiers yields superlinear reliability gains versus improving per-step quality marginally.

Yann LeCun's argument about auto-regressive models diverging exponentially with sequence length.
Open source video
Expect plausible hallucinations when asking models to perform long chains of logical decisions.
Open source video
Mental Modelmedium noveltymoderate evidence

The bitter lesson is extended into a hierarchy: compute beats algorithms, data beats compute, and choosing the right task/objective beats data. Misaligned optimization objectives therefore cannot be rescued by scale, which relocates the hard problem from architecture to task specification.

Why it matters

For teams, this argues that investment in task definition, evaluation targets, and objective selection has higher leverage than more training compute or cleverer architectures. It also implies that model-lab outcomes (e.g., Anthropic vs. OpenAI) are attributable to task focus rather than capability ceilings.

Generalization

In any learned system, the objective's alignment with the deployed task dominates scale effects; picking the wrong target makes every additional unit of compute marginal and possibly harmful.

Compute matters more than algorithms, but data matters more than compute, and doing the right task matters more than data.
Open source video
The success of Anthropic vs. OpenAI comes down to having a clearer north star on task optimization.
Open source video
Tradeoffmedium noveltymoderate evidence

Pretraining and RLHF sit on opposite sides of a mode-coverage trade-off: pretraining is mode covering and creative, RLHF is mode dropping and plausible. This implies a single post-trained artifact is the wrong default for tasks that require exploring low-probability correct solutions, and that routing or preserving a mode-covering path may be needed.

Why it matters

It suggests concrete architecture: keep access to less-aligned or higher-temperature mode-covering generation for search/exploration/branching, and reserve the preference-tuned model for safe communicative surfaces. It also flags that a single model serving both roles will silently underperform on the exploratory role.

Generalization

Alignment and diversity are in tension whenever a preference signal is used for post-training; systems needing both must separate the exploration policy from the communication/serving policy.

The contrast between pre-training (mode covering, creative) and RLHF (mode dropping, plausible).
Open source video
Practicemedium noveltymoderate evidence

Reliability for coding agents comes from the harness, not the model: version control functions as the human-in-the-loop catch, so coding agents work precisely because they operate inside a reversible, reviewable substrate. Running them without such safeguards is equivalent to unguarded autonomous automation.

Why it matters

It tells builders where to invest: cheap reversibility and diff-level review are the load-bearing safety mechanism for agentic coding, and removing them (or extending agents into irreversible actions) is the actual risk escalation.

Generalization

Any agent should run inside an environment that makes its actions cheap to inspect and cheap to undo; reliability engineering is largely about building that substrate rather than improving the model.

Treat coding agents as assistants with version control safety nets rather than autonomous developers.
Open source video
using them without rigorous safeguards like version control is akin to trusting high-stakes autonomous workflows without oversight
Open source video

Deep dives

5

Measuring RLHF-induced mode dropping in a task-relevant way

Research question

Can we construct a coverage metric that separates mode dropping caused by post-training from capability loss caused by the base model, and does that metric predict failure on tasks where the correct answer is a low-probability mode?

Why

Pass-1 asserts hallucination is an artifact of the post-training objective, not a knowledge gap, but offers no metric. Without a coverage/recovery measurement, every claim about mode dropping stays anecdotal and teams cannot tell when a post-training run has crossed into unsafe narrowing. A metric here converts a qualitative GAN analogy into an engineering dial for post-training pipelines.

RLHF acts like a GAN, exhibiting extreme mode dropping where models favor safe, plausible-looking outputs over rarer correct modes.
Open source video
The contrast between pre-training (mode covering, creative) and RLHF (mode dropping, plausible).
Open source video
You get what you optimize for; optimizing for human preference (HF) sacrifices logical precision.
Open source video
Source video

An operational task-scoring function for the assistance/automation boundary

Research question

Does a task score built from error cost, human catch rate, reversibility, and verifiability predict which generative AI pilots succeed versus fail, better than domain or model-capability classifications do?

Why

The summary gives a directional screen (stakes vs. catchability) but no scoring rule, yet claims this boundary explains 95% pilot failure. Turning the boundary into a numeric pre-deployment screen makes it falsifiable and directly usable on a project backlog, and it tests whether the boundary is a property of RLHF or of the surrounding harness.

Today's LLMs are optimized entirely for assistance, not automation.
Open source video
Classifies AI use cases based on whether they require human-in-the-loop assistance (where errors are cheap and caught by humans) or machine-in-the-loop automation (where errors carry financial or operational stakes).
Open source video
Do not deploy LLMs for automated tasks requiring high reliability without a human in the loop.
Open source video
Source video

Fit per-step reliability curves to derive agent horizon budgets

Research question

Does measured agent end-to-end success decay with trajectory length in the exponentially-compounding form predicted, and what step budget or verifier frequency restores a target end-to-end reliability?

Why

LeCun's compounding argument is cited theoretically and never quantified. If per-step success probability can be fitted empirically, harness design (segment length, checkpoint placement, re-entry cost) becomes a tunable parameter with a defensible budget instead of a stylistic preference. This is the difference between 'keep trajectories short' as folklore and as an engineering constraint.

Yann LeCun's argument about auto-regressive models diverging exponentially with sequence length.
Open source video
Expect plausible hallucinations when asking models to perform long chains of logical decisions.
Open source video
Source video

Does cheap programmatic verification shift the deployability boundary?

Research question

On domains with strong automatic checking (tests, type checks, formal checks), can verifier-gated automation reach reliability levels that pure RLHF-optimized models cannot, and does the boundary move as verifier coverage increases?

Why

Pass-1 frames the boundary as inherent to RLHF-optimized models, which would imply no automation is ever safe. But where errors are machine-caught rather than human-caught, the catch mechanism that the boundary assumes should still exist. Resolving this determines whether the boundary is a property of the model objective or of the surrounding substrate — a decisive architectural question.

Do not deploy LLMs for automated tasks requiring high reliability without a human in the loop.
Open source video
You cannot trust LLMs to make decisions with stakes that you pay for when reliability is required.
Open source video
Source video

Separating mode-covering exploration policies from mode-dropping serving policies

Research question

Does routing candidate generation to a mode-covering (base or high-temperature) path and reserving the preference-tuned model for the user-facing surface improve success on tasks where rare correct modes dominate, compared to sampling the aligned model repeatedly?

Why

Pass-1 claims a single post-trained artifact is the wrong default for exploration-heavy tasks but never tests the split-policy architecture. If confirmed, it changes agent runtime design: explicit search/candidate stages with a filter, rather than repeated sampling of an aligned model. If falsified, one aligned model serving both roles is justified, which is cheaper and simpler.

The contrast between pre-training (mode covering, creative) and RLHF (mode dropping, plausible).
Open source video
RLHF acts like a GAN, exhibiting extreme mode dropping where models favor safe, plausible-looking outputs over rarer correct modes.
Open source video
Source video

Article ideas

4

Hallucination Is a Post-Training Bug, Not a Knowledge Gap

If RLHF systematically drops the rare-but-correct modes that multi-step automation depends on, then scaling pretraining cannot fix agent reliability — the fix has to live in the post-training objective, the decoding policy, or the harness. Every reliability roadmap that begins with 'wait for a better model' is therefore misallocated, and the teams that win will be the ones that treat the aligned model as one unreliable subsystem and build verifiers, reversibility, and step checkpoints around it.

Angle

Reframe hallucination from a capability ceiling to an optimization artifact, then trace the concrete engineering consequences for anyone shipping agents.

Source video

The Stakes-and-Catch Screen: A Pre-Deployment Gate for Agent Projects

The assistance/automation boundary is not a domain question but a ratio question — undetected error cost times model error probability versus human catch rate — and projects that fail that screen should not be 'fixed' with a better prompt or a bigger model but re-architected as assistive, or given a verifier that supplies the missing catch. Stated this way, the boundary becomes a falsifiable scoring function that can be run over a backlog before a single line of agent code is written.

Angle

Turn a qualitative talk distinction into an operational, numeric pre-deployment screen, and show what re-architecture looks like when a project fails it.

Source video

Your Coding Agent's Safety Comes From Git, Not From the Model

Coding agents appear to work because version control supplies exactly the human-in-the-loop catch that the RLHF objective assumes exists — cheap inspection, cheap rollback, diff-level review. That means attribute their success to the substrate rather than the model, expect reliability to collapse the moment an agent takes irreversible actions, and treat building inspection/rollback primitives as the prerequisite for extending agents beyond reversible work.

Angle

Attribute agent reliability to its environment and invert the usual 'the model got better at coding' narrative.

Source video

Bitter Lesson, Extended: Task Choice Beats Data Beats Compute

If misaligned objectives cannot be rescued by scale, then the competitive variable in applied AI is task specification, not model access — and lab-level outcome differences should be read as differences in north-star clarity rather than capability ceilings. The practical corollary is that investment in evaluation targets and task definition has higher leverage than additional training compute, and picking the wrong target makes every extra FLOP marginal or harmful.

Angle

Extend the bitter lesson into a hierarchy and argue for reallocating applied-AI budget from compute to task definition and evaluation design.

Source video

Project ideas

4

ModeDrop Probe

beyond-evals

For a fixed base model and task suite, post-training on a preference proxy measurably collapses output entropy on reasoning traces and reduces pass@k on rare-but-correct solutions, and the size of this collapse predicts end-to-end failure rate on multi-step tasks better than benchmark accuracy does.

Proof of concept

Evaluate a base model and its post-trained variant on the same executable multi-step task suite; compute per-task output-distribution divergence, entropy of reasoning traces, and pass@k at high k for tasks whose correct solution requires an unusual path; regress task failure rate on the collapse metrics.

Measurement

Correlation between post-training-induced entropy collapse / pass@k drop on rare modes and observed end-to-end task failure rate; whether accuracy on standard benchmarks stays flat while rare-mode pass@k falls.

Source video

Horizon Budget Calculator

movement-lab

Agent end-to-end success decays as a multiplicative function of per-step correctness, so inserting a verifier every N steps yields superlinear reliability gains versus marginally improving per-step quality, and a maximum viable horizon length can be predicted in advance for a target end-to-end success rate.

Proof of concept

Instrument an agent runtime to log per-step success across traces of varying length; fit per-step success probability; predict end-to-end success for lengths L; then compare predicted versus observed success for runs with and without periodic verifier checkpoints.

Measurement

Fit quality of the exponential-decay model against observed success-vs-length curves; delta in end-to-end success rate per unit of added verification frequency versus per unit of per-step model improvement.

Source video

Stakes-and-Catch Scorer

gatehouse

A task score over error cost, human catch rate, reversibility, and verifiability predicts the success/failure of deployed generative AI use cases, and misclassified cases are exactly those that should have been re-architected as assistive or verifier-gated.

Proof of concept

Score a portfolio of internal or publicized AI deployments on the four axes, bin by assistive versus automated as deployed, and compare the score distribution against reported outcomes.

Measurement

Discriminative power (e.g., AUC or precision/recall) of the score for predicting deployment success versus a baseline that classifies only by domain or model capability.

Source video

Split-Policy Router

new

Routing candidate generation to a mode-covering path and using the preference-tuned model only for the final user-facing surface raises success on tasks where rare correct modes dominate, compared to repeatedly sampling the aligned model.

Proof of concept

Build a two-policy agent: an exploration stage that samples a less-aligned or higher-temperature model for candidates, and a serving stage that uses the aligned model to select and express the answer. Compare against an aligned-model-only baseline on the same rare-mode task suite.

Measurement

Task success rate on rare-mode-dominated tasks at matched token/compute budget; diversity of candidate sets pre-filter; number of correct-but-low-probability solutions surfaced per run.

Source video

Architectural implications

4

Reliability responsibility currently sits inside the model via RLHF alignment.

Before

Task success is assumed to follow from model capability and alignment.

After

Reliability is partitioned at the system boundary: a task-classification gate routes work to assistive (human-verified) or automated (verifier-gated) paths.

Consequence

Harness-level components — verifiers, reversibility, step checkpoints, approval surfaces — become first-class, and the model is treated as one unreliable subsystem among several.

Source video

A single post-trained model serves both exploration and safe communication.

Before

One RLHF'd model handles all generation.

After

Separate mode-covering paths (search, branching, candidate generation) from mode-dropping paths (user-facing answers).

Consequence

Agent architectures likely need explicit search/candidate-generation stages whose outputs are filtered, rather than sampling the aligned model repeatedly for correctness.

Source video

Autoregressive error compounding degrades long-horizon autonomy.

Before

Long tasks are one long generation or trajectory.

After

Long tasks are decomposed into short segments with intermediate verification and re-entry points.

Consequence

Step budget, verification frequency, and resumption cost become tuning parameters for agent runtimes.

Source video

Coding-agent safety derives from an external substrate.

Before

Agent capability is attributed to the model's coding skill.

After

Safety is attributed to the version-control substrate enabling cheap inspection and rollback.

Consequence

Extending agents beyond reversible actions requires building equivalent inspection/rollback primitives first; otherwise expected value turns negative.

Source video

Tradeoffs and failure modes

4

Human-preference alignment vs. logical precision

Benefit

RLHF produces models that are pleasant, safe, and plausible to human raters.

Cost or risk

It prioritizes plausibility over precision, sacrificing logical precision and dropping rare correct modes.

You get what you optimize for; optimizing for human preference (HF) sacrifices logical precision.
Open source video
Source video

Mode covering (pretraining) vs. mode dropping (RLHF)

Benefit

Pretraining covers modes and is creative; RLHF concentrates output on safe, plausible answers.

Cost or risk

The aligned model systematically avoids low-probability-but-correct paths needed for multi-step execution.

The contrast between pre-training (mode covering, creative) and RLHF (mode dropping, plausible).
Open source video
Source video

End-to-end automation autonomy

Benefit

Removes human latency and cost from workflows.

Cost or risk

Models are not reliable enough for tasks whose errors carry financial or operational stakes, producing failures at scale.

You cannot trust LLMs to make decisions with stakes that you pay for when reliability is required.
Open source video
Source video

Long-chain reasoning

Benefit

Enables multi-step planning and execution in a single model call flow.

Cost or risk

Errors compound exponentially with sequence length, so long logical chains fail plausibly rather than loudly.

Expect plausible hallucinations when asking models to perform long chains of logical decisions.
Open source video
Source video

Open questions

5

Can post-training objectives be designed to preserve mode coverage while still aligning behavior (e.g., avoiding the preference-proxy mode-dropping effect)?

Why unresolved

The summary states RLHF mode-drops but offers no alternative recipe or empirical comparison to a mode-preserving post-training method.

Research direction

Compare RLHF, distilled reward models, and diversity-preserving post-training on long-horizon executable tasks where rare correct modes dominate success.

Source video

How do you measure the degree of mode dropping induced by a given post-training run in a task-relevant way?

Why unresolved

Mode dropping is described qualitatively (GAN analogy, plausibility) with no metric or benchmark proposed.

Research direction

Build coverage/recovery metrics: pass@k on low-probability correct solutions, entropy collapse on reasoning traces, and divergence between base and post-trained output distributions.

Source video

What concrete task-selection criterion ('north star') converts a system from assistive to automatable?

Why unresolved

The summary asserts task choice dominates data and compute and cites a lab-level comparison, but gives no operational selection rule beyond the stakes/reliability screen.

Research direction

Formalize a task-scoring function over error cost, human catch rate, reversibility, and verifiability, then test predictive power against pilot success/failure.

Source video

Does the assistance/automation boundary shift for tasks with cheap programmatic verification (tests, type checks, formal checks)?

Why unresolved

The summary frames the boundary as an inherent property of RLHF-optimized models, without addressing domains where errors are machine-caught rather than human-caught.

Research direction

Measure whether verifier-gated automation reaches acceptable reliability on domains with strong automatic checking versus domains relying on human judgment.

Source video

How does the exponential error-compounding argument translate into quantitative step budgets for agent runtimes?

Why unresolved

Only LeCun's theoretical argument is cited; no per-step reliability curve or step-count guidance is provided.

Research direction

Empirically fit per-step success probability across agent traces to derive maximum viable horizon length for a target end-to-end success rate.

Source video

Key claims

7
factualVerification needed

Roughly 95% of generative AI pilots at companies are failing or generating no economic value.

Evidence

the reality that 95% of generative AI pilots fail or generate no economic value

Question

What is the source, sample, and definition of 'failing' in the cited pilot study, and does it distinguish assistive from automation use cases?

Source video
causalVerification needed

Today's LLMs are optimized entirely for assistance rather than automation.

Evidence

Today's LLMs are optimized entirely for assistance, not automation.

Question

Is the assistance/automation gap attributable to the RLHF objective, to evaluation and product incentives, or to both?

Source video
causalVerification needed

Optimizing for human preference causes mode dropping and plausible-but-incorrect outputs.

Evidence

RLHF acts like a GAN, exhibiting extreme mode dropping where models favor safe, plausible-looking outputs over rarer correct modes.

Question

Can mode dropping be isolated as the causal driver of hallucination independent of pretraining data limitations and decoding parameters?

Source video
comparativeVerification needed

Data matters more than compute, and doing the right task matters more than data.

Evidence

Compute matters more than algorithms, but data matters more than compute, and doing the right task matters more than data.

Question

Is there controlled evidence ranking task choice above data and data above compute, or is this an interpretive extension of the bitter lesson?

Source video
causalVerification needed

Autoregressive models diverge exponentially with sequence length, so long logical chains degrade.

Evidence

Yann LeCun's argument about auto-regressive models diverging exponentially with sequence length.

Question

What is the empirical per-step error rate and does measured agent success follow the predicted exponential decay?

Source video
opinionVerification needed

Instruction fine-tuning sometimes shows optimal benchmark performance at an unexpected data scale.

Evidence

The FLAN paper showing that fine-tuning with instruction data had an optimal amount of absolute zero for certain benchmarks.

Question

What was the exact FLAN finding on instruction-data scaling and which benchmarks showed non-monotonic or degenerate optima?

Source video
opinionVerification needed

Model-lab outcomes (Anthropic vs. OpenAI) are explained by clarity of task-optimization focus.

Evidence

The success of Anthropic vs. OpenAI comes down to having a clearer north star on task optimization.

Question

Is there comparative evidence that task-objective clarity, rather than data, talent, or distribution, explains differential outcomes?

Source video

Connections

5