Reinforcement Learning without Verifiable Rewards — Will Brown, Prime Intellect

Overview

• Speaker: Will Brown
• Channel: AI Engineer
• Main topic: Reinforcement learning on messy, real-world agent tasks without verifiable rewards
• Purpose: To explain how to extend reinforcement learning to messy real-world tasks where reward signals are unclear, and present methods for manufacturing reward signals and building self-evolving agent systems. Will Brown discusses reinforcement learning (RL) beyond domains with clean verifiable rewards like math and code. He explores scaling RL to messy real-world agent tasks by manufacturing signal through grounding from source material, calibrated judges for open-ended work, and scaling search and test-time compute. He outlines Prime Intellect's stack and vision for continual learning where agents self-improve in production.

Topic Map

Introduction to RL without Verifiable Rewards

• Explanation: Many real-world tasks lack clean verifiable rewards. Scaling RL requires handling messy environments where the goal or success criteria is not straightforward.
• Key claims:
  • RL and RLVR have taken off, but often we don't have verifiable rewards.
  • Messy real-world tasks require new techniques beyond simple verification.
• Examples:
  • Running around writing reports, booking flights, handling refunds.
• Terminology:
  • RLVR
  • verifiable rewards
  • policy gradient
• Why it matters: Most valuable real-world agent applications are messy and lack binary ground-truth answers.

The Prime Intellect Stack

• Explanation: Overview of Prime Intellect's infrastructure spanning compute, training frameworks, environments, and model orchestration.
• Key claims:
  • Prime Intellect provides tools across the stack from GPU cluster orchestration to the prime-r1 open ML trainer and lab environments.
• Examples:
  • Tasksets, harnesses, verifiers, hosted training, and evals.
• Terminology:
  • prime-r1
  • lab
  • GPU cluster orchestration
• Why it matters: Enables practitioners to act as their own research lab and optimize model weights for custom tasks.

Environments and Verifiable Rewards

• Explanation: Defining environments as tasks, harnesses, and rewards, and contrasting easy verifiable cases (math, code, tool use) with fuzzy real-world agent tasks.
• Key claims:
  • Environments and evals are essentially the same objects, useful for SFT, RL, distillation, and prompting.
  • Verifiable rewards work well for math, code test cases, and tool use with database states.
• Examples:
  • Golden answers in math, test cases in code, check_state(db) for tool use.
• Terminology:
  • verifiable rewards
  • harness
  • test cases
• Why it matters: Establishes the baseline of where RL is easiest before tackling non-verifiable settings.

Challenges in Real Agent Tasks and Evals

• Explanation: Most real tasks are not verifiable: tasks do not scale by hand, open-ended outputs lack clean checks, coverage is unbounded, and models exploit weak verifiers (reward hacking).
• Key claims:
  • Manual curation of evals does not scale.
  • Real-world tasks have unclear signals.
  • Models exploit loose reward proxies through reward hacking.
• Examples:
  • 'Is this a good report?', 'Did it book the right flight?'
• Terminology:
  • reward hacking
  • open-ended outputs
  • unbounded coverage
• Why it matters: Highlights why naive RL fails in production without robust signal manufacturing.

Continual Learning Goal

• Explanation: The objective of continual learning where deployed agents in production generate real traces, mine new tasks and worlds, and train via online RL to produce better agents.
• Key claims:
  • Models should be deployed in realistic messy settings and learn as they go from mistakes.
  • Automated environment design enables self-improvement in production.
• Examples:
  • Production traces mined into new tasks and worlds for online RL.
• Terminology:
  • continual learning
  • online RL
  • production traces
• Why it matters: Shifts model optimization from static offline benchmarks to active self-improving loops.

Manufacturing Signal

• Explanation: Techniques to manufacture signal when rewards are missing: grounding from source material, judges for open-ended work, and scaling search/test-time compute.
• Key claims:
  • Grounding leverages source material like docs, code, and production traces to create grounded tasks.
  • Judges use LLMs scoring against references or rubrics for open-ended work.
• Examples:
  • AB testing with and without source material, multi-judge consensus.
• Terminology:
  • grounding
  • judges
  • scaling search
  • test-time compute
• Why it matters: Provides actionable methodologies to bootstrap supervision and rewards from raw data.

Synthetic Tasks and World Simulators

• Explanation: Building synthetic search tasks, code tasks via backwards reconstruction (holding out tests), and world simulators to enable verifiability in black-box environments.
• Key claims:
  • Working backwards from solutions (e.g. holding out tests, removing files) creates reachable training tasks.
  • World simulators provide high-fidelity synthetic environments where verifiability is baked in.
• Examples:
  • Synthetic code tasks by removing files and requiring the agent to reconstruct code to pass tests.
• Terminology:
  • synthetic tasks
  • world simulators
  • planted gold
  • reverse engineering
• Why it matters: Allows scaling training data and rewards for web apps and tools without backend control.

Key Points

Most real agent tasks lack clean verifiable rewards

• Explanation: Real-world agent tasks such as writing reports, handling refunds, or booking flights involve subjective or fuzzy evaluation signals rather than binary correct answers.
• Evidence: Comparison between math/code verification and open-ended agent tasks like 'is this a good report?'.
• Practical implication: Standard RL algorithms cannot be directly applied without engineering proxy reward signals.

Grounding in source material bootstraps supervision

• Explanation: Using existing documentation, code repos, and production traces as source material allows the creation of grounded tasks with known reference solutions.
• Evidence: Mining production traces of deployed agent interactions to form task distributions.
• Practical implication: Enables supervised and reinforcement learning signals to be mined automatically from real application usage.

Working backwards creates reachable learning tasks

• Explanation: Starting from a known solution or completed artifact and removing components (e.g., test cases or files) creates verifiable challenges for agents to solve.
• Evidence: Synthetic code tasks where agents must reconstruct code to pass held-out tests.
• Practical implication: Solves the data scarcity problem for complex agent capabilities by synthesizing training tasks from existing codebases.

World simulators bridge the gap for unprogrammable apps

• Explanation: For web applications or tools lacking full backend control, training synthetic simulators combined with test-time search allows verifiability to be baked in.
• Evidence: Iterating between a synthetic simulator and real production traces to create high-fidelity environments.
• Practical implication: Unlocks RL training for complex web and tool-use environments previously resistant to reinforcement learning.

Frameworks, Models & Processes

The Prime Intellect Stack

• How it works: A layered architecture powering end-to-end agent development from compute orchestration to model deployment.
• Components:
  • Models (open models tuned to a task)
  • Lab (hosted training, evals, inference)
  • Training (prime-r1 open ML trainer)
  • Compute (GPU cluster orchestration)
• When to use: When setting up scalable infrastructure for open-source reinforcement learning and agent training.

Signal Manufacturing Framework

• How it works: Methods to generate reward signals for unverified domains through grounding, judges, and search.
• Components:
  • Grounding (from source material, docs, traces)
  • Judges (LLMs scoring against rubrics and references)
  • Scaling Search (test-time compute to sharpen tasks and verification)
• When to use: When applying RL to messy, open-ended real-world tasks without ground-truth reward functions.

General Agent Synthesis Loop

• How it works: An online loop where a synthesizer designs tasks and verifiers, a solver attempts them with N rollouts, and a gate filters tasks based on target solve-rate to seed harder tiers.
• Components:
  • Synthesizer (designs task, gold, verifier)
  • Solver (attempts task, N rollouts)
  • Gate (keeps only tasks near target solve-rate)
• When to use: For self-evolving synthetic environment generation and continual learning in agent training.

Examples & Case Studies

Synthetic code tasks generated from real pull requests.

• Illustrates: Working backwards from solutions to create training tasks.
• Lesson: Hold out tests or remove files from real PRs to train models to reconstruct code solutions.

Using production agent traces as raw data.

• Illustrates: Grounding tasks in real-world agent behavior.
• Lesson: Production traces from deployed agents act as a natural distribution for mining future training tasks.

Actionable Takeaways

• Immediate:
  • Recognize that environments and evals share the same underlying structure.
  • Leverage existing documentation, code repos, and production traces for grounding.
• Strategic:
  • Move toward continual learning loops where deployed agents generate their own training tasks.
  • Invest in synthetic world simulators and calibrated judges to solve the reward verification bottleneck.
• Questions to investigate:
  • How can we effectively mitigate reward hacking in open-ended LLM judge setups?
  • What is the optimal threshold for task solve-rate gating in automated environment generation?

Claims Worth Verifying

• Environments and evals are essentially the same objects that can be used across SFT, RL, distillation, and prompting. (conceptual)
• Working backwards from a completed solution (like holding out tests in PRs) enables scalable synthetic task generation. (methodological)

Notable Quotes

"Many of people may have been learning about RLVR over the past year or so, year and a half, as this stuff has really taken off... but often we don't actually have verifiable rewards." (at 0:29) "The environment is the bottleneck. Good tasks and rewards are the hard part, not the trainer." (at 30:05)

Compressed Summary

• Reinforcement learning often lacks verifiable rewards in messy real-world tasks.
• Prime Intellect provides a full stack for open ML training, labs, and GPU orchestration.
• Signal can be manufactured via grounding, calibrated LLM judges, and test-time compute search.
• Synthetic task generation works backwards from solutions, such as holding out tests in code repos.
• Continual learning aims to create self-evolving agent loops running in production.
• Keywords: reinforcement learning, verifiable rewards, agent tasks, world simulators, continual learning
• Core insight: Scaling reinforcement learning to messy real-world agent tasks requires manufacturing reward signals through grounding, synthetic world simulators, and automated environment design.