Robot-Use Agents: Why General-Purpose Models May Win in Robotics

Overview

• Speaker: Ankit Gupta
• Channel: Y Combinator
• Main topic: Robot-use agents and the convergence of general-purpose language models with robotics control
• Purpose: To analyze the transition of general-purpose LLMs and vision-language models into robotic control agents and discuss the implications for robotics startups. In this YC Decoded episode, Ankit Gupta, Francois Chaubard, and founders from Waddle Labs (Hanming Ye, Vincent Song) and Robocurve (Jay Chooi) discuss the rise of robot-use agents. They examine how large language and vision-language models (like RT-2 and Astra) generalize to robot control through pre-training on web data, code-as-policy methods, data transfer, and the platonic representation hypothesis.

Topic Map

The Rise of Robot-Use Agents

• Explanation: Introduction of LLMs acting as puppeteers of robot bodies, enabling general-purpose models to control robots across diverse domains.
• Key claims:
  • LLMs are increasingly tested as robot-use agents.
  • General-purpose models make different robots more capable.
  • Recent viral papers and projects show robots writing code and executing tasks in minutes.
• Examples:
  • Waddle Labs using Claude Code for robots.
  • GPT-4 Astra scoring 95% on robot control tasks.
• Terminology:
  • robot-use agents
  • spatial intelligence
  • computational agents
• Why it matters: It signals a shift from specialized robotics software to general-purpose AI models driving robotic hardware.

Language Models to Robotic Control: RT-2 and Code as Policies

• Explanation: Discussion on early transfer techniques between language models and robot control, covering RT-2 and Code as Policies.
• Key claims:
  • RT-2 fine-tunes vision-language models on robotic trajectory data and web data.
  • Code as Policies allows LLMs to write Python code combining perception and control primitives.
  • Models benefit from massive pre-training on internet text and images.
• Examples:
  • Google's RT-2 paper mapping web knowledge to robotic control.
  • Code as Policies using Python function libraries for robot execution.
• Terminology:
  • RT-2
  • VLA
  • Code as Policies
  • end-to-end control
• Why it matters: Pre-training on non-robotics web data significantly boosts robotic task performance and generalization.

Platonic Representation Hypothesis & Future of Robotics

• Explanation: Examining whether strong language models naturally share strong representations with physical world models.
• Key claims:
  • Strong language models converge on consistent mappings of the world as they train on larger datasets.
  • Vision capabilities in models like Astra leverage massive pre-training on computer-use and CAD data.
  • General-purpose robots capable of handling unseen tasks are approaching rapidly.
• Examples:
  • Astra controlling robot arms using camera inputs and tool calls.
  • DreamCoder and Bayesian program learning comparisons.
• Terminology:
  • platonic representation hypothesis
  • transduction vs program induction
  • spatial intelligence
• Why it matters: It suggests that advancing general intelligence models will automatically solve robotic control without needing purely dedicated robot datasets.

Key Points

Pre-training transfer from web data

• Explanation: Models trained on internet text, images, and code develop spatial and logical reasoning that transfers effectively to robotic control.
• Evidence: RT-2 and Astra benchmarks showing superior performance over models trained solely on robot data.
• Practical implication: Robotics companies should leverage foundation models rather than building robotics-only models from scratch.

Tool-use versus end-to-end control

• Explanation: Distinction between using LLMs to write code/make tool calls versus direct end-to-end action outputs.
• Evidence: Code as Policies writing Python functions versus VLA end-to-end pose outputs.
• Practical implication: Choosing between code generation and direct control depends on task latency and repeatability.

Frameworks, Models & Processes

Code as Policies

• How it works: LLMs are given perception API functions and write executable Python code to control robots and perform complex tasks.
• Components:
  • Perception APIs
  • Control primitives
  • LLM code generator
• When to use: When tasks require repetitive execution, logic structuring, and low-latency safety bounds.

Examples & Case Studies

Astra controlling robot arms to pick up an apple block and place it in a bowl.

• Illustrates: Vision-language models utilizing camera feeds and tool calls to manipulate physical objects.
• Lesson: Pre-trained vision capabilities significantly enhance robotic spatial manipulation.

Actionable Takeaways

• Immediate:
  • Apply to YC at ycombinator.com/apply.
  • Explore integrating vision-language models into robotic control pipelines.
• Strategic:
  • General-purpose intelligence models will redefine robotics capabilities.
  • Combining web data pre-training with robotic task fine-tuning yields superior generalization.
• Questions to investigate:
  • How do we solve the latency bottleneck for real-time robotic control?
  • What is the optimal boundary between weight-space learning and symbolic code execution?

Claims Worth Verifying

• GPT-6 Astra scored 95% on robot control tasks, outperforming Fable 5.1. (benchmark claim)
• Model latency in VLA approaches improves by 2x per month. (technical trend claim)

Notable Quotes

"This year we are going to see many LLMs being tested as robot-use agents." (at 0:13) "If you give the AI model more autonomy and you if you unshackle it a bit more... it can actually do a lot of things that we fine-tune it to do." (at 6:55)

Compressed Summary

• LLMs are transitioning from software assistants to robot-use controllers.
• Pre-training on internet data, images, and code improves robotic spatial intelligence.
• Code-as-policy and vision-language-action models are bridging the digital-physical divide.
• Keywords: robotics, llms, agents, rt2, vision-language
• Core insight: General-purpose language and vision models pre-trained on internet data are rapidly outperforming specialized robotics models by leveraging transferred spatial and logical reasoning.