Robotics and AI meet where a machine must turn camera images and a spoken instruction into motor commands. One documented approach is the vision-language-action model: in the RT-2 paper, submitted to arXiv on July 28, 2023, researchers trained a model on web images and text together with robot demonstrations, and it writes robot actions as strings of numbers in the same way it writes words. In the paper's tests, the best versions succeeded on an average of 62 percent of unseen scenarios, against 32 percent for the earlier RT-1 model.
Key takeaways
- A robot needs sensing, a decision, and movement. AI usually supplies the decision, and in newer research it also interprets instructions and images.
- RT-2 converts each robot action into eight integers drawn from 256 bins, so a language model can output actions as text.
- On unseen objects, backgrounds, and environments, RT-2 averaged 62 percent success in the paper, versus 32 percent for RT-1 and 35 percent for the MOO baseline.
- The authors state that web pretraining did not give the robot new physical motions. Its skills stayed within the range of the robot data.
- The largest RT-2 model ran at about one to three decisions per second on remote cloud hardware, which limits fast or delicate tasks.
What a robot needs besides intelligence
A robot is a machine with sensors, a controller, and actuators such as motors. The controller reads the sensors, picks the next movement, and sends commands to the actuators many times per second. Artificial intelligence enters at the middle step. A factory arm can run a fixed program with no AI at all. A robot that must pick up an object it has never seen needs something that recognizes the object, relates it to an instruction, and chooses a grasp.
For decades these jobs were split among separate programs: one for vision, one for language, one for planning, and one for control. Errors in each stage passed to the next. The research described below tries to merge several of those stages into one trained model.
How RT-2 turns words into movement
A vision-language model is a neural network trained on images paired with text, so it can answer questions about a picture. The RT-2 paper by Anthony Brohan and 53 co-authors fine-tunes such models on robot trajectories as well as on web tasks like visual question answering. The result is a single model that takes camera images and an instruction and returns an action.
The key design choice is how the action is written. Each action covers the change in the gripper's position and rotation (six degrees of freedom), the gripper opening, and a flag that ends the episode. Every continuous value is divided into 256 equal bins, so one action becomes eight integers. The model prints those integers as ordinary tokens, and a decoder converts them back into movement. Because actions look like text, the same training procedure covers both. Two versions were built, one on a 55 billion parameter model (PaLI-X) and one on a 12 billion parameter model (PaLM-E).
What the tests measured
The evaluation included about 6,000 trials. Robots were asked to handle objects, backgrounds, and environments that did not appear in the robot training data. The table lists the averages reported in the paper for tasks seen in training and for the unseen cases.
| Model | Seen tasks (percent success) | Unseen average (percent success) |
|---|---|---|
| RT-1 | 92 | 32 |
| MOO | 75 | 35 |
| RT-2 on PaLI-X, 55 billion parameters | 91 | 62 |
| RT-2 on PaLM-E, 12 billion parameters | 93 | 62 |
The pattern matters more than any single cell. On familiar tasks, RT-1 and RT-2 are roughly equal. The gain appears only on unfamiliar objects, backgrounds, and settings, which suggests that knowledge learned from web images helps the robot recognize what it is being asked about. The authors also report qualitative examples, such as following an instruction to move an object to a particular number or icon, and picking the smallest or largest item. They describe the chain-of-thought results, in which the model first writes out a plan, as initial evidence rather than a settled finding.
Limits the authors state
The paper names two. First, RT-2 does not learn new physical motions from the web. Its movements remain inside the distribution of the robot demonstrations, though it applies them in new ways. The authors link this to the limited variety of skills in the data. Second, running such a large model is costly. The 55 billion parameter version operated at roughly one to three hertz, and the 5 billion parameter version at about five hertz. Both were served from a multi-TPU cloud service that robots queried over a network, since the models are too large for typical onboard hardware. Quantization and distillation are suggested as future work.
These are results from one research lab, using its own robots and a defined set of tabletop tasks. They show a method that works under those conditions. They do not show that a household or warehouse robot can follow any spoken request.
Questions to ask about a claim that a robot uses AI
- Which part is learned: object recognition, language understanding, planning, or the movement itself?
- Were the test objects and rooms different from the training data, and how many trials were run?
- Where does the model run, on the robot or on a remote server, and how many decisions per second does it make?
- Does a person teleoperate or reset the robot during the demonstration?
- Is the reported success rate an average over many trials, or a single recorded run?
FAQ
Is AI the same thing as a robot?
No. AI is software that learns or decides, and a robot is a physical machine. Many robots run fixed programs, and much AI, such as a chatbot, has no body.
Does RT-2 mean a robot can do anything it is told?
No. The authors report that it reuses motions already present in its robot training data. It generalizes to new objects and instructions, but not to physical skills it never saw.
Why write actions as text?
Text tokens let one model and one training method handle both language and control. The robot data are simply added to the web data the model already learns from.
Is 62 percent good enough for real use?
That depends on the job. A 62 percent average on unseen cases in a lab means roughly four failures in every ten attempts in new situations, which is too many for most safety-related work.
Related reading
- AI robot technologies over the past decade
- How AI and robotics affect society
- AI and robots across industries
- Another AI crossover: creativity and art
More in AI Basics.
Last verified: October 2026
Sources
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control, Brohan et al., arXiv:2307.15818, 2023
- RT-2 project page, Google DeepMind robotics team, 2023
- RT-1: Robotics Transformer for Real-World Control at Scale, Brohan et al., arXiv:2212.06817, 2022