RoboARCade

For direct motor control, yes: today’s agents cannot respond fast enough for tasks that demand quick reactions. Distillation and quantization could narrow the gap, but the agent may not need to keep pace with low-level control at all. Robotics already relies on hierarchical control, pairing a slower “System 2” that plans with a faster “System 1” that produces reactive motion. This split only works if System 1 can handle the scene on its own without waiting for the planner. For example, a robot told to cross the street must also avoid traffic that was not visible to the planner. That leaves the agent to revise the plan and retry when something goes wrong, which is where agents have shown promise. RoboARCade removes this division on purpose: the agent chooses every action, so the benchmark measures its embodied reasoning and not the quality of a System 1 policy.

Many of RoboARCade’s assets and tasks are adapted from existing benchmarks, as detailed on the acknowledgements. We increase their difficulty by combining them into longer-horizon tasks and introducing greater pose variation. This reflects a difference in evaluation goals: RoboARCade tests embodied reasoning in agents, while the source benchmarks evaluate real-time robot policies. Accordingly, we allow substantially longer reasoning times, prioritizing capability over latency because we expect latency to improve. The tasks assess long-horizon reasoning, precise manipulation, memory, control across multiple embodiments, and coordination among multiple robots.

The agent runs in a sandbox without access to task assets, privileged object poses, success predicates, or grading results. It cannot modify benchmark or interface files. Although the agent can end an episode, scoring takes place outside the sandbox using the recorded simulator state. Network access is restricted to the services required by the interface and model provider.

RoboARCade is designed to be modular. Each task can specify its scene, rendering and textures, instructions, initial-state randomization, grading criteria, and dynamic scene motions, while using the shared simulation and agent interfaces.

Possibly. Some tasks and assets build on public robotics projects, so a model may have encountered the scenes, objects, or even solutions in its training data. We expect this to help little, for two reasons. First, we randomize initial states while keeping each task feasible, so a memorized solution does not transfer: the agent must plan for the scene in front of it and execute precisely. Second, familiarity with an asset mostly helps a model recognize objects and ground the instruction in the scene, which agents already do well. Adapting plans and executing precisely remain the bottlenecks, and those are what RoboARCade measures. The projects we build on are credited in the acknowledgements and the paper.