Tsinghua × Berkeley open-source uni-robot-agent: With a good Agent framework, open-source models can also manipulate robots zero-shot
The core finding of the paper is that as long as information is well organized, tools are reliable, and feedback is sufficient, open-source models can effectively operate robots without simulator ground truth and without being specially trained for robotics. After switching to this framework, the same Qwen3.8-Flash-Next's score on RoboDojo rose from 1.90 to 31.39. The authors also organized four general capabilities that robot manipulation depends on, as targets for subsequent training.
Paper: https://huggingface.co/papers/2611.01560
How the framework is designed
A single tool call passes through five layers: the Agent layer handles model interaction and context, the orchestration layer schedules requests, the tool layer provides shared operations such as motion, perception, skills, and task-completion judgment, the service layer manages runtime resources, and the backend layer connects to specific simulators, real robots, and models. Different robots and environments only need one adapter written, and the tools' parameters, units, and feedback formats remain consistent. In this way, interaction records from simulation and real robots can be stored together, making them convenient for training later.
Execution uses a pipeline: while a robotic arm is moving, the model can continue reasoning; calls operating the same arm are queued in submission order; and when the plan changes, flush_queue can be used to cancel actions that have not yet started. In addition, there are six designs that ensure reliability:
· Images, camera calibration, and robot state are bound in the same snapshot, avoiding computing new positions from outdated images.
· When it is uncertain whether an action has already been executed, duplicate dispatch is not allowed.
· Movement and gripper opening/closing can be completed simultaneously in one instruction.
· Motion decides when to stop based on measured progress, rather than running a fixed number of steps.
· Each action returns pose error, stop reason, and a result image, so the model can judge whether an object was truly grasped.
· Old images are replaced in batches with text placeholders, which saves context and allows cache reuse.
What "no privileged information" specifically means
In the main experiments, the model is only given RGB images, camera calibration, the robot's proprioceptive state and description, and the task instruction. It is not given depth, object CAD models, hidden scene annotations, feedback from the task judge, and it cannot access the internet. Without depth, the model must compute coordinates itself via multi-view triangulation. For some RoboDojo tasks, the authors added brief rule explanations, such as that the red button needs to be pressed several times and that the blue button is needed for confirmation, and stated that these explanations contain no scoring details.
Simulation results (RoboDojo, 42 tasks, 252 episodes per configuration)
· Qwen3.8-Flash-Next: 31.39 points, 25.00% success rate; 35.89 points with depth added.
· DeepSeek-V4.1-Flash: 27.11 points, 21.39% success rate.
Other frameworks using Qwen for planning: the best, GAP (hybrid mode, using π0.5), scored 21.01; RPent scored 7.75; RoboProbe only 1.90.
· GPT-6 Astra: with RoboProbe 28.97 points, with uni-robot-agent 51.90 points, and 56.76 points with depth added.
Open-source models perform better on memory and open-vocabulary instructions, and weaker on fine manipulation and long-horizon tasks, while VLA-type models are the opposite. The authors also separately ran 2,700 episodes to test score stability: when each task variant is run for only 6 episodes, scores fluctuate within a range of roughly 8 points.
Real-robot results (dual-arm robot, 19 tasks, full score 100)
Astra scored 80.50, with the compositional generalization item earning full 20 points. Qwen scored 53.50. Both only got 8/20 points on unseen tasks. For the tasks "take the bowl out of the microwave" and "unfold the rag," all four systems scored 0.
Which capabilities determine the effectiveness of robot manipulation
· Generations and scale: The success rates of all versions from Qwen3.5 to 3.7 were 0, while Qwen3.8-27B had 8.06%. That is, the new 27B model can complete tasks, while the earlier 122B model could not, and the authors infer from this that changes in training data matter more than parameter count. The GPT series can complete tasks starting from 5.6 Sol, while GLM-5.3 Flash's success rate remains 0.
· Four types of failure: failing to understand the image, failing to find accurate correspondences across multiple views (even GPT-6.1 Sol had 31.5% of triangulation calls fail validation), writing incorrect geometry code (such as reversed coordinate transform directions and mixed quaternion conventions), and grasping approaches that violate physical common sense.
· Correlation with general benchmarks: The correlation coefficient with the embodied reasoning benchmark ERQA is 0.97, with the multimodal understanding benchmark MMMU-Pro is 0.93, with the programming benchmark LiveCodeBench is 0.75, and with the screen grounding benchmark ScreenSpot-Pro is 0.69.
After removing depth and the skill library
When only depth is removed while camera calibration is kept, Astra still retains 98.6% of its score, and Qwen retains 87.5%. When calibration is also removed, Astra drops to 31.29 points and Qwen drops to 9.13 points. After removing the preset skill library, Qwen drops from 31.39 to 30.31 points, indicating that the results mainly do not depend on hand-written skill workflows. Pipelined execution shortened the time used by 4.8% to 73.7% across 5 cases selected by the authors, and the authors also note that these reductions are not entirely due to scheduling.