# Tsinghua and Berkeley open-source uni-robot-agent for zero-shot robot control

> Original title: 清华 × 伯克利开源 uni-robot-agent：开源模型借助一套好用的 Agent 框架，也能零样本操作机器人

- Source: meng shao
- Published: 2026-11-05T01:59:47.000Z
- Bad Influence: https://badinfluence.news/items/w2uf6ki7gtlhe89eq7ijhcava
- Original: https://x.com/shao__meng/status/2118161321720156347

## Summary

Tsinghua and UC Berkeley researchers released uni-robot-agent, a unified framework for robot-use agents. With it, the open model Qwen3.8-Flash-Next raised its RoboDojo score from 1.90 to 31.39 without simulator ground truth or robot-specific training. The paper reports results on simulated and real-world tasks, with GPT-6 Astra scoring 80.50 on the real-world tasks against Qwen's 53.50.

## Why it matters

The paper shows open models can operate robots zero-shot when given a well-organized agent framework, with benchmark scores and ablations to check the claim.

## Post

清华 × 伯克利开源 uni-robot-agent：开源模型借助一套好用的 Agent 框架，也能零样本操作机器人

论文的核心发现是，只要信息组织得好、工具可靠、反馈充分，开源模型不用仿真器真值、也不专门为机器人训练，就能有效地操作机器人。换到这套框架后，同一个 Qwen3.8-Flash-Next 在 RoboDojo 上的分数从 1.90 升到 31.39。作者还整理出机器人操作所依赖的四项通用能力，作为后续训练的目标。

论文：https://huggingface.co/papers/2611.01560

框架怎么设计

一次工具调用会经过五层：Agent 层负责模型交互和上下文，编排层负责调度请求，工具层提供运动、感知、技能和任务结束判定等共用操作，服务层管理运行时资源，后端层对接具体的仿真器、真机和模型。不同的机器人和环境只需要写一个适配器，工具的参数、单位和反馈格式保持一致。这样，仿真和真机产生的交互记录能放在一起，方便以后用来训练。

执行上采用流水线：机械臂移动时模型可以继续推理，操作同一条手臂的调用按提交顺序排队，计划改变时可以用 flush_queue 取消还没开始执行的动作。另外有六个保证可靠性的设计：
· 图像、相机标定和机器人状态绑定在同一个快照里，避免用旧画面算出新位置。
· 不确定某个动作是否已经执行时，不允许重复下发。
· 移动和夹爪开合可以在一条指令里同时完成。
· 运动按实测进度决定何时停止，而不是走固定步数。
· 每个动作返回位姿误差、停止原因和结果图像，让模型判断是否真的抓到了。
· 旧图像分批替换成文字占位，既节省上下文，也能复用缓存。

「不用特权信息」具体指什么

主要实验只给模型 RGB 图像、相机标定、机器人本体状态和描述，以及任务指令。不给深度、物体 CAD 模型、隐藏的场景标注、任务判定器的反馈，也不能上网。没有深度时，模型要靠多视角三角测量自己算出坐标。作者给部分 RoboDojo 任务补充了简短的规则说明，比如红色按钮要按几次需要用蓝色按钮确认，并且声明这些说明不包含评分细则。

仿真结果（RoboDojo，42 个任务，每种配置 252 局）

· Qwen3.8-Flash-Next：31.39 分，成功率 25.00%；加上深度后为 35.89 分。
· DeepSeek-V4.1-Flash：27.11 分，成功率 21.39%。
用 Qwen 做规划的其他框架：最好的 GAP（混合模式，用到了 π0.5）是 21.01 分；RPent 是 7.75 分；RoboProbe 只有 1.90 分。
· GPT-6 Astra：用 RoboProbe 是 28.97 分，用 uni-robot-agent 是 51.90 分，再加深度是 56.76 分。

开源模型在记忆和开放词汇指令上表现较好，在精细操作和长流程任务上较弱，VLA 类模型则正好相反。作者还另外跑了 2700 局检验分数的稳定性：每个任务变体只跑 6 局时，分数会在大约 8 分的范围内波动。

真机结果（双臂机器人，19 项任务，满分 100）

Astra 拿到 80.50 分，其中组合泛化一项拿满 20 分。Qwen 是 53.50 分。两者在没见过的任务上都只有 8/20 分。「从微波炉里取碗」和「展开抹布」这两项任务，四个系统都是 0 分。

哪些能力决定了机器人操作的效果

· 不同代际和规模：Qwen3.5 到 3.7 的各个版本成功率都是 0，Qwen3.8-27B 是 8.06%。也就是说，27B 的新模型能完成任务，而早期 122B 的模型做不到，作者据此推测训练数据的变化比参数量更关键。GPT 系列从 5.6 Sol 起就能完成任务，GLM-5.3 Flash 的成功率仍然是 0。
· 四类失败：看不懂画面、多视角找不准对应点（GPT-6.1 Sol 也有 31.5% 的三角测量调用没通过校验）、几何代码写错（比如坐标变换方向反了、四元数约定混用）、抓取方式不符合物理常识。
· 和通用评测的相关性：与具身推理评测 ERQA 的相关系数是 0.97，与多模态理解评测 MMMU-Pro 是 0.93，与编程评测 LiveCodeBench 是 0.75，与屏幕定位评测 ScreenSpot-Pro 是 0.69。

去掉深度和技能库之后

只去掉深度、保留相机标定时，Astra 还能保住 98.6% 的分数，Qwen 能保住 87.5%。标定也一起去掉后，Astra 降到 31.29 分，Qwen 降到 9.13 分。去掉预设的技能库后，Qwen 从 31.39 分降到 30.31 分，说明成绩主要不是靠手写的技能流程。流水线执行在作者挑选的 5 个案例里缩短了 4.8% 到 73.7% 的用时，作者也说明这些缩短不全是调度带来的。

## AI translation

Tsinghua × Berkeley open-source uni-robot-agent: With a good Agent framework, open-source models can also manipulate robots zero-shot

The core finding of the paper is that as long as information is well organized, tools are reliable, and feedback is sufficient, open-source models can effectively operate robots without simulator ground truth and without being specially trained for robotics. After switching to this framework, the same Qwen3.8-Flash-Next's score on RoboDojo rose from 1.90 to 31.39. The authors also organized four general capabilities that robot manipulation depends on, as targets for subsequent training.

Paper: https://huggingface.co/papers/2611.01560

How the framework is designed

A single tool call passes through five layers: the Agent layer handles model interaction and context, the orchestration layer schedules requests, the tool layer provides shared operations such as motion, perception, skills, and task-completion judgment, the service layer manages runtime resources, and the backend layer connects to specific simulators, real robots, and models. Different robots and environments only need one adapter written, and the tools' parameters, units, and feedback formats remain consistent. In this way, interaction records from simulation and real robots can be stored together, making them convenient for training later.

Execution uses a pipeline: while a robotic arm is moving, the model can continue reasoning; calls operating the same arm are queued in submission order; and when the plan changes, flush_queue can be used to cancel actions that have not yet started. In addition, there are six designs that ensure reliability:
· Images, camera calibration, and robot state are bound in the same snapshot, avoiding computing new positions from outdated images.
· When it is uncertain whether an action has already been executed, duplicate dispatch is not allowed.
· Movement and gripper opening/closing can be completed simultaneously in one instruction.
· Motion decides when to stop based on measured progress, rather than running a fixed number of steps.
· Each action returns pose error, stop reason, and a result image, so the model can judge whether an object was truly grasped.
· Old images are replaced in batches with text placeholders, which saves context and allows cache reuse.

What "no privileged information" specifically means

In the main experiments, the model is only given RGB images, camera calibration, the robot's proprioceptive state and description, and the task instruction. It is not given depth, object CAD models, hidden scene annotations, feedback from the task judge, and it cannot access the internet. Without depth, the model must compute coordinates itself via multi-view triangulation. For some RoboDojo tasks, the authors added brief rule explanations, such as that the red button needs to be pressed several times and that the blue button is needed for confirmation, and stated that these explanations contain no scoring details.

Simulation results (RoboDojo, 42 tasks, 252 episodes per configuration)

· Qwen3.8-Flash-Next: 31.39 points, 25.00% success rate; 35.89 points with depth added.
· DeepSeek-V4.1-Flash: 27.11 points, 21.39% success rate.
Other frameworks using Qwen for planning: the best, GAP (hybrid mode, using π0.5), scored 21.01; RPent scored 7.75; RoboProbe only 1.90.
· GPT-6 Astra: with RoboProbe 28.97 points, with uni-robot-agent 51.90 points, and 56.76 points with depth added.

Open-source models perform better on memory and open-vocabulary instructions, and weaker on fine manipulation and long-horizon tasks, while VLA-type models are the opposite. The authors also separately ran 2,700 episodes to test score stability: when each task variant is run for only 6 episodes, scores fluctuate within a range of roughly 8 points.

Real-robot results (dual-arm robot, 19 tasks, full score 100)

Astra scored 80.50, with the compositional generalization item earning full 20 points. Qwen scored 53.50. Both only got 8/20 points on unseen tasks. For the tasks "take the bowl out of the microwave" and "unfold the rag," all four systems scored 0.

Which capabilities determine the effectiveness of robot manipulation

· Generations and scale: The success rates of all versions from Qwen3.5 to 3.7 were 0, while Qwen3.8-27B had 8.06%. That is, the new 27B model can complete tasks, while the earlier 122B model could not, and the authors infer from this that changes in training data matter more than parameter count. The GPT series can complete tasks starting from 5.6 Sol, while GLM-5.3 Flash's success rate remains 0.
· Four types of failure: failing to understand the image, failing to find accurate correspondences across multiple views (even GPT-6.1 Sol had 31.5% of triangulation calls fail validation), writing incorrect geometry code (such as reversed coordinate transform directions and mixed quaternion conventions), and grasping approaches that violate physical common sense.
· Correlation with general benchmarks: The correlation coefficient with the embodied reasoning benchmark ERQA is 0.97, with the multimodal understanding benchmark MMMU-Pro is 0.93, with the programming benchmark LiveCodeBench is 0.75, and with the screen grounding benchmark ScreenSpot-Pro is 0.69.

After removing depth and the skill library

When only depth is removed while camera calibration is kept, Astra still retains 98.6% of its score, and Qwen retains 87.5%. When calibration is also removed, Astra drops to 31.29 points and Qwen drops to 9.13 points. After removing the preset skill library, Qwen drops from 31.39 to 30.31 points, indicating that the results mainly do not depend on hand-written skill workflows. Pipelined execution shortened the time used by 4.8% to 73.7% across 5 cases selected by the authors, and the authors also note that these reductions are not entirely due to scheduling.

![](https://badinfluence.news/api/img-proxy?u=https%3A%2F%2Fpbs.twimg.com%2Fmedia%2FHWU4fYiaEAAuQ1r.jpg&mode=thumb&exp=1794096000&sig=d216d518778b9e8a)

## Quoted post @zuo_yuxin

> ‼️ Can Qwen directly control the robot as well as GPT-6 Astra?
> 
> 🚀 Introducing uni-robot-agent: a unified harness for Robot-Use Agents (RUAs), a first step toward RUA training and local deployment.
> 
> We tested uni-robot-agent with Qwen3.8-Flash-Next running locally:
> 1️⃣ On RoboDojo, its score rose from 1.90 to 31.39, surpassing the RoboDojo Team’s score with GPT-6 Astra.
> 2️⃣ On real-world tasks, it outscored fine-tuned π0.5 by 5.75 points and zero-shot Dream-1 by 10 points.
> 🧵 [1/7]

https://x.com/zuo_yuxin/status/2117635294115795134