WorldOracle benchmark opens to developers after finding world models mispredict object motion
Overview
Princeton, UT Austin, University of Toronto, NVIDIA and AgiBot introduced WorldOracle, a benchmark that scores action-conditioned world models against real robot executions, and opened the platform to developers on October 16.
In tests of seven models, including DreamDojo, Cosmos 3, IRASim and GE-Sim 2.0, the teams report that arm motion was predicted relatively well while object positions and movements often deviated from reality. Scores also dropped over complete operation sequences: DreamDojo fell from 50.56 to 27.08 and Cosmos 3 from 53.99 to 26.60.
The platform accepts both open-weight and closed-source models. After registration and approval, developers select a track matching the embodiment or manipulation setting their model supports, upload predicted videos following the submission guide, and receive composite scores, sub-scores and comparisons.
Written by AI from the articles below · updated Oct 28, 10:59 AM ET
Check the sources:
Article timeline
The articles in this story. Times are ET.
- The Robot ReportNewsWorldOracle benchmark finds world models often mispredict object motion in robot tasks
AIPrinceton, UT Austin, University of Toronto, NVIDIA and AgiBot introduce WorldOracle, a benchmark that scores action-conditioned world models against real robot executions. Across seven models, including DreamDojo, Cosmos 3, IRASim and GE-Sim 2.0, arm motion was predicted relatively well while object positions and movements often deviated from reality. Scores also fell over complete operation sequences, with DreamDojo dropping from 50.56 to 27.08 and Cosmos 3 from 53.99 to 26.60, and the platform opened to developers on October 16.
Heat Trend
Not enough continuous observations to show a trend yet.