Reality Check: a public leaderboard for robot manipulation VLA models
Original titleMissed this post the first time around, but i think this is a very cool and needed effort to thoroughly benchmark VLA and co.
AISummary
Lucas Beyer praises Reality Check, a new leaderboard for benchmarking VLA and related robot manipulation models. Half of its tasks are fully open, while the other half are held out to detect benchmaxxing by future model versions.
The companion post from Nicolas Keller describes the launch as the first public robot manipulation benchmark, built on 14,400 real-world rollouts across four models.
Source: Lucas Beyer · x.comPublished · added here