if i had to give my gut take (not deeply thought through), it's that:
- i don't think there's any flops justification for this. i assume the specialized robotics models spent more or equal flops on robotics-like/multimodal data as Astra.
- i find it unlikely that Astra's multimodal encoders are somehow better than the robotics team. plus same point as above
- chain-of-thought helps robotics tasks a lot, even at very low reasoning budget. bigger pretrain -> better cot -> better task performance
- transfer increases at scale for optimization/parameter reasons, which is hard for us to probe cheaply (you can't run ablations at gpt-4 scale)
- transfer increases as scale for data scale reasons. to gpt-1, the mutual information it can extract between any two datapoints appears to be very low, at least beyond token frequencies. as you scale models, things that seemed unrelated (documents about Kissinger in two different languages) now are connected. perhaps at sufficient scale everything looks like it's from the same distribution. but then again this is too abrupt
- some unknown algorithmic change in astra made it much better than specialized robotics models
- some amount of all the above, plus they fixed some bugs and added the right datasets, and we would’ve seen this coming since gpt-4 had we done that back then
- specific transfer from CAD and computer use training helped a lot. i think this is unlikely
- im wrong about the compute difference, astra got more flops. or i’m wrong about encoders, astra had better encoders with more compute dumped into them.