This got totally buried cause it came out the same day as Navier Stokes.
But I think this is super interesting evidence about what the true major source of algorithmic progress is.
Dwarkesh Patel says data gains outpaced model gains in pretraining progress
AIDwarkesh Patel and Jerbear pretrained combinations of year-representative open model recipes and data corpora from 2019 to 2025 at small scales. Data improvements contributed 12.0x compute multipliers versus 3.7x for model improvements, or 3.24x as much.
Dwarkesh PatelVerified on X
@dwarkesh_sp
Pretraining progress seems to be coming mostly from data improvements. @who_is_jerbear and I pretrained combinations of year-representative open model recipes and data corpuses across 2019 to 2025 at various small scales. Data improvements contributed 3.24x as many compute multipliers as model improvements did (12.0x vs 3.7x). And the gains stack independently - a better dataset helps every architecture about equally, and vice versa. Here are full results, plus what we think this means for the future of AI progress: https://www.dwarkesh.com/p/pretraining-progress-is-mostly-data
Source details
Dwarkesh Patel · x.comPublished · added here
Original title: This got totally buried cause it came out the same day as Navier Stokes.