Skip to content
Trending storyDeveloping

Dwarkesh Patel says data gains outpaced model gains in pretraining progress

1 article1 sourcesince Nov 10Last article

Overview

AISummary of 1 article

Dwarkesh Patel and Jerbear pretrained open model recipes and data corpora representative of each year from 2019 to 2025, at small scales.

Patel says data improvements gave a 12.0x compute multiplier, against 3.7x for model improvements, or 3.24 times as much. He says the gains stack independently, with a better dataset helping every architecture about equally. The full results are posted on dwarkesh.com.

Written by AI from the articles below · updated Nov 10, 10:42 PM ET

Check the sources:

Latest developmentNov 10, 10:33 PM ET
Dwarkesh Patel says data gains outpaced model gains in pretraining progress

Article timeline

The articles in this story. Times are ET.

Nov 10
  1. Dwarkesh PatelX
    Dwarkesh Patel says data gains outpaced model gains in pretraining progress

    AIDwarkesh Patel and Jerbear pretrained combinations of year-representative open model recipes and data corpora from 2019 to 2025 at small scales. Data improvements contributed 12.0x compute multipliers versus 3.7x for model improvements, or 3.24x as much. The author says the gains stack independently, with a better dataset helping every architecture about equally, and links the full results on dwarkesh.com.

Heat Trend

Current comparable heat
8
Peak comparable heat
10Nov 10, 11:00 PM ET
Change in past 24 hours
–

The trend compares only continuously observed participants, so its scope may be smaller than the current heat total. Hover or tap for hourly heat; use arrow keys with a keyboard.