Lucas Beyer disputes quadrillion-parameter scaling claim as overreaching

Lucas Beyer says the quadrillion-parameter scaling claim in a linked blog post and thread rests on extrapolations that go beyond the underlying paper.

He says the paper is sound, but its experiments cover only 25M to 400M unique tokens and 30M to 400M parameters. Beyer also says the paper's own figure already shows reality diverging from the predicted curve. He adds that multi-epoch training and increasing weight decay with epochs were studied earlier in vision work, including the 2019 S4L paper.