# Lucas Beyer says the quadrillion-parameter scaling claim overreaches

> Original title: I guess 'tis the day i complain about over-hype on this app.

- Source: Lucas Beyer
- Published: 2026-11-03T07:50:05.000Z
- Bad Influence: https://badinfluence.news/items/g1ulsud06o8ailoxoc14u17y2
- Original: https://x.com/giffmana/status/2117524701354361269

## Summary

Lucas Beyer argues that the quadrillion-parameter claim in the linked blog and thread is based on wild extrapolations, while the paper itself is sound. He says the study covers only 25M-400M unique tokens and 30M-400M parameters, and that its figure already shows reality diverging from the prediction. He also notes that multi-epoch training and increasing weight decay with epochs were studied in vision work, including the 2019 S4L paper.

## Post

I guess 'tis the day i complain about over-hype on this app.

This does not show a quadrillion param model, nor how to train one.

The claim (which is only in the blog and this thread, the paper is ok) is based on wild extrapolations, but i promise you that the techniques studied in the paper will not continue carrying more than ~one OOM.

In fact you can are already see here (figure) that reality starts detaching from prediction. It will continue detaching more.

(The study is on 25M-400M unique tokens, 30M-400M params)

Also, just because i can't resist sprinkling a little bit of Schmidhubering, how to do many epochs is something vision folks studied a long time, and the "increase wd with epochs" is a finding were published in S4L paper in 2019. It's of course worth revisiting in LLMs but i generally find things carry over.

![](https://badinfluence.news/api/img-proxy?u=https%3A%2F%2Fpbs.twimg.com%2Fmedia%2FHWL1fanWkAAHTxQ.jpg&mode=thumb&exp=1793923200&sig=a1d0130e401622b4)

## Quoted post @industriaalist

> Humans are ~1,000,000x more sample efficient than LLMs, and there are no good explanations.
> 
> We’ve found that a big part of the answer is that LLMs are *massively* underparameterized by many orders of magnitude. The brain has 100T synapses learning from ~200M tokens of language. Chinchilla prescribes a 1T-parameter model on 20T tokens. That’s 7 OOMs apart in parameters per token.
> 
> After a lot of painstaking work, we present new data-constrained scaling laws and learning algorithms to scale to (extrapolated) quadrillion-parameter transformers.
> 
> w/ @akshayvegesna, @EthanBaron78553, @asatheesh6561
> Full paper led by @EthanBaron78553, with @ShmuelBerman, @andrewgwils

https://x.com/industriaalist/status/2117370524330594698