I guess 'tis the day i complain about over-hype on this app.
This does not show a quadrillion param model, nor how to train one.
The claim (which is only in the blog and this thread, the paper is ok) is based on wild extrapolations, but i promise you that the techniques studied in the paper will not continue carrying more than ~one OOM.
In fact you can are already see here (figure) that reality starts detaching from prediction. It will continue detaching more.
(The study is on 25M-400M unique tokens, 30M-400M params)
Also, just because i can't resist sprinkling a little bit of Schmidhubering, how to do many epochs is something vision folks studied a long time, and the "increase wd with epochs" is a finding were published in S4L paper in 2019. It's of course worth revisiting in LLMs but i generally find things carry over.