Lucas Beyer analyzes Inkling 2 Small's architecture and training budget

Lucas Beyer posts a summary of a long analysis of Inkling 2 Small, which the analysis says has 594B total parameters, 17B active, and a 1M-token context window.

The analysis reports 12.2T pretraining tokens and 3T RL tokens, and notes that Inkling 2 Small replaces the encoder-free design of Inkling 1 with a 292M ViT encoder. Beyer says he finds the ViT change sensible. The analysis flags many architecture and optimizer details as undisclosed.