Lucas Beyer breaks down Inkling 2 Small's architecture and training budget

Interesting Notes on Inkling 2 Small release: Note: every single word is written by me. I used AI to fact check several things I doubted about, but that's it. (1) Architecture (A) 48-layer decoder-only transformer. 594B total. (B) 17B active. Sparse MoE feed-forward backbone. Each token routed to 8 of 384 experts. (C) Attention is a hybrid of local and global layers (no window size or ratio given). (D) Context window up to 1M tokens. (E) ViT is back. The rationale seems to be training efficiency. But it's just my opinion. (2) Training budget This is where things get interesting. In short, they used 8.4 + 3.8 = 12.2T for pretraining and 3T for RL. The SFT phase is a small bootstrap on synthetic data from open models like Kimi K2.5. (A) They have 4 main pretraining stages: • H1: Trained for 8.4T tokens on things like natural language, code, and image data; 68 hours on GB300 NVL72 racks, natively in NVFP4. • H2: Trained for 3.8T on higher-quality, newer data at 512K context. • H3: Trained for 34B on the highest-quality agent trajectories. You can probably call this stage mid-training. • RL was trained for 3T tokens using 1,075 steps, 17M+ rollouts. But let's also look at other metrics. (B) FLOPs: • pretraining ≈ 6 × 17B × 12.55T ≈ 1.3×10²⁴ • RL > 4×10²³ • RL ≈ 30% of pretraining compute, ~24% of the total. (C) Wall-clock: • The RL run (9 days) ran ~3× longer than the main pretraining phase (68 h). GPU count for H1 was not mentioned, so don't compare GPU-hours. • RL alone = 2,048 GPUs × 9 days ≈ 442K GPU-hours. (3) RL recipe Inkling 1 was built from two separate RL runs (reasoning, agentic) using MOPD. Inkling 2 Small just uses one joint RL run over all domains, so there is nothing to merge. The only distillation they do is OPSD for alignment. The model prompted with a conduct spec teaches the same model without it. (A) Joint RL run details: • Every optimizer step took 12 minutes consuming 2.8B tokens of rollouts. The median staleness was 17 updates (or roughly 3.4 hours of policy lag). • The mean rollout has 170K tokens. A p99 rollout of 1.5M tokens inside a 256K window means context was compacted ~5+ times in a single episode. • Per-family mean rollout length (token share ÷ rollout share × 170K): coding ≈ 290K, web/app ≈ 260K, office ≈ 250K, STEM ≈ 185K, search ≈ 120K, vision ≈ 37K, tool-use ≈ 23K, chat/safety ≈ 4K. • Important to note is that coding episodes on average exceed the 256K window, so compaction was the default choice here. • Also interesting to note is that RL used almost half the FLOPs of the 8.4T H1 pretraining phase at roughly 3× the wall-clock. • Effort levels were randomly sampled during RL, and the model was conditioned on them. • They also used a 256K-token context window with compaction for longer trajectories, and compaction training for long-horizon tasks. But we don't know how their compaction mechanism actually works. • For the actual RL algorithm, they use score centering. Basically, they are subtracting a baseline from rewards (REINFORCE with a centered score), a GRPO-family estimator without PPO clipping. We don't know anything about their learning rate, group size, etc. (4) Tiny interesting details (A) We don't know anything about the optimizer in any of the stages. It would be interesting to know whether they used fancier variants like Muon. (B) I could not verify most things about their architecture. These include GQA + SWA, window size, local:global ratio, shared experts, latent MoE, MLA, positional scheme, tokenizer. (C) Sparsity increased from 23× (Inkling 1 Small 276B/12B) to 35× (Inkling 2 Small 594B/17B). (D) The MoE config changed to Kimi K2's. Inkling 1 had 256 routed experts and 2 shared experts, with 6 routed experts active per token, following DeepSeek-V3. Inkling 2 Small's 8-of-384 is Kimi K2's routed config. (E) Inkling 1 was encoder-free. Images are encoded as patches of 40x40 pixels using a four-layer hMLP. 20×20 × 2×2 pooling = 40×40 px per token. But Inkling 2 uses a 292M ViT encoder with 20×20-pixel patches with 2×2 spatial pooling, interleaved with text tokens. So they are using the same token budget for vision encoding but a different embedder. (F) Audio was dropped in Inkling 2 Small. It supports image input but not audio. Inkling 1 and Inkling-Small took native audio. (G) Training data types are text, images, and video but video is not an input modality in this model. (H) Inkling 1 was pretrained on 45 trillion tokens. Then Inkling 2 Small used 27% of the tokens. But the AA Index increased from 12 to 31 !! (I) They reported an incident midway through the main RL run. The tldr is the model, failing a task, got around a network block via a third-party service it reached through its search tools. An async monitor caught it. They paused the run, removed all web-access environments, added a pre-execution tool-call monitor, and resumed with alignment-only phases. Overall, this is an impressive model release. We should wait for something bigger and even more impressive to come though. Because they have called the 8.4T phase "one rung of our predictive scaling ladder". But can we also expect the adoption of fancier architecture ideas like Linear Attention in the next model?