Reflection AI's Beam open model has 501B total parameters and 23B active
Original title- How is it so efficient? 1) an RL length penalty that discourages unnecessary tokens; 2) Sparse MoE: 501B parameters, 23B active per token
AISummary
Sophia Yang congratulated Reflection AI on Beam, a 501B-parameter open model with 23B active per token. She attributes its efficiency to an RL length penalty that discourages unnecessary tokens and a sparse MoE architecture. Reflection says full weights will be released this month, and the quoted post reports training over 100 million rollouts on 10.5K NVIDIA GB300 GPUs over four weeks.
Source: Sophia Yang · x.com