For the first time, we also added a mid-training stage before RL. We trained on several trillion tokens of code, scientific articles, blogs, and more, saving the scarcest, highest-quality data for last.
Cognition adds a mid-training stage before reinforcement learning
AICognition says it added a mid-training stage before reinforcement learning for the first time. The stage trains on several trillion tokens of code, scientific articles, and blogs, with the scarcest, highest-quality data saved for last.
Source details
Cognition · x.comPublished · added here
Original title: For the first time, we also added a mid-training stage before RL. We trained on several trillion tokens of code, scientific articles, blo...