Skip to content
View original post on X: BAAIOfficial
Published
22/100

LLaCO adds spherical noising and log-rank retention to diffusion language models

AILLaCO is a diffusion language model method that uses spherical noising, a geodesic interpolation toward a [MASK]-biased endpoint that avoids norm decay while keeping pretrained semantic biases. It also introduces Log-Rank Retention (Q_log), which measures rank decay on a logarithmic scale to focus on top candidates, and trajectory reallocation, which concentrates compute where token identities shift.
BAAIVerified on XOfficial
@BAAIBeijing
View on X

Part of a thread · earlier post

LLaCO fixes this with a grounded theoretical design:

Spherical Noising: Geodesic interpolation toward a [MASK]-biased endpoint, eliminating norm decay while preserving pretrained semantic biases.

Log-Rank Retention (Q_log): Measuring rank decay on a logarithmic scale to focus strictly on top candidates.

Trajectory Reallocation: Concentrating compute dynamically where token identities actually shift.

Source details

BAAI · x.comPublished

Original title: LLaCO fixes this with a grounded theoretical design:

Topics

Data & trainingPapers & research