Kimi Delta Attention's recency bias shapes how models retain earlier information
Original titleThe same recency bias also appears in models using Kimi Delta Attention, which controls how much earlier information a model keeps. As it...
AISummary
Models using Kimi Delta Attention show a recency bias, as the mechanism controlling how much earlier information is kept shapes memory. As new information mixes into that memory, nearby words share more information, which gives the model clues about how far apart words are.
Source: Zyphra · x.comPublished · added here