Zyphra derives theory explaining recency bias in NoPE attention layers
Original titleWe derive a mathematical theory showing how local layers create this recency bias, and how it propagates through the residual stream, nor...
AISummary
Zyphra derives a mathematical theory showing how local layers produce recency bias that propagates through the residual stream, norms, and MLPs into NoPE attention logits. The authors validate the theory on both randomly initialized and trained networks.
Source: Zyphra · x.comPublished · added here