Ahmad Al-Dahle previews KDA, AttnRes, and LatentMoE architecture innovations
Original titleStill waiting on the tech report but a few innovations: 1/ KDA swaps Gated DeltaNet's single scalar decay for learned per-dimension forge...
AISummary
Ahmad Al-Dahle outlined three architectural innovations ahead of a forthcoming technical report. KDA replaces Gated DeltaNet's single scalar decay with learned per-dimension forgetting, AttnRes retrieves selectively across depth, and LatentMoE activates 16 of 896 experts balanced from router score quantiles.
Source: Ahmad Al-Dahle · x.comPublished · added here