Skip to content
View original post on X: ZyphraOfficial
Published
30/100

Zyphra schedules attention softmax and matrix multiplies to overlap in assembly

AIZyphra writes its attention instruction schedule in assembly so two thread groups alternate, one computing softmax while the other runs matrix multiplications. With lean P, the matrix unit is busy 86% of steady-state cycles, up from 60% using exponentials.
ZyphraVerified on XOfficial
@ZyphraAI
View on X

Part of a thread · earlier post

We write the instruction schedule in assembly so two groups of threads take turns: one computes the softmax (attention weights) while the other performs matrix multiplications.

With lean P, the matrix unit is busy for 86% of steady-state cycles, up from 60% using exponentials.

Source details

Zyphra · x.comPublished · added here

Original title: We write the instruction schedule in assembly so two groups of threads take turns: one computes the softmax (attention weights) while the...

Topics

Deployment & engineeringTutorials & how-to