Zyphra releases ZYFA, an FP8 attention kernel for AMD MI355X GPUs
Article timeline
The articles in this story. Times are ET.
ZyphraOfficialZyphra finds most FP8 error comes from 8-bit input conversionAIZyphra says that across three models it studied, most FP8 error comes from converting inputs to 8 bits. On Mistral-7B, rotating queries and keys to spread out large values roughly halves input error, with almost no extra time when fused with quantization.
- ZyphraOfficialZyphra's ZYFA attention kernel runs in its ZOE engine on AMD MI355X
AIZyphra says ZYFA is the attention engine within its Zyphra Optimization Engine (ZOE) on AMD MI355X hardware. ZOE powers Zyphra Cloud and can be deployed on-premises. The company links a paper, code, and a blog post for ZYFA.
- ZyphraOfficialZyphra says ZYFA keeps more weights and finishes in fewer clock cycles
AIZyphra says rounding small weights to zero makes fewer bits switch, letting AITER run at a higher clock on real inputs but reducing accuracy. Its ZYFA method preserves more of those weights and still finishes sooner because it needs fewer clock cycles.
- ZyphraOfficialZyphra's ZYFA cuts FP8 attention numerical error 62–78% versus AITER
AIZyphra reports that its ZYFA method reduces numerical error by 62–78% compared with AITER's FP8 kernel at head dimension 128 on Mistral-7B and Qwen3-8B at 4K tokens. ZYFA rescales blocks before rounding so small attention weights are not incorrectly rounded to zero in FP8, and it normalizes using the rounded values.
- ZyphraOfficialZyphra's ZYFA cuts softmax work per score from 14 to 6 cycles
AIZyphra says its ZYFA "lean P" replaces softmax's exponential with a multiply-add and an integer conversion, cutting vector work per score from 14 to 6 cycles. The post notes that FP8 makes matrix math 2x faster than BF16 but leaves softmax unchanged, making softmax the bottleneck in long-context prefill.
- ZyphraOfficialZyphra schedules attention softmax and matrix multiplies to overlap in assembly
AIZyphra writes its attention instruction schedule in assembly so two thread groups alternate, one computing softmax while the other runs matrix multiplications. With lean P, the matrix unit is busy 86% of steady-state cycles, up from 60% using exponentials.
- ZyphraOfficialZyphra introduces ZYFA, an FP8 attention kernel for AMD MI355X
AIZyphra introduces Zyphra FlashAttention (ZYFA), an 8-bit floating-point attention kernel for AMD MI355X GPUs. Its prefill kernels deliver 1.16–1.76x the throughput of AMD's AITER inference kernel library on real model inputs at 16K tokens under sustained load.
