# Zyphra's ZYFA cuts FP8 attention numerical error 62–78% versus AITER

> Original title: Small attention weights can be incorrectly rounded to zero in FP8. ZYFA rescales blocks before rounding and normalizes using the rounded ...

- Source: Zyphra
- Published: 2026-11-06T02:36:48.000Z
- Bad Influence: https://badinfluence.news/items/wwvjzww7cvtam7z7q1z07526m
- Original: https://x.com/ZyphraAI/status/2118533024946303159

## Summary

Zyphra reports that its ZYFA method reduces numerical error by 62–78% compared with AITER's FP8 kernel at head dimension 128 on Mistral-7B and Qwen3-8B at 4K tokens. ZYFA rescales blocks before rounding so small attention weights are not incorrectly rounded to zero in FP8, and it normalizes using the rounded values.

## Post

Small attention weights can be incorrectly rounded to zero in FP8. ZYFA rescales blocks before rounding and normalizes using the rounded values.

At head dimension 128, this yields 62–78% less numerical error than AITER’s FP8 kernel on Mistral-7B and Qwen3-8B at 4K tokens.

![](https://badinfluence.news/api/img-proxy?u=https%3A%2F%2Fpbs.twimg.com%2Fmedia%2FHWaKV22bIAAnEWk.jpg&mode=thumb&exp=1794182400&sig=b7cb476954dcddec)