Across the three models we studied, most FP8 error comes from converting the inputs to 8 bits.
On Mistral-7B, rotating queries and keys to spread out large values roughly halves input error, with almost no extra time when fused with quantization.
Zyphra finds most FP8 error comes from 8-bit input conversion
AIZyphra says that across three models it studied, most FP8 error comes from converting inputs to 8 bits. On Mistral-7B, rotating queries and keys to spread out large values roughly halves input error, with almost no extra time when fused with quantization.
Source details
Zyphra · x.comPublished · added here
Original title: Across the three models we studied, most FP8 error comes from converting the inputs to 8 bits.