Choosing which model layers to run at NVFP4 4-bit precision

Baseten explains how to decide which layers of a model can run at 4-bit NVFP4 quantization without losing information the model needs.

Its guide compares three approaches: architecture-based heuristics, isolated-layer sensitivity scoring, and SaturationQuant, which accounts for the effect of other quantized layers. The post also covers calibrating with representative data and using block-level scales over groups of 16 values.

The post is a first-party explanation from Baseten, so it describes the company's own method guidance rather than independent testing of the approaches.