Stanford paper finds bits-per-byte is not fully tokenizer-agnostic

Stanford researchers Rylan Schaeffer and Sanmi Koyejo report that bits-per-byte (BPB), a common metric for comparing models with different tokenizers, is not fully tokenizer-agnostic.

Untrained models that guess uniformly still received different BPB scores depending on tokenizer, while their accuracy stayed at chance.

On trained models, correlations between BPB and task scores ranged from +0.02 to −0.30. The authors say a lower BPB should not be taken as proof of a better model, and that comparisons should also check whether the model answers correctly.