# Stanford paper finds bits-per-byte is not fully tokenizer-agnostic

> Original title: New Stanford paper finds that no metric can be tokenizer-agnostic in every sense, so anyone comparing tokenizers should not trust a singl...

- Source: Rohan Paul
- Published: 2026-11-05T20:40:50.000Z
- Bad Influence: https://badinfluence.news/items/in457x1sdwvdppbfp0l904bnj
- Original: https://x.com/rohanpaul_ai/status/2118443442296832310

## Summary

Stanford researchers Rylan Schaeffer and Sanmi Koyejo report that bits-per-byte (BPB), a common metric for comparing models with different tokenizers, is not fully tokenizer-agnostic. Untrained models that guess uniformly still received different BPB scores by tokenizer, while accuracy stayed at chance. On trained models, correlations between BPB and task scores ran only from +0.02 to −0.30, so the authors say lower BPB should not be taken as proof of a better model.

## Post

New Stanford paper finds that no metric can be tokenizer-agnostic in every sense, so anyone comparing tokenizers should not trust a single score and should also check whether the model answers correctly.

To compare models with different tokenizers, researchers usually report bits-per-byte, or BPB, which divides the loss by bytes instead of tokens. Many papers call it tokenizer-agnostic.

They scored untrained models that guess uniformly, plus released models that share one recipe but use different tokenizers. They also rechecked a published claim that another supposedly tokenizer-neutral metric agrees with BPB.

Models that learned nothing still got different BPB by tokenizer, with English text scoring best near a 24K vocabulary, while accuracy stayed at chance. On models trained with different tokenizers, correlations between BPB and task scores ran only from +0.02 to −0.30.

When comparing tokenizers, don't treat lower BPB as proof of a better model, and check task accuracy on the same text before picking a vocabulary.

![](https://badinfluence.news/api/img-proxy?u=https%3A%2F%2Fpbs.twimg.com%2Fmedia%2FHWY5CMsbgAAiN8X.jpg&mode=thumb&exp=1794182400&sig=410b3c3274e0b612)