# OpenAI's Will Depue says a new byte-level model could replace tokenizers

> Original title: i think this is real and a big deal. as karpathy said, the ugliest part of the transformer is the tokenizer

- Source: will depue
- Published: 2026-10-28T19:31:11.000Z
- Bad Influence: https://badinfluence.news/items/t64f4mj863dkvrbso4ngkyq6f
- Original: https://x.com/willdepue/status/2115526812306719142

## Summary

Will Depue, owner of OpenAI's X account, says a new result is real and a big deal, citing Karpathy's view that the tokenizer is the transformer's ugliest part. Albert Gu says his team's new hierarchical network (H-Net) reaches up to 3-5x efficiency gains in controlled scaling experiments and matches an open-source model trained on 11T tokens with about 10x less data and compute.

## Post

i think this is real and a big deal. as karpathy said, the ugliest part of the transformer is the tokenizer

## Quoted post @_albertgu

> I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data.
> 
> Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less?
> 
> Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.

https://x.com/_albertgu/status/2115502110578290900