Super interesting result: H-Net learns to dynamically tokenize variable names in coding data as separate tokens, and it scales much better on code compared to BPE
H-Net learns to tokenize code variable names and scales better than BPE
AITri Dao says H-Net dynamically tokenizes variable names in coding data as separate tokens. He reports it scales much better on code than BPE. The post quotes @_albertgu, who says the new H-Net version gains up to 3-5x efficiency in controlled scaling law experiments and matches an 11T-token open-source model with about 10x less data and compute at larger scale.
Tri DaoVerified on X
@tri_dao
I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data. Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less? Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.
Source details
Tri Dao · x.comPublished · added here
Original title: Super interesting result: H-Net learns to dynamically tokenize variable names in coding data as separate tokens, and it scales much bette...