Albert Gu's H-Net learns language from raw bytes, claiming 3-5x efficiency gains

AITri Dao says H-Net dynamically tokenizes variable names in coding data as separate tokens. He reports it scales much better on code than BPE. The post quotes @_albertgu, who says the new H-Net version gains up to 3-5x efficiency in controlled scaling law experiments and matches an 11T-token open-source model with about 10x less data and compute at larger scale.

AIWill Depue, owner of OpenAI's X account, says a new result is real and a big deal, citing Karpathy's view that the tokenizer is the transformer's ugliest part. Albert Gu says his team's new hierarchical network (H-Net) reaches up to 3-5x efficiency gains in controlled scaling experiments and matches an open-source model trained on 11T tokens with about 10x less data and compute.