Study finds AI models use more tokens for some languages, raising costs
Overview
The Coalition for Language Equity in AI Research published a study finding that AI models need far more tokens to process some languages than others.
Among the 20 most popular internet languages, the most expensive used 1.5 to 5 times as many tokens as the cheapest. For one model, the gap grew to 17 times across 200 languages.
The researchers call this extra cost a "tokenizer tax." According to the study, it raises costs and latency and reduces the effective context capacity available to speakers of those languages.
Written by AI from the articles below · updated Oct 26, 6:12 AM ET
Check the sources:
Article timeline
The articles in this story. Times are ET.
- Rest of WorldNewsAI models cost more for non-English speakers, study finds
AIA study by the Coalition for Language Equity in AI Research found AI models need far more tokens for some languages than others. Among the 20 most popular internet languages, the most expensive used 1.5 to 5 times more tokens than the cheapest, a gap that grew 17 times across 200 languages in one model. The researchers call this a "tokenizer tax," which raises costs and latency and reduces effective context capacity.
Heat Trend
- Current comparable heat
- 13
- Peak comparable heat
- 24Oct 27, 9:00 AM ET
- Change in past 24 hours
- +166%
The trend compares only continuously observed participants, so its scope may be smaller than the current heat total. Hover or tap for hourly heat; use arrow keys with a keyboard.