NVIDIA and AI4Bharat release a 2.8 trillion token Indic pretraining corpus and generation pipeline
Original titleScaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline
NVIDIA and AI4Bharat announce Nemotron-pretraining-Indic-v1, about 2.8 trillion tokens across 22 Indian languages, with an open GPU-accelerated pipeline used to create it.
The corpus includes native-script data for all 22 languages and romanized data for twelve, and the release is described as coming soon.
In a Nemotron 3.5 Super VL comparison, the Indic continued pretraining stage raised scores on all five reported Indic benchmarks, with gains from 4.98 to 9.82 points.
The post explains how the corpus was built, its native and romanized coverage, and the benchmark gains after continued pretraining, with stated limits.
Source: Hugging Face Blog · huggingface.coPublished · added here