Mixing English data beats hyperparameter tuning for low-resource Arabic pre-training
Overview
Apple Machine Learning Research reports that mixing high-resource English data into Arabic pre-training outperformed hyperparameter tuning across about 1,000 runs.
The authors tested models from 150M to 1.43B parameters and report gains equal to 2–3 times the unique Arabic data on validation loss, and 2–13 times on downstream accuracy. The authors, Paul Jeha, Anastasiia Sedova, Louis Béthune, Skyler Seto, Jes Frellsen, Pierre Ablin and Natalie Schluter, say validation loss on the target language underestimates the value of mixing. They recommend prioritizing the mixing ratio and transferring hyperparameters from a small proxy model via μP.
Written by AI from the articles below · updated Oct 13, 11:07 AM ET
Check the sources:
Article timeline
The articles in this story. Times are ET.
- Apple Machine Learning ResearchOfficialBilingual pre-training with English data outperforms hyperparameter tuning for Arabic models
AIApple Machine Learning Research reports that mixing high-resource English data into low-resource Arabic pre-training outperformed hyperparameter tuning across about 1000 runs. Models from 150M to 1.43B parameters showed gains equal to 2–3× the unique Arabic data on validation loss and 2–13× on downstream accuracy. The authors say target-language validation loss underestimates mixing's value and recommend prioritizing the mixing ratio and transferring hyperparameters via μP.
Heat trend
Not enough continuous observations to show a trend yet.