Bilingual pre-training with English data outperforms hyperparameter tuning for Arabic models

Apple Machine Learning Research reports that mixing high-resource English data into low-resource Arabic pre-training outperformed hyperparameter tuning across about 1000 runs.

Models from 150M to 1.43B parameters showed gains equal to 2–3× the unique Arabic data on validation loss and 2–13× on downstream accuracy.

The authors say target-language validation loss underestimates mixing's value and recommend prioritizing the mixing ratio and transferring hyperparameters via μP.