Mix, Don't Tune: Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settings

A systematic comparison across four model scales shows that mixing data from a high-resource auxiliary language yields larger improvements than hyperparameter tuning in data-constrained language model pre-training. Using Arabic as the low-resource target and English as the auxiliary, mixing boosts performance by an amount equivalent to 2–3× the unique target data on validation loss and 2–13× on downstream task accuracy.