@inproceedings{dervishi-etal-2025-training,
    title = "Training compute-optimal transformer encoder models",
    author = "Dervishi, Megi  and
      Allauzen, Alexandre  and
      Synnaeve, Gabriel  and
      LeCun, Yann",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.1804/",
    pages = "35602--35617",
    ISBN = "979-8-89176-332-6",
    abstract = "Transformer encoders are critical for a wide range of Natural Language Processing (NLP) tasks, yet their compute{--}efficiency remains poorly understood. We present the first comprehensive empirical investigation of compute-optimal pretraining for encoder transformers using the Masked Language Modeling (MLM) objective. Across hundreds of carefully controlled runs we vary model size, data size, batch size, learning rate, and masking ratio, with increasing compute budget. The compute-optimal data-to-model ratio of Transformer encoder models is 10 to 100 times larger than the ratio of auto-regressive models. Using these recipes, we train OptiBERT, a family of compute-optimal BERT-style models that matches or surpasses leading baselines{---}including ModernBERT and NeoBERT{---}on GLUE and MTEB while training with dramatically less FLOPS."
}