@inproceedings{di-liello-etal-2022-effective,
    title = "Effective Pretraining Objectives for Transformer-based Autoencoders",
    author = "Di Liello, Luca  and
      Gabburo, Matteo  and
      Moschitti, Alessandro",
    editor = "Goldberg, Yoav  and
      Kozareva, Zornitsa  and
      Zhang, Yue",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2022",
    month = dec,
    year = "2022",
    address = "Abu Dhabi, United Arab Emirates",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2022.findings-emnlp.405/",
    doi = "10.18653/v1/2022.findings-emnlp.405",
    pages = "5533--5547",
    abstract = "In this paper, we study trade-offs between efficiency, cost and accuracy when pre-training Transformer encoders with different pre-training objectives. For this purpose, we analyze features of common objectives and combine them to create new effective pre-training approaches. Specifically, we designed light token generators based on a straightforward statistical approach, which can replace ELECTRA computationally heavy generators, thus highly reducing cost. Our experiments also show that (i) there are more efficient alternatives to BERT{'}s MLM, and (ii) it is possible to efficiently pre-train Transformer-based models using lighter generators without a significant drop in performance."
}