@inproceedings{ruder-etal-2022-modular,
    title = "Modular and Parameter-Efficient Fine-Tuning for {NLP} Models",
    author = "Ruder, Sebastian  and
      Pfeiffer, Jonas  and
      Vuli{\'c}, Ivan",
    editor = "El-Beltagy, Samhaa R.  and
      Qiu, Xipeng",
    booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts",
    month = dec,
    year = "2022",
    address = "Abu Dubai, UAE",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2022.emnlp-tutorials.5/",
    doi = "10.18653/v1/2022.emnlp-tutorials.5",
    pages = "23--29",
    abstract = "State-of-the-art language models in NLP perform best when fine-tuned even on small datasets, but due to their increasing size, fine-tuning and downstream usage have become extremely compute-intensive. Being able to efficiently and effectively fine-tune the largest pre-trained models is thus key in order to reap the benefits of the latest advances in NLP. In this tutorial, we provide a comprehensive overview of parameter-efficient fine-tuning methods. We highlight their similarities and differences by presenting them in a unified view. We explore the benefits and usage scenarios of a neglected property of such parameter-efficient models{---}modularity{---}such as composition of modules to deal with previously unseen data conditions. We finally highlight how both properties{---}{---}parameter efficiency and modularity{---}{---}can be useful in the real-world setting of adapting pre-trained models to under-represented languages and domains with scarce annotated data for several downstream applications."
}