@inproceedings{mosbach-2023-analyzing,
    title = "Analyzing Pre-trained and Fine-tuned Language Models",
    author = "Mosbach, Marius",
    editor = "Elazar, Yanai  and
      Ettinger, Allyson  and
      Kassner, Nora  and
      Ruder, Sebastian  and
      A. Smith, Noah",
    booktitle = "Proceedings of the Big Picture Workshop",
    month = dec,
    year = "2023",
    address = "Singapore",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2023.bigpicture-1.10/",
    doi = "10.18653/v1/2023.bigpicture-1.10",
    pages = "123--134",
    abstract = "Since the introduction of transformer-based language models in 2018, the current generation of natural language processing (NLP) models continues to demonstrate impressive capabilities on a variety of academic benchmarks and real-world applications. This progress is based on a simple but general pipeline which consists of pre-training neural language models on large quantities of text, followed by an adaptation step that fine-tunes the pre-trained model to perform a specific NLP task of interest. However, despite the impressive progress on academic benchmarks and the widespread deployment of pre-trained and fine-tuned language models in industry we still lack a fundamental understanding of how and why pre-trained and fine-tuned language models work as well as the individual steps of the pipeline that produce them. We makes several contributions towards improving our understanding of pre-trained and fine-tuned language models, ranging from analyzing the linguistic knowledge of pre-trained language models and how it is affected by fine-tuning, to a rigorous analysis of the fine-tuning process itself and how the choice of adaptation technique affects the generalization of models and thereby provide new insights about previously unexplained phenomena and the capabilities of pre-trained and fine-tuned language models."
}