@inproceedings{mehri-eskenazi-2021-gensf,
    title = "{G}en{SF}: Simultaneous Adaptation of Generative Pre-trained Models and Slot Filling",
    author = "Mehri, Shikib  and
      Eskenazi, Maxine",
    editor = "Li, Haizhou  and
      Levow, Gina-Anne  and
      Yu, Zhou  and
      Gupta, Chitralekha  and
      Sisman, Berrak  and
      Cai, Siqi  and
      Vandyke, David  and
      Dethlefs, Nina  and
      Wu, Yan  and
      Li, Junyi Jessy",
    booktitle = "Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue",
    month = jul,
    year = "2021",
    address = "Singapore and Online",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/jlcl-multiple-ingestion/2021.sigdial-1.51/",
    doi = "10.18653/v1/2021.sigdial-1.51",
    pages = "489--498",
    abstract = "In transfer learning, it is imperative to achieve strong alignment between a pre-trained model and a downstream task. Prior work has done this by proposing task-specific pre-training objectives, which sacrifices the inherent scalability of the transfer learning paradigm. We instead achieve strong alignment by simultaneously modifying both the pre-trained model and the formulation of the downstream task, which is more efficient and preserves the scalability of transfer learning. We present GenSF (Generative Slot Filling), which leverages a generative pre-trained open-domain dialog model for slot filling. GenSF (1) adapts the pre-trained model by incorporating inductive biases about the task and (2) adapts the downstream task by reformulating slot filling to better leverage the pre-trained model`s capabilities. GenSF achieves state-of-the-art results on two slot filling datasets with strong gains in few-shot and zero-shot settings. We achieve a 9 F1 score improvement in zero-shot slot filling. This highlights the value of strong alignment between the pre-trained model and the downstream task."
}