@inproceedings{clive-etal-2022-control,
    title = "Control Prefixes for Parameter-Efficient Text Generation",
    author = "Clive, Jordan  and
      Cao, Kris  and
      Rei, Marek",
    editor = "Bosselut, Antoine  and
      Chandu, Khyathi  and
      Dhole, Kaustubh  and
      Gangal, Varun  and
      Gehrmann, Sebastian  and
      Jernite, Yacine  and
      Novikova, Jekaterina  and
      Perez-Beltrachini, Laura",
    booktitle = "Proceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM)",
    month = dec,
    year = "2022",
    address = "Abu Dhabi, United Arab Emirates (Hybrid)",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/jlcl-multiple-ingestion/2022.gem-1.31/",
    doi = "10.18653/v1/2022.gem-1.31",
    pages = "363--382",
    abstract = "Prefix-tuning is a parameter-efficient and powerful technique for adapting a pre-trained language model to a downstream application. However, it uses the same dataset-level tuned set of parameters for all examples in the dataset. We extend the framework with a dynamic method, Control Prefixes, which allows for the effective inclusion of input-dependent information, thereby demonstrating how prefix-tuning can be used for controlled text generation tasks. The method incorporates attribute-level learnable representations into different layers of a pre-trained Transformer, enabling the generated text to be guided in a particular direction. We provide a systematic evaluation of the technique and apply it to five datasets from the GEM benchmark for natural language generation (NLG). Using only 0.1{--}2{\%} additional trainable parameters, we show Control Prefixes can even outperform full fine-tuning methods, and present state-of-the-art results on several data-to-text datasets, including WebNLG. We also examine the common case where input-dependent information is unavailable at test time and show Control Prefixes can excel in this setting also."
}