@inproceedings{riley-chiang-2022-continuum,
    title = "A Continuum of Generation Tasks for Investigating Length Bias and Degenerate Repetition",
    author = "Riley, Darcey  and
      Chiang, David",
    editor = "Bastings, Jasmijn  and
      Belinkov, Yonatan  and
      Elazar, Yanai  and
      Hupkes, Dieuwke  and
      Saphra, Naomi  and
      Wiegreffe, Sarah",
    booktitle = "Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP",
    month = dec,
    year = "2022",
    address = "Abu Dhabi, United Arab Emirates (Hybrid)",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest_wac_2008/2022.blackboxnlp-1.36/",
    doi = "10.18653/v1/2022.blackboxnlp-1.36",
    pages = "426--440",
    abstract = "Language models suffer from various degenerate behaviors. These differ between tasks: machine translation (MT) exhibits length bias, while tasks like story generation exhibit excessive repetition. Recent work has attributed the difference to task constrainedness, but evidence for this claim has always involved many confounding variables. To study this question directly, we introduce a new experimental framework that allows us to smoothly vary task constrainedness, from MT at one end to fully open-ended generation at the other, while keeping all other aspects fixed. We find that: (1) repetition decreases smoothly with constrainedness, explaining the difference in repetition across tasks; (2) length bias surprisingly also decreases with constrainedness, suggesting some other cause for the difference in length bias; (3) across the board, these problems affect the mode, not the whole distribution; (4) the differences cannot be attributed to a change in the entropy of the distribution, since another method of changing the entropy, label smoothing, does not produce the same effect."
}