@inproceedings{sorodoc-etal-2022-challenges,
    title = "Challenges in including extra-linguistic context in pre-trained language models",
    author = "Sorodoc, Ionut  and
      Aina, Laura  and
      Boleda, Gemma",
    editor = "Tafreshi, Shabnam  and
      Sedoc, Jo{\~a}o  and
      Rogers, Anna  and
      Drozd, Aleksandr  and
      Rumshisky, Anna  and
      Akula, Arjun",
    booktitle = "Proceedings of the Third Workshop on Insights from Negative Results in NLP",
    month = may,
    year = "2022",
    address = "Dublin, Ireland",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2022.insights-1.18/",
    doi = "10.18653/v1/2022.insights-1.18",
    pages = "134--138",
    abstract = "To successfully account for language, computational models need to take into account both the linguistic context (the content of the utterances) and the extra-linguistic context (for instance, the participants in a dialogue). We focus on a referential task that asks models to link entity mentions in a TV show to the corresponding characters, and design an architecture that attempts to account for both kinds of context. In particular, our architecture combines a previously proposed specialized module (an ``entity library'') for character representation with transfer learning from a pre-trained language model. We find that, although the model does improve linguistic contextualization, it fails to successfully integrate extra-linguistic information about the participants in the dialogue. Our work shows that it is very challenging to incorporate extra-linguistic information into pre-trained language models."
}