@inproceedings{valerio-miceli-barone-etal-2022-distributionally,
    title = "Distributionally Robust Recurrent Decoders with Random Network Distillation",
    author = "Valerio Miceli Barone, Antonio  and
      Birch, Alexandra  and
      Sennrich, Rico",
    editor = "Gella, Spandana  and
      He, He  and
      Majumder, Bodhisattwa Prasad  and
      Can, Burcu  and
      Giunchiglia, Eleonora  and
      Cahyawijaya, Samuel  and
      Min, Sewon  and
      Mozes, Maximilian  and
      Li, Xiang Lorraine  and
      Augenstein, Isabelle  and
      Rogers, Anna  and
      Cho, Kyunghyun  and
      Grefenstette, Edward  and
      Rimell, Laura  and
      Dyer, Chris",
    booktitle = "Proceedings of the 7th Workshop on Representation Learning for NLP",
    month = may,
    year = "2022",
    address = "Dublin, Ireland",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.repl4nlp-1.1",
    doi = "10.18653/v1/2022.repl4nlp-1.1",
    pages = "1--8",
    abstract = "Neural machine learning models can successfully model language that is similar to their training distribution, but they are highly susceptible to degradation under distribution shift, which occurs in many practical applications when processing out-of-domain (OOD) text. This has been attributed to {``}shortcut learning{''}{''}:'' relying on weak correlations over arbitrary large contexts. We propose a method based on OOD detection with Random Network Distillation to allow an autoregressive language model to automatically disregard OOD context during inference, smoothly transitioning towards a less expressive but more robust model as the data becomes more OOD, while retaining its full context capability when operating in-distribution. We apply our method to a GRU architecture, demonstrating improvements on multiple language modeling (LM) datasets.",
}