@inproceedings{martins-etal-2022-former,
    title = "$\infty$-former: Infinite Memory Transformer",
    author = "Martins, Pedro Henrique  and
      Marinho, Zita  and
      Martins, Andre",
    editor = "Muresan, Smaranda  and
      Nakov, Preslav  and
      Villavicencio, Aline",
    booktitle = "Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = may,
    year = "2022",
    address = "Dublin, Ireland",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/jlcl-multiple-ingestion/2022.acl-long.375/",
    doi = "10.18653/v1/2022.acl-long.375",
    pages = "5468--5485",
    abstract = "Transformers are unable to model long-term memories effectively, since the amount of computation they need to perform grows with the context length. While variations of efficient transformers have been proposed, they all have a finite memory capacity and are forced to drop old information. In this paper, we propose the $\infty$-former, which extends the vanilla transformer with an unbounded long-term memory. By making use of a continuous-space attention mechanism to attend over the long-term memory, the $\infty$-former`s attention complexity becomes independent of the context length, trading off memory length with precision.In order to control where precision is more important, $\infty$-former maintains {\textquotedblleft}sticky memories,{\textquotedblright} being able to model arbitrarily long contexts while keeping the computation budget fixed.Experiments on a synthetic sorting task, language modeling, and document grounded dialogue generation demonstrate the $\infty$-former`s ability to retain information from long sequences."
}