@inproceedings{lu-etal-2023-efficient,
    title = "Efficient Classification of Long Documents via State-Space Models",
    author = "Lu, Peng  and
      Wang, Suyuchen  and
      Rezagholizadeh, Mehdi  and
      Liu, Bang  and
      Kobyzev, Ivan",
    editor = "Bouamor, Houda  and
      Pino, Juan  and
      Bali, Kalika",
    booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing",
    month = dec,
    year = "2023",
    address = "Singapore",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/Author-page-Marten-During-lu/2023.emnlp-main.404/",
    doi = "10.18653/v1/2023.emnlp-main.404",
    pages = "6559--6565",
    abstract = "Transformer-based models have achieved state-of-the-art performance on numerous NLP applications. However, long documents which are prevalent in real-world scenarios cannot be efficiently processed by transformers with the vanilla self-attention module due to their quadratic computation complexity and limited length extrapolation ability. Instead of tackling the computation difficulty for self-attention with sparse or hierarchical structures, in this paper, we investigate the use of State-Space Models (SSMs) for long document classification tasks. We conducted extensive experiments on six long document classification datasets, including binary, multi-class, and multi-label classification, comparing SSMs (with and without pre-training) to self-attention-based models. We also introduce the SSM-pooler model and demonstrate that it achieves comparable performance while being on average 36{\%} more efficient. Additionally our method exhibits higher robustness to the input noise even in the extreme scenario of 40{\%}."
}