@inproceedings{jha-etal-2023-transformer,
    title = "Transformer-based Models for Long-Form Document Matching: Challenges and Empirical Analysis",
    author = "Jha, Akshita  and
      Samavedhi, Adithya  and
      Rakesh, Vineeth  and
      Chandrashekar, Jaideep  and
      Reddy, Chandan",
    editor = "Vlachos, Andreas  and
      Augenstein, Isabelle",
    booktitle = "Findings of the Association for Computational Linguistics: EACL 2023",
    month = may,
    year = "2023",
    address = "Dubrovnik, Croatia",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest_wac_2008/2023.findings-eacl.178/",
    doi = "10.18653/v1/2023.findings-eacl.178",
    pages = "2345--2355",
    abstract = "Recent advances in the area of long document matching have primarily focused on using transformer-based models for long document encoding and matching. There are two primary challenges associated with these models. Firstly, the performance gain provided by transformer-based models comes at a steep cost {--} both in terms of the required training time and the resource (memory and energy) consumption. The second major limitation is their inability to handle more than a pre-defined input token length at a time. In this work, we empirically demonstrate the effectiveness of simple neural models (such as feed-forward networks, and CNNs) and simple embeddings (like GloVe, and Paragraph Vector) over transformer-based models on the task of document matching. We show that simple models outperform the more complex BERT-based models while taking significantly less training time, energy, and memory. The simple models are also more robust to variations in document length and text perturbations."
}