Abstract
Transformer-based “behemoths” have grown in popularity, as well as structurally, shattering multiple NLP benchmarks along the way. However, their real-world usability remains a question. In this work, we empirically assess the feasibility of applying transformer-based models in real-world ad-hoc retrieval applications by comparison to a “greener and more sustainable” alternative, comprising only 620 trainable parameters. We present an analysis of their efficacy and efficiency and show that considering limited computational resources, the lighter model running on the CPU achieves a 3 to 20 times speedup in training and 7 to 47 times in inference while maintaining a comparable retrieval performance. Code to reproduce the efficiency experiments is available on “https://github.com/bioinformatics-ua/EACL2021-reproducibility/“.- Anthology ID:
- 2021.eacl-main.293
- Volume:
- Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume
- Month:
- April
- Year:
- 2021
- Address:
- Online
- Venue:
- EACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 3343–3353
- Language:
- URL:
- https://aclanthology.org/2021.eacl-main.293
- DOI:
- 10.18653/v1/2021.eacl-main.293
- Cite (ACL):
- Tiago Almeida and Sérgio Matos. 2021. Benchmarking a transformer-FREE model for ad-hoc retrieval. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3343–3353, Online. Association for Computational Linguistics.
- Cite (Informal):
- Benchmarking a transformer-FREE model for ad-hoc retrieval (Almeida & Matos, EACL 2021)
- PDF:
- https://preview.aclanthology.org/nodalida-main-page/2021.eacl-main.293.pdf
- Code
- bioinformatics-ua/EACL2021-reproducibility
- Data
- CORD-19