Abstract
High-quality span representations are crucial to natural language processing tasks involving span prediction and classification. Most existing methods derive a span representation by aggregation of token representations within the span. In contrast, we aim to improve span representations by considering span-span interactions as well as more comprehensive span-token interactions. Specifically, we introduce layers of span-level attention on top of a normal token-level transformer encoder. Given that attention between all span pairs results in O(n4) complexity (n being the sentence length) and not all span interactions are intuitively meaningful, we restrict the range of spans that a given span could attend to, thereby reducing overall complexity to O(n3). We conduct experiments on various span-related tasks and show superior performance of our model surpassing baseline models. Our code is publicly available at https://github.com/jipy0222/Span-Level-Attention.- Anthology ID:
- 2023.findings-emnlp.747
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2023
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 11184–11192
- Language:
- URL:
- https://aclanthology.org/2023.findings-emnlp.747
- DOI:
- 10.18653/v1/2023.findings-emnlp.747
- Cite (ACL):
- Pengyu Ji, Songlin Yang, and Kewei Tu. 2023. Improving Span Representation by Efficient Span-Level Attention. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11184–11192, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- Improving Span Representation by Efficient Span-Level Attention (Ji et al., Findings 2023)
- PDF:
- https://preview.aclanthology.org/emnlp-22-attachments/2023.findings-emnlp.747.pdf