Segmented Recurrent Transformer: An Efficient Sequence-to-Sequence Model

Yinghan Long, Sayeed Chowdhury, Kaushik Roy

Abstract
Transformers have shown dominant performance across a range of domains including language and vision. However, their computational cost grows quadratically with the sequence length, making their usage prohibitive for resource-constrained applications. To counter this, our approach is to divide the whole sequence into segments and apply attention to the individual segments. We propose a segmented recurrent transformer (SRformer) that combines segmented (local) attention with recurrent attention. The loss caused by reducing the attention window length is compensated by aggregating information across segments with recurrent attention. SRformer leverages Recurrent Accumulate-and-Fire (RAF) neurons’ inherent memory to update the cumulative product of keys and values. The segmented attention and lightweight RAF neurons ensure the efficiency of the proposed transformer. Such an approach leads to models with sequential processing capability at a lower computation/memory cost. We apply the proposed method to T5 and BART transformers. The modified models are tested on summarization datasets including CNN-dailymail, XSUM, ArXiv, and MediaSUM. Notably, using segmented inputs of varied sizes, the proposed model achieves 6-22% higher ROUGE1 scores than a segmented transformer and outperforms other recurrent transformer approaches. Furthermore, compared to full attention, the proposed model reduces the computational complexity of cross attention by around 40%.
Anthology ID:
2023.findings-emnlp.558
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2023
Month:
December
Year:
2023
Address:
Singapore
Editors:
Houda Bouamor, Juan Pino, Kalika Bali
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
8325–8337
Language:
URL:
https://aclanthology.org/2023.findings-emnlp.558
DOI:
10.18653/v1/2023.findings-emnlp.558
Bibkey:
Cite (ACL):
Yinghan Long, Sayeed Chowdhury, and Kaushik Roy. 2023. Segmented Recurrent Transformer: An Efficient Sequence-to-Sequence Model. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8325–8337, Singapore. Association for Computational Linguistics.
Cite (Informal):
Segmented Recurrent Transformer: An Efficient Sequence-to-Sequence Model (Long et al., Findings 2023)
Copy Citation:
PDF:
https://preview.aclanthology.org/emnlp-22-attachments/2023.findings-emnlp.558.pdf
PDF Search