FeatureBART: Feature Based Sequence-to-Sequence Pre-Training for Low-Resource NMT
Abhisek Chakrabarty, Raj Dabre, Chenchen Ding, Hideki Tanaka, Masao Utiyama, Eiichiro Sumita
Abstract
In this paper we present FeatureBART, a linguistically motivated sequence-to-sequence monolingual pre-training strategy in which syntactic features such as lemma, part-of-speech and dependency labels are incorporated into the span prediction based pre-training framework (BART). These automatically extracted features are incorporated via approaches such as concatenation and relevance mechanisms, among which the latter is known to be better than the former. When used for low-resource NMT as a downstream task, we show that these feature based models give large improvements in bilingual settings and modest ones in multilingual settings over their counterparts that do not use features.- Anthology ID:
- 2022.coling-1.443
- Volume:
- Proceedings of the 29th International Conference on Computational Linguistics
- Month:
- October
- Year:
- 2022
- Address:
- Gyeongju, Republic of Korea
- Editors:
- Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, Seung-Hoon Na
- Venue:
- COLING
- SIG:
- Publisher:
- International Committee on Computational Linguistics
- Note:
- Pages:
- 5014–5020
- Language:
- URL:
- https://preview.aclanthology.org/build-pipeline-with-new-library/2022.coling-1.443/
- DOI:
- Cite (ACL):
- Abhisek Chakrabarty, Raj Dabre, Chenchen Ding, Hideki Tanaka, Masao Utiyama, and Eiichiro Sumita. 2022. FeatureBART: Feature Based Sequence-to-Sequence Pre-Training for Low-Resource NMT. In Proceedings of the 29th International Conference on Computational Linguistics, pages 5014–5020, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
- Cite (Informal):
- FeatureBART: Feature Based Sequence-to-Sequence Pre-Training for Low-Resource NMT (Chakrabarty et al., COLING 2022)
- PDF:
- https://preview.aclanthology.org/build-pipeline-with-new-library/2022.coling-1.443.pdf