Cross-Align: Modeling Deep Cross-lingual Interactions for Word Alignment

Siyu Lai, Zhen Yang, Fandong Meng, Yufeng Chen, Jinan Xu, Jie Zhou


Abstract
Word alignment which aims to extract lexicon translation equivalents between source and target sentences, serves as a fundamental tool for natural language processing. Recent studies in this area have yielded substantial improvements by generating alignments from contextualized embeddings of the pre-trained multilingual language models. However, we find that the existing approaches capture few interactions between the input sentence pairs, which degrades the word alignment quality severely, especially for the ambiguous words in the monolingual context. To remedy this problem, we propose Cross-Align to model deep interactions between the input sentence pairs, in which the source and target sentences are encoded separately with the shared self-attention modules in the shallow layers, while cross-lingual interactions are explicitly constructed by the cross-attention modules in the upper layers. Besides, to train our model effectively, we propose a two-stage training framework, where the model is trained with a simple Translation Language Modeling (TLM) objective in the first stage and then finetuned with a self-supervised alignment objective in the second stage. Experiments show that the proposed Cross-Align achieves the state-of-the-art (SOTA) performance on four out of five language pairs.
Anthology ID:
2022.emnlp-main.244
Volume:
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
Month:
December
Year:
2022
Address:
Abu Dhabi, United Arab Emirates
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3715–3725
Language:
URL:
https://aclanthology.org/2022.emnlp-main.244
DOI:
Bibkey:
Cite (ACL):
Siyu Lai, Zhen Yang, Fandong Meng, Yufeng Chen, Jinan Xu, and Jie Zhou. 2022. Cross-Align: Modeling Deep Cross-lingual Interactions for Word Alignment. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3715–3725, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Cite (Informal):
Cross-Align: Modeling Deep Cross-lingual Interactions for Word Alignment (Lai et al., EMNLP 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/2022.emnlp-main.244.pdf