Abstract
This paper describes the AST submission to the WMT23 Shared Task on Parallel Data Curation. We experiment with two approaches for curating data from the provided web-scraped texts. We use sentence alignment to identify document alignments in the data and extract parallel sentence pairs from the aligned documents. All other sentences, not aligned in that step, are paired based on cosine similarity before we apply various different filters. For filtering, we use language detection, fluency classification, word alignments, cosine distance as calculated by multilingual sentence embedding models, and Bicleaner AI. Our best model outperforms the baseline by 1.9 BLEU points on average over the four provided evaluation sets.- Anthology ID:
- 2023.wmt-1.38
- Volume:
- Proceedings of the Eighth Conference on Machine Translation
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Philipp Koehn, Barry Haddow, Tom Kocmi, Christof Monz
- Venue:
- WMT
- SIG:
- SIGMT
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 366–374
- Language:
- URL:
- https://aclanthology.org/2023.wmt-1.38
- DOI:
- 10.18653/v1/2023.wmt-1.38
- Cite (ACL):
- Steinthor Steingrimsson. 2023. A Sentence Alignment Approach to Document Alignment and Multi-faceted Filtering for Curating Parallel Sentence Pairs from Web-crawled Data. In Proceedings of the Eighth Conference on Machine Translation, pages 366–374, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- A Sentence Alignment Approach to Document Alignment and Multi-faceted Filtering for Curating Parallel Sentence Pairs from Web-crawled Data (Steingrimsson, WMT 2023)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2023.wmt-1.38.pdf