Abstract
There are several parallel corpora available for many language pairs, such as CCMatrix, built from mass downloads of web content and automatic detection of segments in one language and the translation equivalent in another. These techniques can produce large parallel corpora, but of questionable quality. In many cases, the segments are not in the required languages, or if they are, they are not translation equivalents. In this article, we present an algorithm for filtering out the segments in languages other than the required ones and re-scoring the segments using SBERT. A use case on the Spanish-Asturian and Spanish-Catalan CCMatrix corpus is presented.- Anthology ID:
- 2023.eamt-1.5
- Volume:
- Proceedings of the 24th Annual Conference of the European Association for Machine Translation
- Month:
- June
- Year:
- 2023
- Address:
- Tampere, Finland
- Editors:
- Mary Nurminen, Judith Brenner, Maarit Koponen, Sirkku Latomaa, Mikhail Mikhailov, Frederike Schierl, Tharindu Ranasinghe, Eva Vanmassenhove, Sergi Alvarez Vidal, Nora Aranberri, Mara Nunziatini, Carla Parra Escartín, Mikel Forcada, Maja Popovic, Carolina Scarton, Helena Moniz
- Venue:
- EAMT
- SIG:
- Publisher:
- European Association for Machine Translation
- Note:
- Pages:
- 39–45
- Language:
- URL:
- https://aclanthology.org/2023.eamt-1.5
- DOI:
- Cite (ACL):
- Antoni Oliver González and Sergi Álvarez. 2023. Filtering and rescoring the CCMatrix corpus for Neural Machine Translation training. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 39–45, Tampere, Finland. European Association for Machine Translation.
- Cite (Informal):
- Filtering and rescoring the CCMatrix corpus for Neural Machine Translation training (González & Álvarez, EAMT 2023)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-1/2023.eamt-1.5.pdf