Evaluating the Noisy Channel Model for the Normalization of Historical Texts: Basque, Spanish and Slovene
Izaskun Etxeberria, Iñaki Alegria, Larraitz Uria, Mans Hulden
Abstract
This paper presents a method for the normalization of historical texts using a combination of weighted finite-state transducers and language models. We have extended our previous work on the normalization of dialectal texts and tested the method against a 17th century literary work in Basque. This preprocessed corpus is made available in the LREC repository. The performance of this method for learning relations between historical and contemporary word forms is evaluated against resources in three languages. The method we present learns to map phonological changes using a noisy channel model. The model is based on techniques commonly used for phonological inference and producing Grapheme-to-Grapheme conversion systems encoded as weighted transducers and produces F-scores above 80% in the task for Basque. A wider evaluation shows that the approach performs equally well with all the languages in our evaluation suite: Basque, Spanish and Slovene. A comparison against other methods that address the same task is also provided.- Anthology ID:
- L16-1169
- Volume:
- Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)
- Month:
- May
- Year:
- 2016
- Address:
- Portorož, Slovenia
- Editors:
- Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis
- Venue:
- LREC
- SIG:
- Publisher:
- European Language Resources Association (ELRA)
- Note:
- Pages:
- 1064–1069
- Language:
- URL:
- https://aclanthology.org/L16-1169
- DOI:
- Cite (ACL):
- Izaskun Etxeberria, Iñaki Alegria, Larraitz Uria, and Mans Hulden. 2016. Evaluating the Noisy Channel Model for the Normalization of Historical Texts: Basque, Spanish and Slovene. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 1064–1069, Portorož, Slovenia. European Language Resources Association (ELRA).
- Cite (Informal):
- Evaluating the Noisy Channel Model for the Normalization of Historical Texts: Basque, Spanish and Slovene (Etxeberria et al., LREC 2016)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/L16-1169.pdf