Detecting spelling variants in non-standard texts

Fabian Barteld


Abstract
Spelling variation in non-standard language, e.g. computer-mediated communication and historical texts, is usually treated as a deviation from a standard spelling, e.g. 2mr as an non-standard spelling for tomorrow. Consequently, in normalization – the standard approach of dealing with spelling variation – so-called non-standard words are mapped to their corresponding standard words. However, there is not always a corresponding standard word. This can be the case for single types (like emoticons in computer-mediated communication) or a complete language, e.g. texts from historical languages that did not develop to a standard variety. The approach presented in this thesis proposal deals with spelling variation in absence of reference to a standard. The task is to detect pairs of types that are variants of the same morphological word. An approach for spelling-variant detection is presented, where pairs of potential spelling variants are generated with Levenshtein distance and subsequently filtered by supervised machine learning. The approach is evaluated on historical Low German texts. Finally, further perspectives are discussed.
Anthology ID:
E17-4002
Volume:
Proceedings of the Student Research Workshop at the 15th Conference of the European Chapter of the Association for Computational Linguistics
Month:
April
Year:
2017
Address:
Valencia, Spain
Editors:
Florian Kunneman, Uxoa Iñurrieta, John J. Camilleri, Mariona Coll Ardanuy
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
11–22
Language:
URL:
https://aclanthology.org/E17-4002
DOI:
Bibkey:
Cite (ACL):
Fabian Barteld. 2017. Detecting spelling variants in non-standard texts. In Proceedings of the Student Research Workshop at the 15th Conference of the European Chapter of the Association for Computational Linguistics, pages 11–22, Valencia, Spain. Association for Computational Linguistics.
Cite (Informal):
Detecting spelling variants in non-standard texts (Barteld, EACL 2017)
Copy Citation:
PDF:
https://preview.aclanthology.org/landing_page/E17-4002.pdf