Abstract
Informal romanization is an idiosyncratic process used by humans in informal digital communication to encode non-Latin script languages into Latin character sets found on common keyboards. Character substitution choices differ between users but have been shown to be governed by the same main principles observed across a variety of languages—namely, character pairs are often associated through phonetic or visual similarity. We propose a noisy-channel WFST cascade model for deciphering the original non-Latin script from observed romanized text in an unsupervised fashion. We train our model directly on romanized data from two languages: Egyptian Arabic and Russian. We demonstrate that adding inductive bias through phonetic and visual priors on character mappings substantially improves the model’s performance on both languages, yielding results much closer to the supervised skyline. Finally, we introduce a new dataset of romanized Russian, collected from a Russian social network website and partially annotated for our experiments.- Anthology ID:
- 2020.acl-main.737
- Volume:
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Month:
- July
- Year:
- 2020
- Address:
- Online
- Editors:
- Dan Jurafsky, Joyce Chai, Natalie Schluter, Joel Tetreault
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 8308–8319
- Language:
- URL:
- https://preview.aclanthology.org/build-pipeline-with-new-library/2020.acl-main.737/
- DOI:
- 10.18653/v1/2020.acl-main.737
- Cite (ACL):
- Maria Ryskina, Matthew R. Gormley, and Taylor Berg-Kirkpatrick. 2020. Phonetic and Visual Priors for Decipherment of Informal Romanization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8308–8319, Online. Association for Computational Linguistics.
- Cite (Informal):
- Phonetic and Visual Priors for Decipherment of Informal Romanization (Ryskina et al., ACL 2020)
- PDF:
- https://preview.aclanthology.org/build-pipeline-with-new-library/2020.acl-main.737.pdf
- Code
- ryskina/romanization-decipherment