Abstract
Quantifying and predicting morphological productivity is a long-standing challenge in corpus linguistics and psycholinguistics. The same challenge reappears in natural language processing in the context of handling words that were not seen in the training set (out-of-vocabulary, or OOV, words). Prior research showed that a good indicator of the productivity of a morpheme is the number of words involving it that occur exactly once (the hapax legomena). A technical connection was adduced between this result and Good-Turing smoothing, which assigns probability mass to unseen events on the basis of the simplifying assumption that word frequencies are stationary. In a large-scale study of 133 affixes in Wikipedia, we develop evidence that success in fact depends on tapping the frequency range in which the assumptions of Good-Turing are violated.- Anthology ID:
- W18-5814
- Volume:
- Proceedings of the Fifteenth Workshop on Computational Research in Phonetics, Phonology, and Morphology
- Month:
- October
- Year:
- 2018
- Address:
- Brussels, Belgium
- Venue:
- EMNLP
- SIG:
- SIGMORPHON
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 125–130
- Language:
- URL:
- https://aclanthology.org/W18-5814
- DOI:
- 10.18653/v1/W18-5814
- Cite (ACL):
- Janet Pierrehumbert and Ramon Granell. 2018. On Hapax Legomena and Morphological Productivity. In Proceedings of the Fifteenth Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 125–130, Brussels, Belgium. Association for Computational Linguistics.
- Cite (Informal):
- On Hapax Legomena and Morphological Productivity (Pierrehumbert & Granell, EMNLP 2018)
- PDF:
- https://preview.aclanthology.org/auto-file-uploads/W18-5814.pdf