Abstract
Current Japanese word segmentation methods, that use a morpheme-based approach, may produce different segmentations for the same strings. This occurs when these strings appear in different sentences. The cause is the influence of different contexts around these strings affecting the probabilistic models used in segmentation algorithms. This paper presents an alternative to the current morpheme-based scheme for Japanese word segmentation. The proposed scheme focuses on segmenting inflections as single words instead of separating the auxiliary verbs and other morphemes from the stems. Some morphological segmentation rules are presented for each type of word and these rules are implemented in a program which is properly described. The program is used to generate a segmentation of a sentence corpus, whose consistency is calculated and compared with the current morpheme-based segmentation of the same corpus. The experiments show that this method produces a much more consistent segmentation than the morpheme-based one.- Anthology ID:
- P19-2060
- Volume:
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop
- Month:
- July
- Year:
- 2019
- Address:
- Florence, Italy
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 429–435
- Language:
- URL:
- https://aclanthology.org/P19-2060
- DOI:
- 10.18653/v1/P19-2060
- Cite (ACL):
- Stalin Aguirre and Josafá Aguiar. 2019. A Japanese Word Segmentation Proposal. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 429–435, Florence, Italy. Association for Computational Linguistics.
- Cite (Informal):
- A Japanese Word Segmentation Proposal (Aguirre & Aguiar, ACL 2019)
- PDF:
- https://preview.aclanthology.org/ingestion-script-update/P19-2060.pdf