A Japanese Word Segmentation Proposal

Stalin Aguirre, Josafá Aguiar


Abstract
Current Japanese word segmentation methods, that use a morpheme-based approach, may produce different segmentations for the same strings. This occurs when these strings appear in different sentences. The cause is the influence of different contexts around these strings affecting the probabilistic models used in segmentation algorithms. This paper presents an alternative to the current morpheme-based scheme for Japanese word segmentation. The proposed scheme focuses on segmenting inflections as single words instead of separating the auxiliary verbs and other morphemes from the stems. Some morphological segmentation rules are presented for each type of word and these rules are implemented in a program which is properly described. The program is used to generate a segmentation of a sentence corpus, whose consistency is calculated and compared with the current morpheme-based segmentation of the same corpus. The experiments show that this method produces a much more consistent segmentation than the morpheme-based one.
Anthology ID:
P19-2060
Volume:
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop
Month:
July
Year:
2019
Address:
Florence, Italy
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
429–435
Language:
URL:
https://aclanthology.org/P19-2060
DOI:
10.18653/v1/P19-2060
Bibkey:
Cite (ACL):
Stalin Aguirre and Josafá Aguiar. 2019. A Japanese Word Segmentation Proposal. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 429–435, Florence, Italy. Association for Computational Linguistics.
Cite (Informal):
A Japanese Word Segmentation Proposal (Aguirre & Aguiar, ACL 2019)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/P19-2060.pdf