Abstract
Word segmentation is crucial in natural language processing tasks for unsegmented languages. In Japanese, many out-of-vocabulary words appear in the phonetic syllabary katakana, making segmentation more difficult due to the lack of clues found in mixed script settings. In this paper, we propose a straightforward approach based on a variant of tf-idf and apply it to the problem of word segmentation in Japanese. Even though our method uses only an unlabeled corpus, experimental results show that it achieves performance comparable to existing methods that use manually labeled corpora. Furthermore, it improves performance of simple word segmentation models trained on a manually labeled corpus.- Anthology ID:
- I17-2013
- Volume:
- Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers)
- Month:
- November
- Year:
- 2017
- Address:
- Taipei, Taiwan
- Editors:
- Greg Kondrak, Taro Watanabe
- Venue:
- IJCNLP
- SIG:
- Publisher:
- Asian Federation of Natural Language Processing
- Note:
- Pages:
- 74–79
- Language:
- URL:
- https://preview.aclanthology.org/build-pipeline-with-new-library/I17-2013/
- DOI:
- Cite (ACL):
- Yoshinari Fujinuma and Alvin Grissom II. 2017. Substring Frequency Features for Segmentation of Japanese Katakana Words with Unlabeled Corpora. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 74–79, Taipei, Taiwan. Asian Federation of Natural Language Processing.
- Cite (Informal):
- Substring Frequency Features for Segmentation of Japanese Katakana Words with Unlabeled Corpora (Fujinuma & Grissom II, IJCNLP 2017)
- PDF:
- https://preview.aclanthology.org/build-pipeline-with-new-library/I17-2013.pdf