Automatically Extracting Variant-Normalization Pairs for Japanese Text Normalization
Itsumi Saito, Kyosuke Nishida, Kugatsu Sadamitsu, Kuniko Saito, Junji Tomita
Abstract
Social media texts, such as tweets from Twitter, contain many types of non-standard tokens, and the number of normalization approaches for handling such noisy text has been increasing. We present a method for automatically extracting pairs of a variant word and its normal form from unsegmented text on the basis of a pair-wise similarity approach. We incorporated the acquired variant-normalization pairs into Japanese morphological analysis. The experimental results show that our method can extract widely covered variants from large Twitter data and improve the recall of normalization without degrading the overall accuracy of Japanese morphological analysis.- Anthology ID:
- I17-1094
- Volume:
- Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
- Month:
- November
- Year:
- 2017
- Address:
- Taipei, Taiwan
- Editors:
- Greg Kondrak, Taro Watanabe
- Venue:
- IJCNLP
- SIG:
- Publisher:
- Asian Federation of Natural Language Processing
- Note:
- Pages:
- 937–946
- Language:
- URL:
- https://aclanthology.org/I17-1094
- DOI:
- Cite (ACL):
- Itsumi Saito, Kyosuke Nishida, Kugatsu Sadamitsu, Kuniko Saito, and Junji Tomita. 2017. Automatically Extracting Variant-Normalization Pairs for Japanese Text Normalization. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 937–946, Taipei, Taiwan. Asian Federation of Natural Language Processing.
- Cite (Informal):
- Automatically Extracting Variant-Normalization Pairs for Japanese Text Normalization (Saito et al., IJCNLP 2017)
- PDF:
- https://preview.aclanthology.org/teach-a-man-to-fish/I17-1094.pdf