Abstract
Vietnamese word segmentation (VWS) is a challenging basic issue for natural language processing. This paper addresses the problem of how does dictionary size influence VWS performance, proposes two novel measures: square overlap ratio (SOR) and relaxed square overlap ratio (RSOR), and validates their effectiveness. The SOR measure is the product of dictionary overlap ratio and corpus overlap ratio, and the RSOR measure is the relaxed version of SOR measure under an unsupervised condition. The two measures both indicate the suitable degree between segmentation dictionary and object corpus waiting for segmentation. The experimental results show that the more suitable, neither smaller nor larger, dictionary size is better to achieve the state-of-the-art performance for dictionary-based Vietnamese word segmenters.- Anthology ID:
- L16-1172
- Volume:
- Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)
- Month:
- May
- Year:
- 2016
- Address:
- Portorož, Slovenia
- Venue:
- LREC
- SIG:
- Publisher:
- European Language Resources Association (ELRA)
- Note:
- Pages:
- 1079–1083
- Language:
- URL:
- https://aclanthology.org/L16-1172
- DOI:
- Cite (ACL):
- Wuying Liu and Lin Wang. 2016. How does Dictionary Size Influence Performance of Vietnamese Word Segmentation?. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 1079–1083, Portorož, Slovenia. European Language Resources Association (ELRA).
- Cite (Informal):
- How does Dictionary Size Influence Performance of Vietnamese Word Segmentation? (Liu & Wang, LREC 2016)
- PDF:
- https://preview.aclanthology.org/ingestion-script-update/L16-1172.pdf