The Impact of Text Normalization on Multiword Expressions Discovery in Persian

Katarzyna Marszałek-Kowalewska


Abstract
This paper evaluates normalization procedures of Persian text for a downstream NLP task - multiword expressions (MWEs) discovery. We discuss the challenges the Persian language poses for NLP and evaluate open-source tools that try to address these difficulties. The best-performing tool is later used in the main task - MWEs discovery. In order to discover MWEs, we use association measures and a subpart of the MirasText corpus. The results show that an F-score is 26% higher in the case of normalized input data.
Anthology ID:
2021.ranlp-1.106
Volume:
Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021)
Month:
September
Year:
2021
Address:
Held Online
Venue:
RANLP
SIG:
Publisher:
INCOMA Ltd.
Note:
Pages:
929–939
Language:
URL:
https://aclanthology.org/2021.ranlp-1.106
DOI:
Bibkey:
Cite (ACL):
Katarzyna Marszałek-Kowalewska. 2021. The Impact of Text Normalization on Multiword Expressions Discovery in Persian. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 929–939, Held Online. INCOMA Ltd..
Cite (Informal):
The Impact of Text Normalization on Multiword Expressions Discovery in Persian (Marszałek-Kowalewska, RANLP 2021)
Copy Citation:
PDF:
https://preview.aclanthology.org/update-css-js/2021.ranlp-1.106.pdf