Abstract
This paper evaluates normalization procedures of Persian text for a downstream NLP task - multiword expressions (MWEs) discovery. We discuss the challenges the Persian language poses for NLP and evaluate open-source tools that try to address these difficulties. The best-performing tool is later used in the main task - MWEs discovery. In order to discover MWEs, we use association measures and a subpart of the MirasText corpus. The results show that an F-score is 26% higher in the case of normalized input data.- Anthology ID:
- 2021.ranlp-1.106
- Volume:
- Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021)
- Month:
- September
- Year:
- 2021
- Address:
- Held Online
- Editors:
- Ruslan Mitkov, Galia Angelova
- Venue:
- RANLP
- SIG:
- Publisher:
- INCOMA Ltd.
- Note:
- Pages:
- 929–939
- Language:
- URL:
- https://aclanthology.org/2021.ranlp-1.106
- DOI:
- Cite (ACL):
- Katarzyna Marszałek-Kowalewska. 2021. The Impact of Text Normalization on Multiword Expressions Discovery in Persian. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 929–939, Held Online. INCOMA Ltd..
- Cite (Informal):
- The Impact of Text Normalization on Multiword Expressions Discovery in Persian (Marszałek-Kowalewska, RANLP 2021)
- PDF:
- https://preview.aclanthology.org/bionlp-24-ingestion/2021.ranlp-1.106.pdf