Abstract
This paper introduces a new approach to estimating the text document complexity. Common readability indices are based on average length of sentences and words. In contrast to these methods, we propose to count the number of rare words occurring abnormally often in the document. We use the reference corpus of texts and the quantile approach in order to determine what words are rare, and what frequencies are abnormal. We construct a general text complexity model, which can be adjusted for the specific task, and introduce two special models. The experimental design is based on a set of thematically similar pairs of Wikipedia articles, labeled using crowdsourcing. The experiments demonstrate the competitiveness of the proposed approach.- Anthology ID:
- R19-1031
- Volume:
- Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019)
- Month:
- September
- Year:
- 2019
- Address:
- Varna, Bulgaria
- Editors:
- Ruslan Mitkov, Galia Angelova
- Venue:
- RANLP
- SIG:
- Publisher:
- INCOMA Ltd.
- Note:
- Pages:
- 270–275
- Language:
- URL:
- https://aclanthology.org/R19-1031
- DOI:
- 10.26615/978-954-452-056-4_031
- Cite (ACL):
- Maksim Eremeev and Konstantin Vorontsov. 2019. Lexical Quantile-Based Text Complexity Measure. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 270–275, Varna, Bulgaria. INCOMA Ltd..
- Cite (Informal):
- Lexical Quantile-Based Text Complexity Measure (Eremeev & Vorontsov, RANLP 2019)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-5/R19-1031.pdf