Abstract
Automatic Readability Assessment (ARA) predicts the level of difficulty of a text, e.g. at Grade 1 to Grade 12. ARA is an ordinal classification task since the predicted levels follow an underlying order, from easy to difficult. However, most neural ARA models ignore the distance between the gold level and predicted level, treating all levels as independent labels. This paper investigates whether distance-sensitive loss functions can improve ARA performance. We evaluate a variety of loss functions on neural ARA models, and show that ordinal log-loss can produce statistically significant improvement over the standard cross-entropy loss in terms of adjacent accuracy in a majority of our datasets.- Anthology ID:
- 2024.bea-1.28
- Volume:
- Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024)
- Month:
- June
- Year:
- 2024
- Address:
- Mexico City, Mexico
- Editors:
- Ekaterina Kochmar, Marie Bexte, Jill Burstein, Andrea Horbach, Ronja Laarmann-Quante, Anaïs Tack, Victoria Yaneva, Zheng Yuan
- Venue:
- BEA
- SIG:
- SIGEDU
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 343–350
- Language:
- URL:
- https://aclanthology.org/2024.bea-1.28
- DOI:
- Cite (ACL):
- Ho Hung Lim and John Lee. 2024. Improving Readability Assessment with Ordinal Log-Loss. In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), pages 343–350, Mexico City, Mexico. Association for Computational Linguistics.
- Cite (Informal):
- Improving Readability Assessment with Ordinal Log-Loss (Lim & Lee, BEA 2024)
- PDF:
- https://preview.aclanthology.org/jeptaln-2024-ingestion/2024.bea-1.28.pdf