SemEval-2024 Task 7: Numeral-Aware Language Understanding and Generation
Chung-chi Chen, Jian-tao Huang, Hen-hsen Huang, Hiroya Takamura, Hsin-hsi Chen
Abstract
Numbers are frequently utilized in both our daily narratives and professional documents, such as clinical notes, scientific papers, financial documents, and legal court orders. The ability to understand and generate numbers is thus one of the essential aspects of evaluating large language models. In this vein, we propose a collection of datasets in SemEval-2024 Task 7 - NumEval. This collection encompasses several tasks focused on numeral-aware instances, including number prediction, natural language inference, question answering, reading comprehension, reasoning, and headline generation. This paper offers an overview of the dataset and presents the results of all subtasks in NumEval. Additionally, we contribute by summarizing participants’ methods and conducting an error analysis. To the best of our knowledge, NumEval represents one of the early tasks that perform peer evaluation in SemEval’s history. We will further share observations from this aspect and provide suggestions for future SemEval tasks.- Anthology ID:
- 2024.semeval-1.213
- Volume:
- Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024)
- Month:
- June
- Year:
- 2024
- Address:
- Mexico City, Mexico
- Editors:
- Atul Kr. Ojha, A. Seza Doğruöz, Harish Tayyar Madabushi, Giovanni Da San Martino, Sara Rosenthal, Aiala Rosá
- Venue:
- SemEval
- SIG:
- SIGLEX
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 1482–1491
- Language:
- URL:
- https://aclanthology.org/2024.semeval-1.213
- DOI:
- Cite (ACL):
- Chung-chi Chen, Jian-tao Huang, Hen-hsen Huang, Hiroya Takamura, and Hsin-hsi Chen. 2024. SemEval-2024 Task 7: Numeral-Aware Language Understanding and Generation. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), pages 1482–1491, Mexico City, Mexico. Association for Computational Linguistics.
- Cite (Informal):
- SemEval-2024 Task 7: Numeral-Aware Language Understanding and Generation (Chen et al., SemEval 2024)
- PDF:
- https://preview.aclanthology.org/jeptaln-2024-ingestion/2024.semeval-1.213.pdf