Abstract
Numerical reasoning is challenging even for large pre-trained language models. We show that while T5 models are capable of generating relevant headlines with proper numerical values, they can also make mistakes in reading comprehension and miscalculate numerical values. To overcome these issues, we propose a two-step training process: first train models to read text and generate formal representations of calculations, then train models to read calculations and generate numerical values. On the SemEval 2024 Task 7 headline fill-in-the-blank task, our two-stage Flan-T5-based approach achieved 88% accuracy. On the headline generation task, our T5-based approach achieved RougeL of 0.390, BERT F1 Score of 0.453, and MoverScore of 0.587.- Anthology ID:
- 2024.semeval-1.6
- Volume:
- Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024)
- Month:
- June
- Year:
- 2024
- Address:
- Mexico City, Mexico
- Editors:
- Atul Kr. Ojha, A. Seza Doğruöz, Harish Tayyar Madabushi, Giovanni Da San Martino, Sara Rosenthal, Aiala Rosá
- Venue:
- SemEval
- SIG:
- SIGLEX
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 34–39
- Language:
- URL:
- https://aclanthology.org/2024.semeval-1.6
- DOI:
- Cite (ACL):
- Hinoki Crum and Steven Bethard. 2024. hinoki at SemEval-2024 Task 7: Numeral-Aware Headline Generation (English). In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), pages 34–39, Mexico City, Mexico. Association for Computational Linguistics.
- Cite (Informal):
- hinoki at SemEval-2024 Task 7: Numeral-Aware Headline Generation (English) (Crum & Bethard, SemEval 2024)
- PDF:
- https://preview.aclanthology.org/jeptaln-2024-ingestion/2024.semeval-1.6.pdf