Abstract
In the intersection of language understanding and numerical reasoning, a formidable challenge arises in natural language processing (NLP). Our study delves into the realm of NumEval, focusing on numeral-aware language understanding and generation using the QP, QQA and QNLI datasets. We harness the potential of the Orca2 model, Fine-tuning it in both normal and Chain-of-Thought modes with prompt tuning to enhance accuracy. Despite initial conjectures, our findings reveal intriguing disparities in model performance. While standard training methodologies yield commendable accuracy rates. The core contribution of this work lies in its elucidation of the intricate interplay between dataset sequencing and model performance. We expected to achieve a general model with the Fine Tuning model on the QP and QNLI datasets respectively, which has good accuracy in all three datasets. However, this goal was not achieved, and in order to achieve this goal, we introduce our structure.- Anthology ID:
- 2024.semeval-1.153
- Volume:
- Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024)
- Month:
- June
- Year:
- 2024
- Address:
- Mexico City, Mexico
- Editors:
- Atul Kr. Ojha, A. Seza Doğruöz, Harish Tayyar Madabushi, Giovanni Da San Martino, Sara Rosenthal, Aiala Rosá
- Venue:
- SemEval
- SIG:
- SIGLEX
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 1058–1063
- Language:
- URL:
- https://aclanthology.org/2024.semeval-1.153
- DOI:
- Cite (ACL):
- Mahmood Kalantari, Mehdi Feghhi, and Taha Khany Alamooti. 2024. MAMET at SemEval-2024 Task 7: Supervised Enhanced Reasoning Agent Model. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), pages 1058–1063, Mexico City, Mexico. Association for Computational Linguistics.
- Cite (Informal):
- MAMET at SemEval-2024 Task 7: Supervised Enhanced Reasoning Agent Model (Kalantari et al., SemEval 2024)
- PDF:
- https://preview.aclanthology.org/ingestion-checklist/2024.semeval-1.153.pdf