AlphaIntellect at SemEval-2024 Task 6: Detection of Hallucinations in Generated Text
Sohan Choudhury, Priyam Saha, Subharthi Ray, Shankha Das, Dipankar Das
Abstract
One major issue in natural language generation (NLG) models is detecting hallucinations (semantically inaccurate outputs). This study investigates a hallucination detection system designed for three distinct NLG tasks: definition modeling, paraphrase generation, and machine translation. The system uses feedforward neural networks for classification and SentenceTransformer models for similarity scores and sentence embeddings. Even though the SemEval-2024 benchmark shows good results, there is still room for improvement. Promising paths toward improving performance include considering multi-task learning methods, including strategies for handling out-of-domain data minimizing bias, and investigating sophisticated architectures.- Anthology ID:
- 2024.semeval-1.137
- Volume:
- Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024)
- Month:
- June
- Year:
- 2024
- Address:
- Mexico City, Mexico
- Editors:
- Atul Kr. Ojha, A. Seza Doğruöz, Harish Tayyar Madabushi, Giovanni Da San Martino, Sara Rosenthal, Aiala Rosá
- Venue:
- SemEval
- SIG:
- SIGLEX
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 952–958
- Language:
- URL:
- https://aclanthology.org/2024.semeval-1.137
- DOI:
- Cite (ACL):
- Sohan Choudhury, Priyam Saha, Subharthi Ray, Shankha Das, and Dipankar Das. 2024. AlphaIntellect at SemEval-2024 Task 6: Detection of Hallucinations in Generated Text. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), pages 952–958, Mexico City, Mexico. Association for Computational Linguistics.
- Cite (Informal):
- AlphaIntellect at SemEval-2024 Task 6: Detection of Hallucinations in Generated Text (Choudhury et al., SemEval 2024)
- PDF:
- https://preview.aclanthology.org/ingestion-checklist/2024.semeval-1.137.pdf