AlphaIntellect at SemEval-2024 Task 6: Detection of Hallucinations in Generated Text

Sohan Choudhury, Priyam Saha, Subharthi Ray, Shankha Das, Dipankar Das


Abstract
One major issue in natural language generation (NLG) models is detecting hallucinations (semantically inaccurate outputs). This study investigates a hallucination detection system designed for three distinct NLG tasks: definition modeling, paraphrase generation, and machine translation. The system uses feedforward neural networks for classification and SentenceTransformer models for similarity scores and sentence embeddings. Even though the SemEval-2024 benchmark shows good results, there is still room for improvement. Promising paths toward improving performance include considering multi-task learning methods, including strategies for handling out-of-domain data minimizing bias, and investigating sophisticated architectures.
Anthology ID:
2024.semeval-1.137
Volume:
Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024)
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Atul Kr. Ojha, A. Seza Doğruöz, Harish Tayyar Madabushi, Giovanni Da San Martino, Sara Rosenthal, Aiala Rosá
Venue:
SemEval
SIG:
SIGLEX
Publisher:
Association for Computational Linguistics
Note:
Pages:
952–958
Language:
URL:
https://aclanthology.org/2024.semeval-1.137
DOI:
10.18653/v1/2024.semeval-1.137
Bibkey:
Cite (ACL):
Sohan Choudhury, Priyam Saha, Subharthi Ray, Shankha Das, and Dipankar Das. 2024. AlphaIntellect at SemEval-2024 Task 6: Detection of Hallucinations in Generated Text. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), pages 952–958, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
AlphaIntellect at SemEval-2024 Task 6: Detection of Hallucinations in Generated Text (Choudhury et al., SemEval 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-4/2024.semeval-1.137.pdf
Supplementary material:
 2024.semeval-1.137.SupplementaryMaterial.zip
Supplementary material:
 2024.semeval-1.137.SupplementaryMaterial.txt