SemEval-2026 Task 5: Rating Plausibility of Word Senses in Ambiguous Stories through Narrative Understanding

Janosch Gehring, Selina Meyer, Michael Roth


Abstract
We introduce SemEval-2026 Task 5 on "Rating Plausibility of Word Senses in Ambiguous Stories through Narrative Understanding". The dataset for this task consists of 4-5 sentence English short stories. In each story, one sentence includes a lexical ambiguity and different senses are to be judged in terms of plausibility on a Likert scale. The task is intentionally constructed to be challenging by stories only providing sparse contextual cues. We give an overview of well-performing, frequent and interesting approaches used by participating systems. From a total of 175 registered participants and 27 submitted system description papers, the best system achieved an "accuracy within standard deviation" score of 93.3%.
Anthology ID:
2026.semeval-1.448
Volume:
Proceedings of the 20th International Workshop on Semantic Evaluation (2026)
Month:
July
Year:
2026
Address:
San Diego, California, USA
Editors:
Ekaterina Kochmar, Debanjan Ghosh, Kai North, Mamoru Komachi
Venues:
SemEval | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3691–3703
Language:
URL:
https://preview.aclanthology.org/ingest-acl-workshops/2026.semeval-1.448/
DOI:
Bibkey:
Cite (ACL):
Janosch Gehring, Selina Meyer, and Michael Roth. 2026. SemEval-2026 Task 5: Rating Plausibility of Word Senses in Ambiguous Stories through Narrative Understanding. In Proceedings of the 20th International Workshop on Semantic Evaluation (2026), pages 3691–3703, San Diego, California, USA. Association for Computational Linguistics.
Cite (Informal):
SemEval-2026 Task 5: Rating Plausibility of Word Senses in Ambiguous Stories through Narrative Understanding (Gehring et al., SemEval 2026)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingest-acl-workshops/2026.semeval-1.448.pdf
Supplementarymaterial:
 2026.semeval-1.448.SupplementaryMaterial.zip