@inproceedings{gehring-roth-2025-ambistory,
    title = "{A}mbi{S}tory: A Challenging Dataset of Lexically Ambiguous Short Stories",
    author = "Gehring, Janosch  and
      Roth, Michael",
    editor = "Frermann, Lea  and
      Stevenson, Mark",
    booktitle = "Proceedings of the 14th Joint Conference on Lexical and Computational Semantics (*SEM 2025)",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.starsem-1.12/",
    pages = "152--171",
    ISBN = "979-8-89176-340-1",
    abstract = "Word sense disambiguation is the task of selecting a word{'}s applicable word sense in a given context. However, ambiguous texts may lack the information necessary to disambiguate words completely, resulting in multiple word senses with varying degrees of plausibility. We design a dataset around this premise: Our samples consist of 4{--}5 sentence short stories, where the sentence with the word to be disambiguated is itself ambiguous and surrounding sentences only contain indirect clues towards the more plausible word sense. We collect annotations from humans who rate the plausibility of a given word sense on a scale from 1{--}5. In total, our dataset contains 19,701 human word sense annotations on 1,899 stories. We investigate the performance of large language models on our data and find that many poorly correlate with human judgments. We also find that fine-tuning on our data can increase performance."
}