@inproceedings{otmakhova-etal-2026-fluke,
    title = "{FLUKE}: A Linguistically-Driven and Task-Agnostic Framework for Robustness Evaluation",
    author = "Otmakhova, Yulia  and
      Truong, Thinh Hung  and
      Mahendra, Rahmad  and
      Zhai, Zenan  and
      Zhu, Rongxin  and
      Beck, Daniel  and
      Lau, Jey Han",
    editor = "Demberg, Vera  and
      Inui, Kentaro  and
      Marquez, Llu{\'i}s",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {EACL} 2026",
    month = mar,
    year = "2026",
    address = "Rabat, Morocco",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-eacl/2026.findings-eacl.269/",
    pages = "5103--5123",
    ISBN = "979-8-89176-386-9",
    abstract = "We present FLUKE (Framework for LingUistically-driven and tasK-agnostic robustness Evaluation), a framework for assessing model robustness through systematic minimal variations of test data. FLUKE introduces controlled variations across linguistic levels {---} from orthography to dialect and style {---} and leverages large language models (LLMs) with human validation to generate modifications. We demonstrate FLUKE{'}s utility by evaluating both fine-tuned models and LLMs across six diverse NLP tasks (four classification and two generation tasks), and reveal that (1) the impact of linguistic variations is highly task-dependent, with some tests being critical for certain tasks but irrelevant for others; (2) LLMs still exhibit significant brittleness to certain linguistic variations, with reasoning LLMs surprisingly showing less robustness on some tasks compared to base models, and scaling improving robustness only for surface-level modifications; (3) models are overall more brittle to natural, fluent modifications such as syntax or style changes (and especially to negation), compared to corruption-style tests such as letter flipping; (4) the ability of a model to use a linguistic feature in generation does not correlate to its robustness to this feature on downstream tasks. These findings highlight the importance of systematic robustness testing for understanding model behaviors."
}