@inproceedings{hasan-2026-structhallu,
    title = "{S}truct{H}allu-Drift: Benchmarking Structured Hallucinations Under Schema Evolution in {LLM}s",
    author = "Hasan, Mujtaba",
    editor = "Gupta, Vivek  and
      Ding, Kaize  and
      Kokel, Harsha  and
      Zhao, Yue  and
      Agarwal, Amit  and
      Wang, Yu  and
      Glass, Michael  and
      Zhang, Yu  and
      Srinivas, Kavitha  and
      Chen, Xiusi  and
      Hassanzadeh, Oktie  and
      Zhu, Qi  and
      Chang, Shuaichen  and
      Luo, Yuan",
    booktitle = "Proceedings of the First Workshop on Structured Understanding, Retrieval, and Generation in the {LLM} Era ({SURG}e{LLM} 2026)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl-workshops/2026.surgellm-1.22/",
    pages = "333--343",
    ISBN = "979-8-89176-406-4",
    abstract = "Large Language Models (LLMs) are increasingly used to generate structured outputs{---}JSON objects, SQL queries, and structured records{---}from formal schemas. While recent advances in constrained decoding and schema-aware prompting have improved syntactic compliance, the semantic reliability of these outputs remains poorly characterized. We investigate this gap through the lens of schema drift{---}the inevitable evolution of database schemas in production environments through column renamings, type changes, and constraint modifications.We introduce StructHallu-Drift, a benchmark and evaluation framework for studying structured hallucinations under schema evolution. We contribute: (1) a six-category hallucination taxonomy that disentangles syntactic validity from semantic fidelity; (2) a controlled evaluation suite applying realistic schema mutations at three severity levels to established NL-to-structure datasets; and (3) a systematic evaluation of four LLMs spanning 7B to 70B parameters across three structured output tasks.Experiments on 1,200 schema{--}model evaluation instances reveal four key findings: (i) 39{--}54{\%} of structured outputs contain at least one semantic hallucination; (ii) schema drift severity has surprisingly minimal effect on hallucination rates ({\ensuremath{\sim}}44{\%} across all levels, p = 0.59), suggesting imperfect schema conditioning under our prompting setup; (iii) output format is the dominant factor in generation reliability, with SQL achieving {\ensuremath{\sim}}85{\%} semantic validity while schema-grounded record generation drops to 7{--}24{\%}; (iv) each model exhibits a distinct hallucination fingerprint, implying that mitigation strategies must be model-specific rather than universal. We publicly release our benchmark and evaluation toolkit."
}