@inproceedings{rabbani-etal-2026-dialdefer,
    title = "{D}ial{D}efer: A Framework for Detecting and Mitigating {LLM} Dialogic Deference",
    author = "Rabbani, Parisa  and
      Sahoo, Priyam  and
      Mathew, Ruben  and
      Mondal, Aishee  and
      Ketharaman, Harshita  and
      Bozdag, Nimet Beyza  and
      Hakkani-T{\textbackslash}{''}ur, Dilek",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.2067/",
    pages = "44630--44671",
    ISBN = "979-8-89176-390-6",
    abstract = "LLMs are increasingly used as third-party judges, yet their reliability when evaluating speakers in dialogue remains poorly understood. We show that LLMs judge identical claims differently depending on framing: the same content receives different verdicts when presented as a statement to verify ({''}Is this statement correct?'') versus attributed to a speaker ({''}Is this speaker correct?''). We call this dialogic deference and introduce DialDefer, a framework for detecting and mitigating these framing-induced judgment shifts. Our Dialogic Deference Score (DDS) captures directional shifts that aggregate accuracy obscures. Across ten domains, 3k+ instances, and five models, conversational framing induces large shifts (mean |DDS| = 15.9 percentage points (pp) across models, p {\ensuremath{<}} .0001) while accuracy remains stable ({\ensuremath{<}}2 pp), with effects amplifying 2{--}5{\texttimes} on naturalistic Reddit conversations. This effect is domain-dependent: a single model can shift toward disagreement (skepticism) on graduate-level science and toward agreement (deference) on social judgment. Ablations reveal that human-vs-LLM attribution drives the largest shifts (17.7 pp swing), suggesting models treat disagreement with humans as more costly than with AI. Mitigation attempts can reduce deference but over-correct into skepticism, revealing a calibration problem beyond accuracy optimization."
}