@inproceedings{dong-etal-2026-using,
    title = "Using Perspectival Words Is Harder Than Vocabulary Words for Humans {---}and {E}ven More So for Multimodal Language Models",
    author = "Dong, Dota Tianai  and
      Luo, Yifan  and
      Wang, Po-Ya Angela  and
      Ozyurek, Asli  and
      Rubio-Fernandez, Paula",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.84/",
    pages = "1845--1870",
    ISBN = "979-8-89176-390-6",
    abstract = "Multimodal language models (MLMs) increasingly demonstrate human-like communication, yet their use of everyday perspectival words remains poorly understood. To address this gap, we compare humans and MLMs in their use of three word types, which we predict impose increasing cognitive demands: vocabulary (e.g., `boat' or `cup'), possessives (e.g., `mine' vs. `yours'), and demonstratives (e.g., `this one' vs. `that one'). Testing seven MLMs against human participants, we find that perspectival words are harder than vocabulary words for both groups. The gap is even larger for MLMs: while models approach human-level performance on using vocabulary, they exhibit clear deficits with possessives and even greater difficulties with demonstratives. Ablation analyses point to limitations in perspective-taking and spatial reasoning as key sources of these gaps in MLMs. Instruction-based prompting helps close the gap for possessives but still leaves demonstratives far below human performance. These results show that, unlike vocabulary, perspectival words pose a greater challenge in human communication{---}and this difficulty is further amplified in MLMs, revealing a crucial shortfall in their pragmatic and social-cognitive abilities."
}