@inproceedings{zhang-etal-2026-vision,
    title = "Vision-Language Models Mistake Head Orientation for Gaze Direction: Nonverbal Conversation Cues",
    author = "Zhang, Zory  and
      Feng, Pinyuan  and
      Wang, Bingyang  and
      Zhao, Tianwei  and
      Yu, Suyang  and
      Gao, Qingying  and
      Deng, Hokin  and
      Ma, Ziqiao  and
      Li, Yijiang  and
      Luo, Dezhi",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.504/",
    pages = "10370--10389",
    ISBN = "979-8-89176-395-1",
    abstract = "Where someone looks is a nonverbal communication cue that children and adults readily use.How well can Vision-Language Models (VLMs) infer gaze targets? To construct evaluation stimuli, we captured 1,360 real-world photos of scenes in which a person gazes at one of several objects on a table. Importantly, we also controlled the gazer{'}s head orientation: sometimes it was directed toward the gaze target, sometimes toward a distractor object, and sometimes left unconstrained. We found a substantial performance gap between VLMs and humans, ruled out alternative explanations such as resolution and object-naming skills, and identified the main reason for the gap as VLMs inferring gaze direction using head orientation rather than eye appearance.Such a bias is likely due to data rather than architecture, as suggested by a proof-of-concept experiment finetuning a transformer-based vision model.Future work should investigate whether these findings hold broadly across various deep learning methods trained on existing data, and whether better data mitigates this problem for all architectures.Pinpointing the reason sets the stage for technologies that can interpret gaze targets to have more efficient interactions with humans."
}