@inproceedings{torres-fonseca-etal-2026-safetyalfred,
    title = "{S}afety{ALFRED}: Evaluating Safety-Conscious Planning of Vision Language Models",
    author = "Torres-Fonseca, Josue  and
      Deng, Naihao  and
      Dai, Yinpei  and
      Storks, Shane  and
      Zhang, Yichi  and
      Mihalcea, Rada  and
      Kennington, Casey  and
      Chai, Joyce",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.1852/",
    pages = "37190--37211",
    ISBN = "979-8-89176-395-1",
    abstract = "Multimodal Large Language Models (MLLMs) are increasingly adopted as autonomous agents in interactive environments, yet their ability to proactively address safety hazards remains insufficient. We introduce SafetyALFRED, built upon the embodied agent benchmark ALFRED, augmented with six categories of real-world kitchen hazards. While existing safety evaluations focus on hazard recognition through disembodied question answering (QA) settings, we evaluate eleven state-of-the-art models from the Qwen, Gemma, and Gemini families on not only hazard recognition, but also active risk mitigation through embodied task planning. Our experimental results reveal a significant alignment gap: while models can accurately recognize hazards in QA settings, average mitigation success rates for these hazards are low in comparison. Our findings demonstrate that static evaluations through QA are insufficient for physical safety, advocating for a paradigm shift toward benchmarks that prioritize multi-step corrective actions in embodied context."
}