@inproceedings{kong-etal-2026-malurlbench,
    title = "{M}al{URLB}ench: A Benchmark Evaluating Agents' Vulnerabilities When Processing Web {URL}s",
    author = "Kong, Dezhang  and
      Wu, Zhuxi  and
      Liu, Shiqi  and
      Tan, ZhiCheng  and
      Lu, Kuichen  and
      Li, Minghao  and
      Liu, Qichen  and
      Chu, Shengyu  and
      Xu, Zhenhua  and
      Liu, Xuan  and
      Han, Meng",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.716/",
    pages = "14589--14601",
    ISBN = "979-8-89176-395-1",
    abstract = "LLM-based web agents have become increasingly popular for their utility in daily life and work. However, they exhibit critical vulnerabilities when processing malicious URLs: accepting a disguised malicious URL enables subsequent access to unsafe webpages, which can cause severe damage to service providers and users. Despite this risk, no benchmark currently targets this emerging threat. To address this gap, we propose MalURLBench, the first benchmark for evaluating LLMs' vulnerabilities to malicious URLs. MalURLBench contains 61,845 attack instances spanning 10 real-world scenarios and 7 categories of real malicious websites. Experiments with 12 popular LLMs reveal that existing models struggle to detect elaborately disguised malicious URLs. We further identify and analyze key factors that impact attack success rates and propose URLGuard, a lightweight defense module. We believe this work will provide a foundational resource for advancing the security of web agents."
}