@inproceedings{wu-etal-2025-believe,
    title = "Should {I} Believe in What Medical {AI} Says? A {C}hinese Benchmark for Medication Based on Knowledge and Reasoning",
    author = "Wu, Yue  and
      Huang, Yangmin  and
      Du, Qianyun  and
      Lai, Lixian  and
      He, Zhiyang  and
      Hu, Jiaxue  and
      Tao, Xiaodong",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/landing_page/2025.acl-short.91/",
    pages = "1155--1164",
    ISBN = "979-8-89176-252-7",
    abstract = "Large language models (LLMs) show potential in healthcare but often generate hallucinations, especially when handling unfamiliar information. In medication, a systematic benchmark to evaluate model capabilities is lacking, which is critical given the high-risk nature of medical information. This paper introduces a Chinese benchmark aimed at assessing models in medication tasks, focusing on knowledge and reasoning across six datasets: indication, dosage and administration, contraindicated population, mechanisms of action, drug recommendation, and drug interaction. We evaluate eight closed-source and five open-source models to identify knowledge boundaries, providing the first systematic analysis of limitations and risks in proprietary medical models."
}