@inproceedings{li-etal-2025-scedit,
    title = "{S}c{E}dit: Script-based Assessment of Knowledge Editing",
    author = "Li, Xinye  and
      Zheng, Zunwen  and
      Zhang, Qian  and
      Zhuang, Dekai  and
      Kang, Jiabao  and
      Xu, Liyan  and
      Liu, Qingbin  and
      Chen, Xi  and
      Tu, Zhiying  and
      Chu, Dianhui  and
      Sui, Dianbo",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/mtsummit-25-ingestion/2025.findings-acl.104/",
    doi = "10.18653/v1/2025.findings-acl.104",
    pages = "2032--2052",
    ISBN = "979-8-89176-256-5",
    abstract = "Knowledge Editing (KE) has gained increasing attention, yet current KE tasks remain relatively simple. Under current evaluation frameworks, many editing methods achieve exceptionally high scores, sometimes nearing perfection. However, few studies integrate KE into real-world application scenarios (e.g., recent interest in LLM-as-agent). To support our analysis, we introduce a novel script-based benchmark {--} ScEdit (Script-based Knowledge Editing Benchmark) {--} which encompasses both counterfactual and temporal edits. We integrate token-level and text-level evaluation methods, comprehensively analyzing existing KE techniques. The benchmark extends traditional fact-based ({``}What''-type question) evaluation to action-based ({``}How''-type question) evaluation. We observe that all KE methods exhibit a drop in performance on established metrics and face challenges on text-level metrics, indicating a challenging task. Our benchmark is available at https://github.com/asdfo123/ScEdit."
}