@inproceedings{li-etal-2025-alert,
    title = "{ALERT}: An {LLM}-powered Benchmark for Automatic Evaluation of Recommendation Explanations",
    author = "Li, Yichuan  and
      Zhang, Xinyang  and
      Zhang, Chenwei  and
      Li, Mao  and
      Liu, Tianyi  and
      Chen, Pei  and
      Gao, Yifan  and
      Lee, Kyumin  and
      Ding, Kaize  and
      Wang, Zhengyang  and
      Zhang, Zhihan  and
      Shang, Jingbo  and
      Li, Xian  and
      Chilimbi, Trishul",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.naacl-long.137/",
    pages = "2704--2719",
    ISBN = "979-8-89176-189-6",
    abstract = "Recommendation explanation systems have become increasingly vital with the widespread adoption of recommender systems. However, existing recommendation explanation evaluation benchmarks suffer from limited item diversity, impractical user profiling requirements, and unreliable and unscalable evaluation protocols. We present ALERT, a model-agnostic recommendation explanation evaluation benchmark. The benchmark comprises three main contributions: 1) a diverse dataset encompassing 15 Amazon e-commerce categories with 2,761 user-item interactions, incorporating implicit preferences through purchase histories;2) two novel LLM-powered automatic evaluators that enable scalable and human-preference aligned evaluation of explanations; and 3) a robust divide-and-aggregate approach that synthesizes multiple LLM judgments, achieving 70{\%} concordance with expert human evaluation and substantially outperforming existing methods.ALERT facilitates comprehensive evaluation of recommendation explanations across diverse domains, advancing the development of more effective explanation systems."
}