@inproceedings{meshi-etal-2026-convapparel,
    title = "{C}onv{A}pparel: A Benchmark Dataset and Validation Framework for User Simulators in Conversational Recommenders",
    author = "Meshi, Ofer  and
      Balog, Krisztian  and
      Goldman, Sally  and
      Caciularu, Avi  and
      Tennenholtz, Guy  and
      Jeong, Jihwan  and
      Globerson, Amir  and
      Boutilier, Craig",
    editor = "Demberg, Vera  and
      Inui, Kentaro  and
      Marquez, Llu{\'i}s",
    booktitle = "Proceedings of the 19th Conference of the {E}uropean Chapter of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = mar,
    year = "2026",
    address = "Rabat, Morocco",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-eacl/2026.eacl-long.244/",
    pages = "5270--5304",
    ISBN = "979-8-89176-380-7",
    abstract = "The promise of *LLM-based user simulators* to improve conversational AI is hindered by a critical ``realism gap,'' leading to systems that are optimized for simulated interactions, but may fail to perform well in the real world. We introduce *ConvApparel*, a new dataset of human-AI conversations designed to address this gap. Its unique dual-agent data collection protocol, using both ``good'' and ``bad'' recommenders, enables counterfactual validation by capturing a wide spectrum of user experiences, enriched with first-person annotations of user satisfaction.We propose a comprehensive validation framework that combines *statistical alignment*, a *human-likeness score*, and *counterfactual validation* to test for generalization.Our experiments reveal a significant realism gap across all simulators. However, the framework also shows that data-driven simulators outperform a prompted baseline, particularly in counterfactual validation where they adapt more realistically to unseen behaviors, suggesting they embody more robust, if imperfect, user models."
}