Haoxin Wang
2025
ECom-Bench: Can LLM Agent Resolve Real-World E-commerce Customer Support Issues?
Haoxin Wang
|
Xianhan Peng
|
Huang Cheng
|
Yizhe Huang
|
Ming Gong
|
Chenghan Yang
|
Yang Liu
|
Jiang Lin
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track
In this paper, we introduce , the first benchmark framework for evaluating LLM agent with multimodal capabilities in the e-commerce customer support domain. ECom-Bench features dynamic user simulation based on persona information collected from real e-commerce customer interactions and a realistic task dataset derived from authentic e-commerce dialogues. These tasks, covering a wide range of business scenarios, are designed to reflect real-world complexities, making highly challenging. For instance, even advanced models like GPT-4o achieve only a 10–20% pass3 metric in our benchmark, highlighting the substantial difficulties posed by complex e-commerce scenarios. The code and data have been made publicly available at https://github.com/XiaoduoAILab/ECom-Bench to facilitate further research and development in this domain.
Search
Fix author
Co-authors
- Huang Cheng 1
- Ming Gong 1
- Yizhe Huang 1
- Jiang Lin 1
- Yang Liu (刘扬) 1
- show all...