Bartosz Przybył
2025
Evaluating Conversational Agents with Persona-driven User Simulations based on Large Language Models: A Sales Bot Case Study
Justyna Gromada
|
Alicja Kasicka
|
Ewa Komkowska
|
Lukasz Krajewski
|
Natalia Krawczyk
|
Morgan Veyret
|
Bartosz Przybył
|
Lina M. Rojas-Barahona
|
Michał K. Szczerbak
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track
We present a novel approach to conversational agent evaluation using Persona-driven User Simulations based on Large Language Models (LLMs). Our methodology first uses LLMs to generate diverse customer personas, which are then used to configure a single LLM-based user simulator. This simulator evaluates SalesBot 2.0, a proactive conversational sales agent. We introduce a dataset of these personas, along with corresponding goals and conversation scenarios, enabling comprehensive testing across different customer types with varying assertiveness levels and precision of needs. Our evaluation framework assesses both the simulator’s adherence to persona instructions and the bot’s performance across multiple dimensions, combining human annotation with LLM-as-a-judge assessments using commercial and open-source models. Results demonstrate that our LLM-based simulator effectively emulates nuanced customer roles, and that cross-selling strategies can be implemented with minimal impact on customer satisfaction, varying by customer type.
Search
Fix author
Co-authors
- Justyna Gromada 1
- Alicja Kasicka 1
- Ewa Komkowska 1
- Lukasz Krajewski 1
- Natalia Krawczyk 1
- show all...