Self-Improving Customer Review Response Generation Based on LLMs

Guy Azov, Tatiana Pelc, Adi Fledel Alon, Gila Kamhi

Abstract
Previous studies have demonstrated that proactive interaction with user reviews has a positive impact on the perception of app users and encourages them to submit revised ratings. Nevertheless, developers encounter challenges in managing a high volume of reviews, particularly in the case of popular apps with a substantial influx of daily reviews. Consequently, there is a demand for automated solutions aimed at streamlining the process of responding to user reviews. To address this, we have developed a new system for generating automatic responses by leveraging user-contributed documents with the help of retrieval-augmented generation (RAG) and advanced Large Language Models (LLMs). Our solution, named SCRABLE, represents an adaptive customer review response automation that enhances itself with self-optimizing prompts and a judging mechanism based on LLMs. Additionally, we introduce an automatic scoring mechanism that mimics the role of a human evaluator to assess the quality of responses generated in customer review domains. Extensive experiments and analyses conducted on real-world datasets reveal that our method is effective in producing high-quality responses, yielding improvement of more than 8.5% compared to the baseline. Further validation through manual examination of the generated responses underscores the efficacy our proposed system.
Anthology ID:
2024.ecnlp-1.5
Volume:
Proceedings of the Seventh Workshop on e-Commerce and NLP @ LREC-COLING 2024
Month:
May
Year:
2024
Address:
Torino, Italia
Editors:
Shervin Malmasi, Besnik Fetahu, Nicola Ueffing, Oleg Rokhlenko, Eugene Agichtein, Ido Guy
Venues:
ECNLP | WS
SIG:
Publisher:
ELRA and ICCL
Note:
Pages:
40–57
Language:
URL:
https://aclanthology.org/2024.ecnlp-1.5
DOI:
Bibkey:
Cite (ACL):
Guy Azov, Tatiana Pelc, Adi Fledel Alon, and Gila Kamhi. 2024. Self-Improving Customer Review Response Generation Based on LLMs. In Proceedings of the Seventh Workshop on e-Commerce and NLP @ LREC-COLING 2024, pages 40–57, Torino, Italia. ELRA and ICCL.
Cite (Informal):
Self-Improving Customer Review Response Generation Based on LLMs (Azov et al., ECNLP-WS 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/landing_page/2024.ecnlp-1.5.pdf
PDF Search