A Methodology for Evaluating Interaction Strategies of Task-Oriented Conversational Agents

Marco Guerini, Sara Falcone, Bernardo Magnini

Abstract
In task-oriented conversational agents, more attention has been usually devoted to assessing task effectiveness, rather than to how the task is achieved. However, conversational agents are moving towards more complex and human-like interaction capabilities (e.g. the ability to use a formal/informal register, to show an empathetic behavior), for which standard evaluation methodologies may not suffice. In this paper, we provide a novel methodology to assess - in a completely controlled way - the impact on the quality of experience of agent’s interaction strategies. The methodology is based on a within subject design, where two slightly different transcripts of the same interaction with a conversational agent are presented to the user. Through a series of pilot experiments we prove that this methodology allows fast and cheap experimentation/evaluation, focusing on aspects that are overlooked by current methods.
Anthology ID:
W18-5704
Volume:
Proceedings of the 2018 EMNLP Workshop SCAI: The 2nd International Workshop on Search-Oriented Conversational AI
Month:
October
Year:
2018
Address:
Brussels, Belgium
Editors:
Aleksandr Chuklin, Jeff Dalton, Julia Kiseleva, Alexey Borisov, Mikhail Burtsev
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
24–32
Language:
URL:
https://preview.aclanthology.org/build-pipeline-with-new-library/W18-5704/
DOI:
10.18653/v1/W18-5704
Bibkey:
Cite (ACL):
Marco Guerini, Sara Falcone, and Bernardo Magnini. 2018. A Methodology for Evaluating Interaction Strategies of Task-Oriented Conversational Agents. In Proceedings of the 2018 EMNLP Workshop SCAI: The 2nd International Workshop on Search-Oriented Conversational AI, pages 24–32, Brussels, Belgium. Association for Computational Linguistics.
Cite (Informal):
A Methodology for Evaluating Interaction Strategies of Task-Oriented Conversational Agents (Guerini et al., EMNLP 2018)
Copy Citation:
PDF:
https://preview.aclanthology.org/build-pipeline-with-new-library/W18-5704.pdf
PDF Search Fix metadata