@inproceedings{guerini-etal-2018-methodology,
    title = "A Methodology for Evaluating Interaction Strategies of Task-Oriented Conversational Agents",
    author = "Guerini, Marco  and
      Falcone, Sara  and
      Magnini, Bernardo",
    editor = "Chuklin, Aleksandr  and
      Dalton, Jeff  and
      Kiseleva, Julia  and
      Borisov, Alexey  and
      Burtsev, Mikhail",
    booktitle = "Proceedings of the 2018 {EMNLP} Workshop {SCAI}: The 2nd International Workshop on Search-Oriented Conversational {AI}",
    month = oct,
    year = "2018",
    address = "Brussels, Belgium",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/jlcl-multiple-ingestion/W18-5704/",
    doi = "10.18653/v1/W18-5704",
    pages = "24--32",
    abstract = "In task-oriented conversational agents, more attention has been usually devoted to assessing task effectiveness, rather than to \textit{how} the task is achieved. However, conversational agents are moving towards more complex and human-like interaction capabilities (e.g. the ability to use a formal/informal register, to show an empathetic behavior), for which standard evaluation methodologies may not suffice. In this paper, we provide a novel methodology to assess - in a completely controlled way - the impact on the quality of experience of agent`s interaction strategies. The methodology is based on a within subject design, where two slightly different transcripts of the same interaction with a conversational agent are presented to the user. Through a series of pilot experiments we prove that this methodology allows fast and cheap experimentation/evaluation, focusing on aspects that are overlooked by current methods."
}