Effective questions in referential visual dialogue
Mauricio Mazuecos, Alberto Testoni, Raffaella Bernardi, Luciana Benotti
Abstract
An interesting challenge for situated dialogue systems is referential visual dialog: by asking questions, the system has to identify the referent to which the user refers to. Task success is the standard metric used to evaluate these systems. However, it does not consider how effective each question is, that is how much each question contributes to the goal. We propose a new metric, that measures question effectiveness. As a preliminary study, we report the new metric for state of the art publicly available models on GuessWhat?!. Surprisingly, successful dialogues do not have a higher percentage of effective questions than failed dialogues. This suggests that a system with high task success is not necessarily one that generates good questions.- Anthology ID:
- 2020.winlp-1.9
- Volume:
- Proceedings of the The Fourth Widening Natural Language Processing Workshop
- Month:
- July
- Year:
- 2020
- Address:
- Seattle, USA
- Venue:
- WiNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 31–35
- Language:
- URL:
- https://aclanthology.org/2020.winlp-1.9
- DOI:
- 10.18653/v1/2020.winlp-1.9
- Cite (ACL):
- Mauricio Mazuecos, Alberto Testoni, Raffaella Bernardi, and Luciana Benotti. 2020. Effective questions in referential visual dialogue. In Proceedings of the The Fourth Widening Natural Language Processing Workshop, pages 31–35, Seattle, USA. Association for Computational Linguistics.
- Cite (Informal):
- Effective questions in referential visual dialogue (Mazuecos et al., WiNLP 2020)