From Explainability to Explanation: Using a Dialogue Setting to Elicit Annotations with Justifications

Nazia Attari, Martin Heckmann, David Schlangen

Abstract
Despite recent attempts in the field of explainable AI to go beyond black box prediction models, typically already the training data for supervised machine learning is collected in a manner that treats the annotator as a “black box”, the internal workings of which remains unobserved. We present an annotation method where a task is given to a pair of annotators who collaborate on finding the best response. With this we want to shed light on the questions if the collaboration increases the quality of the responses and if this “thinking together” provides useful information in itself, as it at least partially reveals their reasoning steps. Furthermore, we expect that this setting puts the focus on explanation as a linguistic act, vs. explainability as a property of models. In a crowd-sourcing experiment, we investigated three different annotation tasks, each in a collaborative dialogical (two annotators) and monological (one annotator) setting. Our results indicate that our experiment elicits collaboration and that this collaboration increases the response accuracy. We see large differences in the annotators’ behavior depending on the task. Similarly, we also observe that the dialog patterns emerging from the collaboration vary significantly with the task.
Anthology ID:
W19-5938
Volume:
Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue
Month:
September
Year:
2019
Address:
Stockholm, Sweden
Editors:
Satoshi Nakamura, Milica Gasic, Ingrid Zukerman, Gabriel Skantze, Mikio Nakano, Alexandros Papangelis, Stefan Ultes, Koichiro Yoshino
Venue:
SIGDIAL
SIG:
SIGDIAL
Publisher:
Association for Computational Linguistics
Note:
Pages:
331–335
Language:
URL:
https://aclanthology.org/W19-5938
DOI:
10.18653/v1/W19-5938
Bibkey:
Cite (ACL):
Nazia Attari, Martin Heckmann, and David Schlangen. 2019. From Explainability to Explanation: Using a Dialogue Setting to Elicit Annotations with Justifications. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 331–335, Stockholm, Sweden. Association for Computational Linguistics.
Cite (Informal):
From Explainability to Explanation: Using a Dialogue Setting to Elicit Annotations with Justifications (Attari et al., SIGDIAL 2019)
Copy Citation:
PDF:
https://preview.aclanthology.org/add_acl24_videos/W19-5938.pdf
PDF Search