The coding and annotation of multimodal dialogue acts

Volha Petukhova, Harry Bunt


Abstract
Recent years have witnessed a growing interest in annotating linguistic data at the semantic level, including the annotation of dialogue corpus data. The annotation scheme developed as international standard for dialogue act annotation ISO 24617-2 is based on the DIT++ scheme (Bunt, 2006; 2009) which combines the multidimensional DIT scheme (Bunt, 1994) with concepts from DAMSL (Allen and Core , 1997) and various other schemes. This scheme is designed in a such way that it can be applied not only to spoken dialogue, as is the case for most of the previously defined dialogue annotation schemes, but also to multimodal dialogue. This paper describes how the ISO 24617-2 annotation scheme can be used, together with the DIT++ method of ‘multidimensional segmentation', to annotate nonverbal and multimodal dialogue behaviour. We analyse the fundamental distinction between (a) the coding of surface features; (b) form-related semantic classification; and (c) semantic annotation in terms of dialogue acts, supported by experimental studies of (a) and (b). We discuss examples of specification languages for representing the results of each of these activities, show how dialogue act annotations can be attached to XML representations of functional segments of multimodal data.
Anthology ID:
L12-1044
Volume:
Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)
Month:
May
Year:
2012
Address:
Istanbul, Turkey
Venue:
LREC
SIG:
Publisher:
European Language Resources Association (ELRA)
Note:
Pages:
1293–1300
Language:
URL:
http://www.lrec-conf.org/proceedings/lrec2012/pdf/180_Paper.pdf
DOI:
Bibkey:
Cite (ACL):
Volha Petukhova and Harry Bunt. 2012. The coding and annotation of multimodal dialogue acts. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), pages 1293–1300, Istanbul, Turkey. European Language Resources Association (ELRA).
Cite (Informal):
The coding and annotation of multimodal dialogue acts (Petukhova & Bunt, LREC 2012)
Copy Citation:
PDF:
http://www.lrec-conf.org/proceedings/lrec2012/pdf/180_Paper.pdf