Frame-Based Annotation of Multimodal Corpora: Tracking (A)Synchronies in Meaning Construction

Frederico Belcavello, Marcelo Viridiano, Alexandre Diniz da Costa, Ely Edison da Silva Matos, Tiago Timponi Torrent

Abstract
Multimodal aspects of human communication are key in several applications of Natural Language Processing, such as Machine Translation and Natural Language Generation. Despite recent advances in integrating multimodality into Computational Linguistics, the merge between NLP and Computer Vision techniques is still timid, especially when it comes to providing fine-grained accounts for meaning construction. This paper reports on research aiming to determine appropriate methodology and develop a computational tool to annotate multimodal corpora according to a principled structured semantic representation of events, relations and entities: FrameNet. Taking a Brazilian television travel show as corpus, a pilot study was conducted to annotate the frames that are evoked by the audio and the ones that are evoked by visual elements. We also implemented a Multimodal Annotation tool which allows annotators to choose frames and locate frame elements both in the text and in the images, while keeping track of the time span in which those elements are active in each modality. Results suggest that adding a multimodal domain to the linguistic layer of annotation and analysis contributes both to enrich the kind of information that can be tagged in a corpus, and to enhance FrameNet as a model of linguistic cognition.
Anthology ID:
2020.framenet-1.4
Volume:
Proceedings of the International FrameNet Workshop 2020: Towards a Global, Multilingual FrameNet
Month:
May
Year:
2020
Address:
Marseille, France
Editors:
Tiago T. Torrent, Collin F. Baker, Oliver Czulo, Kyoko Ohara, Miriam R. L. Petruck
Venue:
Framenet
SIG:
Publisher:
European Language Resources Association
Note:
Pages:
23–30
Language:
English
URL:
https://aclanthology.org/2020.framenet-1.4
DOI:
Bibkey:
Cite (ACL):
Frederico Belcavello, Marcelo Viridiano, Alexandre Diniz da Costa, Ely Edison da Silva Matos, and Tiago Timponi Torrent. 2020. Frame-Based Annotation of Multimodal Corpora: Tracking (A)Synchronies in Meaning Construction. In Proceedings of the International FrameNet Workshop 2020: Towards a Global, Multilingual FrameNet, pages 23–30, Marseille, France. European Language Resources Association.
Cite (Informal):
Frame-Based Annotation of Multimodal Corpora: Tracking (A)Synchronies in Meaning Construction (Belcavello et al., Framenet 2020)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-4/2020.framenet-1.4.pdf
PDF Search