Greyson Kim


2021

pdf
MONAH: Multi-Modal Narratives for Humans to analyze conversations
Joshua Y. Kim | Kalina Yacef | Greyson Kim | Chunfeng Liu | Rafael Calvo | Silas Taylor
Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume

In conversational analyses, humans manually weave multimodal information into the transcripts, which is significantly time-consuming. We introduce a system that automatically expands the verbatim transcripts of video-recorded conversations using multimodal data streams. This system uses a set of preprocessing rules to weave multimodal annotations into the verbatim transcripts and promote interpretability. Our feature engineering contributions are two-fold: firstly, we identify the range of multimodal features relevant to detect rapport-building; secondly, we expand the range of multimodal annotations and show that the expansion leads to statistically significant improvements in detecting rapport-building.