Scene Graph Modification Based on Natural Language Commands

Xuanli He, Quan Hung Tran, Gholamreza Haffari, Walter Chang, Zhe Lin, Trung Bui, Franck Dernoncourt, Nhan Dam

Abstract
Structured representations like graphs and parse trees play a crucial role in many Natural Language Processing systems. In recent years, the advancements in multi-turn user interfaces necessitate the need for controlling and updating these structured representations given new sources of information. Although there have been many efforts focusing on improving the performance of the parsers that map text to graphs or parse trees, very few have explored the problem of directly manipulating these representations. In this paper, we explore the novel problem of graph modification, where the systems need to learn how to update an existing scene graph given a new user’s command. Our novel models based on graph-based sparse transformer and cross attention information fusion outperform previous systems adapted from the machine translation and graph generation literature. We further contribute our large graph modification datasets to the research community to encourage future research for this new problem.
Anthology ID:
2020.findings-emnlp.87
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2020
Month:
November
Year:
2020
Address:
Online
Editors:
Trevor Cohn, Yulan He, Yang Liu
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
972–990
Language:
URL:
https://aclanthology.org/2020.findings-emnlp.87
DOI:
10.18653/v1/2020.findings-emnlp.87
Bibkey:
Cite (ACL):
Xuanli He, Quan Hung Tran, Gholamreza Haffari, Walter Chang, Zhe Lin, Trung Bui, Franck Dernoncourt, and Nhan Dam. 2020. Scene Graph Modification Based on Natural Language Commands. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 972–990, Online. Association for Computational Linguistics.
Cite (Informal):
Scene Graph Modification Based on Natural Language Commands (He et al., Findings 2020)
Copy Citation:
PDF:
https://preview.aclanthology.org/naacl24-info/2020.findings-emnlp.87.pdf
Code
 xlhex/SceneGraphModification
Data
MS COCO
PDF Search Code