Adjusting Image Attributes of Localized Regions with Low-level Dialogue

Tzu-Hsiang Lin, Alexander Rudnicky, Trung Bui, Doo Soon Kim, Jean Oh

Abstract
Natural Language Image Editing (NLIE) aims to use natural language instructions to edit images. Since novices are inexperienced with image editing techniques, their instructions are often ambiguous and contain high-level abstractions which require complex editing steps. Motivated by this inexperience aspect, we aim to smooth the learning curve by teaching the novices to edit images using low-level command terminologies. Towards this end, we develop a task-oriented dialogue system to investigate low-level instructions for NLIE. Our system grounds language on the level of edit operations, and suggests options for users to choose from. Though compelled to express in low-level terms, user evaluation shows that 25% of users found our system easy-to-use, resonating with our motivation. Analysis shows that users generally adapt to utilizing the proposed low-level language interface. We also identified object segmentation as the key factor to user satisfaction. Our work demonstrates advantages of low-level, direct language-action mapping approach that can be applied to other problem domains beyond image editing such as audio editing or industrial design.
Anthology ID:
2020.lrec-1.51
Volume:
Proceedings of the Twelfth Language Resources and Evaluation Conference
Month:
May
Year:
2020
Address:
Marseille, France
Venue:
LREC
SIG:
Publisher:
European Language Resources Association
Note:
Pages:
405–412
Language:
English
URL:
https://aclanthology.org/2020.lrec-1.51
DOI:
Bibkey:
Cite (ACL):
Tzu-Hsiang Lin, Alexander Rudnicky, Trung Bui, Doo Soon Kim, and Jean Oh. 2020. Adjusting Image Attributes of Localized Regions with Low-level Dialogue. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 405–412, Marseille, France. European Language Resources Association.
Cite (Informal):
Adjusting Image Attributes of Localized Regions with Low-level Dialogue (Lin et al., LREC 2020)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/2020.lrec-1.51.pdf
Code
 tzuhsial/ImageEditingWithDialogue
Data
COCO
PDF Search Code