Machine-in-the-Loop Rewriting for Creative Image Captioning

Vishakh Padmakumar, He He


Abstract
Machine-in-the-loop writing aims to build models that assist humans to accomplish their writing tasks more effectively. Prior work has found that providing users a machine-written draft or sentence-level continuations has limited success since the generated text tends to deviate from users’ intention. To allow the user to retain control over the content, we train a rewriting model that, when prompted, modifies specified spans of text within the user’s original draft to introduce descriptive and figurative elements in the text.We evaluate the model on its ability to collaborate with humans on the task of creative image captioning. On a user study through Amazon Mechanical Turk, our model is rated to be more helpful by users than a baseline infilling language model. In addition, third-party evaluation shows that users write more descriptive and figurative captions when collaborating with our model compared to completing the task alone.However, the improvement is not uniform across user groups: the model is more helpful to skilled users, which risks widening the gap between skilled and novice users, highlighting a need for careful, user-centric evaluation of interactive systems.
Anthology ID:
2022.naacl-main.42
Volume:
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Month:
July
Year:
2022
Address:
Seattle, United States
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
573–586
Language:
URL:
https://aclanthology.org/2022.naacl-main.42
DOI:
10.18653/v1/2022.naacl-main.42
Bibkey:
Cite (ACL):
Vishakh Padmakumar and He He. 2022. Machine-in-the-Loop Rewriting for Creative Image Captioning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 573–586, Seattle, United States. Association for Computational Linguistics.
Cite (Informal):
Machine-in-the-Loop Rewriting for Creative Image Captioning (Padmakumar & He, NAACL 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/2022.naacl-main.42.pdf
Video:
 https://preview.aclanthology.org/ingestion-script-update/2022.naacl-main.42.mp4
Code
 vishakhpk/mil-creative-captioning