Visual Recipe Flow: A Dataset for Learning Visual State Changes of Objects with Recipe Flows

Keisuke Shirai, Atsushi Hashimoto, Taichi Nishimura, Hirotaka Kameko, Shuhei Kurita, Yoshitaka Ushiku, Shinsuke Mori

Abstract
We present a new multimodal dataset called Visual Recipe Flow, which enables us to learn a cooking action result for each object in a recipe text. The dataset consists of object state changes and the workflow of the recipe text. The state change is represented as an image pair, while the workflow is represented as a recipe flow graph. We developed a web interface to reduce human annotation costs. The dataset allows us to try various applications, including multimodal information retrieval.
Anthology ID:
2022.coling-1.315
Volume:
Proceedings of the 29th International Conference on Computational Linguistics
Month:
October
Year:
2022
Address:
Gyeongju, Republic of Korea
Editors:
Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, Seung-Hoon Na
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
3570–3577
Language:
URL:
https://aclanthology.org/2022.coling-1.315
DOI:
Bibkey:
Cite (ACL):
Keisuke Shirai, Atsushi Hashimoto, Taichi Nishimura, Hirotaka Kameko, Shuhei Kurita, Yoshitaka Ushiku, and Shinsuke Mori. 2022. Visual Recipe Flow: A Dataset for Learning Visual State Changes of Objects with Recipe Flows. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3570–3577, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Cite (Informal):
Visual Recipe Flow: A Dataset for Learning Visual State Changes of Objects with Recipe Flows (Shirai et al., COLING 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/landing_page/2022.coling-1.315.pdf
PDF Search