Procedural Text Generation from an Execution Video

Atsushi Ushiku, Hayato Hashimoto, Atsushi Hashimoto, Shinsuke Mori


Abstract
In recent years, there has been a surge of interest in automatically describing images or videos in a natural language. These descriptions are useful for image/video search, etc. In this paper, we focus on procedure execution videos, in which a human makes or repairs something and propose a method for generating procedural texts from them. Since video/text pairs available are limited in size, the direct application of end-to-end deep learning is not feasible. Thus we propose to train Faster R-CNN network for object recognition and LSTM for text generation and combine them at run time. We took pairs of recipe and cooking video, generated a recipe from a video, and compared it with the original recipe. The experimental results showed that our method can produce a recipe as accurate as the state-of-the-art scene descriptions.
Anthology ID:
I17-1033
Volume:
Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
Month:
November
Year:
2017
Address:
Taipei, Taiwan
Venue:
IJCNLP
SIG:
Publisher:
Asian Federation of Natural Language Processing
Note:
Pages:
326–335
Language:
URL:
https://aclanthology.org/I17-1033
DOI:
Bibkey:
Cite (ACL):
Atsushi Ushiku, Hayato Hashimoto, Atsushi Hashimoto, and Shinsuke Mori. 2017. Procedural Text Generation from an Execution Video. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 326–335, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Cite (Informal):
Procedural Text Generation from an Execution Video (Ushiku et al., IJCNLP 2017)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/I17-1033.pdf