What Action Causes This? Towards Naive Physical Action-Effect Prediction

Qiaozi Gao, Shaohua Yang, Joyce Chai, Lucy Vanderwende


Abstract
Despite recent advances in knowledge representation, automated reasoning, and machine learning, artificial agents still lack the ability to understand basic action-effect relations regarding the physical world, for example, the action of cutting a cucumber most likely leads to the state where the cucumber is broken apart into smaller pieces. If artificial agents (e.g., robots) ever become our partners in joint tasks, it is critical to empower them with such action-effect understanding so that they can reason about the state of the world and plan for actions. Towards this goal, this paper introduces a new task on naive physical action-effect prediction, which addresses the relations between concrete actions (expressed in the form of verb-noun pairs) and their effects on the state of the physical world as depicted by images. We collected a dataset for this task and developed an approach that harnesses web image data through distant supervision to facilitate learning for action-effect prediction. Our empirical results have shown that web data can be used to complement a small number of seed examples (e.g., three examples for each action) for model learning. This opens up possibilities for agents to learn physical action-effect relations for tasks at hand through communication with humans with a few examples.
Anthology ID:
P18-1086
Volume:
Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
July
Year:
2018
Address:
Melbourne, Australia
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
934–945
Language:
URL:
https://aclanthology.org/P18-1086
DOI:
10.18653/v1/P18-1086
Bibkey:
Cite (ACL):
Qiaozi Gao, Shaohua Yang, Joyce Chai, and Lucy Vanderwende. 2018. What Action Causes This? Towards Naive Physical Action-Effect Prediction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 934–945, Melbourne, Australia. Association for Computational Linguistics.
Cite (Informal):
What Action Causes This? Towards Naive Physical Action-Effect Prediction (Gao et al., ACL 2018)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/P18-1086.pdf
Presentation:
 P18-1086.Presentation.pdf
Video:
 https://vimeo.com/285801281