Abstract
We present a procedure for learning to ground symbols from a sequence of stimuli consisting of an arbitrarily complex noun phrase (e.g. “all but one green square above both red circles.”) and its designation in the visual scene. Our distinctive approach combines: a) lazy few-shot learning to relate open-class words like green and above to their visual percepts; and b) symbolic reasoning with closed-class word categories like quantifiers and negation. We use this combination to estimate new training examples for grounding symbols that occur within a noun phrase but aren’t designated by that noun phase (e.g, red in the above example), thereby potentially gaining data efficiency. We evaluate the approach in a visual reference resolution task, in which the learner starts out unaware of concepts that are part of the domain model and how they relate to visual percepts.- Anthology ID:
- 2022.naacl-main.358
- Volume:
- Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
- Month:
- July
- Year:
- 2022
- Address:
- Seattle, United States
- Editors:
- Marine Carpuat, Marie-Catherine de Marneffe, Ivan Vladimir Meza Ruiz
- Venue:
- NAACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 4863–4874
- Language:
- URL:
- https://aclanthology.org/2022.naacl-main.358
- DOI:
- 10.18653/v1/2022.naacl-main.358
- Cite (ACL):
- Rimvydas Rubavicius and Alex Lascarides. 2022. Interactive Symbol Grounding with Complex Referential Expressions. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4863–4874, Seattle, United States. Association for Computational Linguistics.
- Cite (Informal):
- Interactive Symbol Grounding with Complex Referential Expressions (Rubavicius & Lascarides, NAACL 2022)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-2/2022.naacl-main.358.pdf
- Code
- itl-ed/igre