@inproceedings{rubavicius-lascarides-2022-interactive,
    title = "Interactive Symbol Grounding with Complex Referential Expressions",
    author = "Rubavicius, Rimvydas  and
      Lascarides, Alex",
    editor = "Carpuat, Marine  and
      de Marneffe, Marie-Catherine  and
      Meza Ruiz, Ivan Vladimir",
    booktitle = "Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
    month = jul,
    year = "2022",
    address = "Seattle, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/landing_page/2022.naacl-main.358/",
    doi = "10.18653/v1/2022.naacl-main.358",
    pages = "4863--4874",
    abstract = "We present a procedure for learning to ground symbols from a sequence of stimuli consisting of an arbitrarily complex noun phrase (e.g. {\textquotedblleft}all but one green square above both red circles.{\textquotedblright}) and its designation in the visual scene. Our distinctive approach combines: a) lazy few-shot learning to relate open-class words like green and above to their visual percepts; and b) symbolic reasoning with closed-class word categories like quantifiers and negation. We use this combination to estimate new training examples for grounding symbols that occur \textit{within} a noun phrase but aren{'}t designated by that noun phase (e.g, red in the above example), thereby potentially gaining data efficiency. We evaluate the approach in a visual reference resolution task, in which the learner starts out unaware of concepts that are part of the domain model and how they relate to visual percepts."
}