Abstract
The existing work on vision and language navigation mainly relies on navigation-related losses to establish the connection between vision and language modalities, neglecting aspects of helping the navigation agent build a deep understanding of the visual environment.In our work, we provide indirect supervision to the navigation agent through a hint generator that provides detailed visual descriptions.The hint generator assists the navigation agent in developing a global understanding of the visual environment. It directs the agent’s attention toward related navigation details, including the relevant sub-instruction, potential challenges in recognition and ambiguities in grounding, and the targeted viewpoint description. To train the hint generator, we construct a synthetic dataset based on landmarks in the instructions and visible and distinctive objects in the visual environment.We evaluate our method on the R2R and R4R datasets and achieve state-of-the-art on several metrics. The experimental results demonstrate that generating hints not only enhances the navigation performance but also helps improve the agent’s interpretability.- Anthology ID:
- 2024.findings-eacl.7
- Volume:
- Findings of the Association for Computational Linguistics: EACL 2024
- Month:
- March
- Year:
- 2024
- Address:
- St. Julian’s, Malta
- Editors:
- Yvette Graham, Matthew Purver
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 92–103
- Language:
- URL:
- https://aclanthology.org/2024.findings-eacl.7
- DOI:
- Cite (ACL):
- Yue Zhang, Quan Guo, and Parisa Kordjamshidi. 2024. NavHint: Vision and Language Navigation Agent with a Hint Generator. In Findings of the Association for Computational Linguistics: EACL 2024, pages 92–103, St. Julian’s, Malta. Association for Computational Linguistics.
- Cite (Informal):
- NavHint: Vision and Language Navigation Agent with a Hint Generator (Zhang et al., Findings 2024)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-2/2024.findings-eacl.7.pdf