@inproceedings{chaturvedi-etal-2024-nebula,
    title = "Nebula: A discourse aware {M}inecraft Builder",
    author = "Chaturvedi, Akshay  and
      Thompson, Kate  and
      Asher, Nicholas",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2024.findings-emnlp.374/",
    doi = "10.18653/v1/2024.findings-emnlp.374",
    pages = "6431--6443",
    abstract = "When engaging in collaborative tasks, humans efficiently exploit the semantic structure of a conversation to optimize verbal and nonverbal interactions. But in recent ``language to code'' or ``language to action'' models, this information is lacking. We show how incorporating the prior discourse and nonlinguistic context of a conversation situated in a nonlinguistic environment can improve the ``language to action'' component of such interactions. We finetune an LLM to predict actions based on prior context; our model, Nebula, doubles the net-action F1 score over the baseline on this task of Jayannavar et al. (2020). We also investigate our model{'}s ability to construct shapes and understand location descriptions using a synthetic dataset."
}