@inproceedings{han-zarriess-2019-sketch,
    title = "Sketch Me if You Can: Towards Generating Detailed Descriptions of Object Shape by Grounding in Images and Drawings",
    author = "Han, Ting  and
      Zarrie{\ss}, Sina",
    editor = "van Deemter, Kees  and
      Lin, Chenghua  and
      Takamura, Hiroya",
    booktitle = "Proceedings of the 12th International Conference on Natural Language Generation",
    month = oct # "–" # nov,
    year = "2019",
    address = "Tokyo, Japan",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/add-emnlp-2024-awards/W19-8618/",
    doi = "10.18653/v1/W19-8618",
    pages = "136--140",
    abstract = "A lot of recent work in Language {\&} Vision has looked at generating descriptions or referring expressions for objects in scenes of real-world images, though focusing mostly on relatively simple language like object names, color and location attributes (e.g., brown chair on the left). This paper presents work on Draw-and-Tell, a dataset of detailed descriptions for common objects in images where annotators have produced fine-grained attribute-centric expressions distinguishing a target object from a range of similar objects. Additionally, the dataset comes with hand-drawn sketches for each object. As Draw-and-Tell is medium-sized and contains a rich vocabulary, it constitutes an interesting challenge for CNN-LSTM architectures used in state-of-the-art image captioning models. We explore whether the additional modality given through sketches can help such a model to learn to accurately ground detailed language referring expressions to object shapes. Our results are encouraging."
}