@inproceedings{suter-etal-2021-grounding,
    title = "Grounding Plural Phrases: Countering Evaluation Biases by Individuation",
    author = "Suter, Julia  and
      Parcalabescu, Letitia  and
      Frank, Anette",
    editor = "Xin  and
      Hu, Ronghang  and
      Hudson, Drew  and
      Fu, Tsu-Jui  and
      Rohrbach, Marcus  and
      Fried, Daniel",
    booktitle = "Proceedings of the Second Workshop on Advances in Language and Vision Research",
    month = jun,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/landing_page/2021.alvr-1.4/",
    doi = "10.18653/v1/2021.alvr-1.4",
    pages = "22--28",
    abstract = "Phrase grounding (PG) is a multimodal task that grounds language in images. PG systems are evaluated on well-known benchmarks, using Intersection over Union (IoU) as evaluation metric. This work highlights a disconcerting bias in the evaluation of grounded plural phrases, which arises from representing sets of objects as a union box covering all component bounding boxes, in conjunction with the IoU metric. We detect, analyze and quantify an evaluation bias in the grounding of plural phrases and define a novel metric, c-IoU, based on a union box{'}s component boxes. We experimentally show that our new metric greatly alleviates this bias and recommend using it for fairer evaluation of plural phrases in PG tasks."
}