@inproceedings{chai-etal-2020-evaluation,
    title = "Evaluation of Coreference Resolution Systems Under Adversarial Attacks",
    author = "Chai, Haixia  and
      Zhao, Wei  and
      Eger, Steffen  and
      Strube, Michael",
    editor = "Braud, Chlo{\'e}  and
      Hardmeier, Christian  and
      Li, Junyi Jessy  and
      Louis, Annie  and
      Strube, Michael",
    booktitle = "Proceedings of the First Workshop on Computational Approaches to Discourse",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2020.codi-1.16/",
    doi = "10.18653/v1/2020.codi-1.16",
    pages = "154--159",
    abstract = "A substantial overlap of coreferent mentions in the CoNLL dataset magnifies the recent progress on coreference resolution. This is because the CoNLL benchmark fails to evaluate the ability of coreference resolvers that requires linking novel mentions unseen at train time. In this work, we create a new dataset based on CoNLL, which largely decreases mention overlaps in the entire dataset and exposes the limitations of published resolvers on two aspects{---}lexical inference ability and understanding of low-level orthographic noise. Our findings show (1) the requirements for embeddings, used in resolvers, and for coreference resolutions are, by design, in conflict and (2) adversarial approaches are sometimes not legitimate to mitigate the obstacles, as they may falsely introduce mention overlaps in adversarial training and test sets, thus giving an inflated impression for the improvements."
}