@inproceedings{peinelt-etal-2019-aiming,
    title = "Aiming beyond the Obvious: Identifying Non-Obvious Cases in Semantic Similarity Datasets",
    author = "Peinelt, Nicole  and
      Liakata, Maria  and
      Nguyen, Dong",
    editor = "Korhonen, Anna  and
      Traum, David  and
      M{\`a}rquez, Llu{\'i}s",
    booktitle = "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics",
    month = jul,
    year = "2019",
    address = "Florence, Italy",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/iwcs-25-ingestion/P19-1268/",
    doi = "10.18653/v1/P19-1268",
    pages = "2792--2798",
    abstract = "Existing datasets for scoring text pairs in terms of semantic similarity contain instances whose resolution differs according to the degree of difficulty. This paper proposes to distinguish obvious from non-obvious text pairs based on superficial lexical overlap and ground-truth labels. We characterise existing datasets in terms of containing difficult cases and find that recently proposed models struggle to capture the non-obvious cases of semantic similarity. We describe metrics that emphasise cases of similarity which require more complex inference and propose that these are used for evaluating systems for semantic similarity."
}