Testing Paraphrase Models on Recognising Sentence Pairs at Different Degrees of Semantic Overlap

Qiwei Peng, David Weir, Julie Weeds

Abstract
Paraphrase detection is useful in many natural language understanding applications. Current works typically formulate this problem as a sentence pair binary classification task. However, this setup is not a good fit for many of the intended applications of paraphrase models. In particular, such applications often involve finding the closest paraphrases of the target sentence from a group of candidate sentences where they exhibit different degrees of semantic overlap with the target sentence. To apply models to this paraphrase retrieval scenario, the model must be sensitive to the degree to which two sentences are paraphrases of one another. However, many existing datasets ignore and fail to test models in this setup. In response, we propose adversarial paradigms to create evaluation datasets, which could examine the sensitivity to different degrees of semantic overlap. Empirical results show that, while paraphrase models and different sentence encoders appear successful on standard evaluations, measuring the degree of semantic overlap still remains a big challenge for them.
Anthology ID:
2023.starsem-1.24
Volume:
Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023)
Month:
July
Year:
2023
Address:
Toronto, Canada
Editors:
Alexis Palmer, Jose Camacho-collados
Venue:
*SEM
SIG:
SIGLEX
Publisher:
Association for Computational Linguistics
Note:
Pages:
259–269
Language:
URL:
https://aclanthology.org/2023.starsem-1.24
DOI:
10.18653/v1/2023.starsem-1.24
Bibkey:
Cite (ACL):
Qiwei Peng, David Weir, and Julie Weeds. 2023. Testing Paraphrase Models on Recognising Sentence Pairs at Different Degrees of Semantic Overlap. In Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 259–269, Toronto, Canada. Association for Computational Linguistics.
Cite (Informal):
Testing Paraphrase Models on Recognising Sentence Pairs at Different Degrees of Semantic Overlap (Peng et al., *SEM 2023)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-4/2023.starsem-1.24.pdf
PDF Search