Abstract
We evaluate the compositionality of general-purpose sentence encoders by proposing two different metrics to quantify compositional understanding capability of sentence encoders. We introduce a novel metric, Polarity Sensitivity Scoring (PSS), which utilizes sentiment perturbations as a proxy for measuring compositionality. We then compare results from PSS with those obtained via our proposed extension of a metric called Tree Reconstruction Error (TRE) (CITATION) where compositionality is evaluated by measuring how well a true representation producing model can be approximated by a model that explicitly combines representations of its primitives.- Anthology ID:
- 2020.repl4nlp-1.22
- Volume:
- Proceedings of the 5th Workshop on Representation Learning for NLP
- Month:
- July
- Year:
- 2020
- Address:
- Online
- Editors:
- Spandana Gella, Johannes Welbl, Marek Rei, Fabio Petroni, Patrick Lewis, Emma Strubell, Minjoon Seo, Hannaneh Hajishirzi
- Venue:
- RepL4NLP
- SIG:
- SIGREP
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 185–193
- Language:
- URL:
- https://aclanthology.org/2020.repl4nlp-1.22
- DOI:
- 10.18653/v1/2020.repl4nlp-1.22
- Cite (ACL):
- Hanoz Bhathena, Angelica Willis, and Nathan Dass. 2020. Evaluating Compositionality of Sentence Representation Models. In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 185–193, Online. Association for Computational Linguistics.
- Cite (Informal):
- Evaluating Compositionality of Sentence Representation Models (Bhathena et al., RepL4NLP 2020)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2020.repl4nlp-1.22.pdf