Abstract
Meaning is context-dependent, but many properties of language (should) remain the same even if we transform the context. For example, sentiment or speaker properties should be the same in a translation and original of a text. We introduce language invariant properties: i.e., properties that should not change when we transform text, and how they can be used to quantitatively evaluate the robustness of transformation algorithms. Language invariant properties can be used to define novel benchmarks to evaluate text transformation methods. In our work we use translation and paraphrasing as examples, but our findings apply more broadly to any transformation. Our results indicate that many NLP transformations change properties. We additionally release a tool as a proof of concept to evaluate the invariance of transformation applications.- Anthology ID:
- 2022.nlppower-1.9
- Volume:
- Proceedings of NLP Power! The First Workshop on Efficient Benchmarking in NLP
- Month:
- May
- Year:
- 2022
- Address:
- Dublin, Ireland
- Editors:
- Tatiana Shavrina, Vladislav Mikhailov, Valentin Malykh, Ekaterina Artemova, Oleg Serikov, Vitaly Protasov
- Venue:
- nlppower
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 84–92
- Language:
- URL:
- https://aclanthology.org/2022.nlppower-1.9
- DOI:
- 10.18653/v1/2022.nlppower-1.9
- Cite (ACL):
- Federico Bianchi, Debora Nozza, and Dirk Hovy. 2022. Language Invariant Properties in Natural Language Processing. In Proceedings of NLP Power! The First Workshop on Efficient Benchmarking in NLP, pages 84–92, Dublin, Ireland. Association for Computational Linguistics.
- Cite (Informal):
- Language Invariant Properties in Natural Language Processing (Bianchi et al., nlppower 2022)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2022.nlppower-1.9.pdf
- Code
- milanlproc/language-invariant-properties
- Data
- HatEval