Abstract
Style is an integral part of natural language. However, evaluation methods for style measures are rare, often task-specific and usually do not control for content. We propose the modular, fine-grained and content-controlled similarity-based STyle EvaLuation framework (STEL) to test the performance of any model that can compare two sentences on style. We illustrate STEL with two general dimensions of style (formal/informal and simple/complex) as well as two specific characteristics of style (contrac’tion and numb3r substitution). We find that BERT-based methods outperform simple versions of commonly used style measures like 3-grams, punctuation frequency and LIWC-based approaches. We invite the addition of further tasks and task instances to STEL and hope to facilitate the improvement of style-sensitive measures.- Anthology ID:
- 2021.emnlp-main.569
- Volume:
- Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
- Month:
- November
- Year:
- 2021
- Address:
- Online and Punta Cana, Dominican Republic
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 7109–7130
- Language:
- URL:
- https://aclanthology.org/2021.emnlp-main.569
- DOI:
- 10.18653/v1/2021.emnlp-main.569
- Cite (ACL):
- Anna Wegmann and Dong Nguyen. 2021. Does It Capture STEL? A Modular, Similarity-based Linguistic Style Evaluation Framework. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7109–7130, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
- Cite (Informal):
- Does It Capture STEL? A Modular, Similarity-based Linguistic Style Evaluation Framework (Wegmann & Nguyen, EMNLP 2021)
- PDF:
- https://preview.aclanthology.org/nodalida-main-page/2021.emnlp-main.569.pdf
- Code
- nlpsoc/stel