Abstract
Regarding the problem of automatically generating paraphrases with modified styles or attributes, the difficulty lies in the lack of parallel corpora. Numerous advances have been proposed for the generation. However, significant problems remain with the auto-evaluation of style transfer tasks. Based on the summary of Pang and Gimpel (2018) and Mir et al. (2019), style transfer evaluations rely on three metrics: post-transfer style classification accuracy, content or semantic similarity, and naturalness or fluency. We elucidate the dangerous current state of style transfer auto-evaluation research. Moreover, we propose ways to aggregate the three metrics into one evaluator. This abstract aims to bring researchers to think about the future of style transfer and style transfer evaluation research.- Anthology ID:
- D19-5557
- Volume:
- Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019)
- Month:
- November
- Year:
- 2019
- Address:
- Hong Kong, China
- Venue:
- WNUT
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 444–445
- Language:
- URL:
- https://aclanthology.org/D19-5557
- DOI:
- 10.18653/v1/D19-5557
- Cite (ACL):
- Richard Yuanzhe Pang. 2019. Towards Actual (Not Operational) Textual Style Transfer Auto-Evaluation. In Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019), pages 444–445, Hong Kong, China. Association for Computational Linguistics.
- Cite (Informal):
- Towards Actual (Not Operational) Textual Style Transfer Auto-Evaluation (Pang, WNUT 2019)
- PDF:
- https://preview.aclanthology.org/nodalida-main-page/D19-5557.pdf