On Text Style Transfer via Style-Aware Masked Language Models
Sharan Narasimhan, Pooja H, Suvodip Dey, Maunendra Sankar Desarkar
Abstract
Text Style Transfer (TST) is performable through approaches such as latent space disentanglement, cycle-consistency losses, prototype editing etc. The prototype editing approach, which is known to be quite successful in TST, involves two key phases a) Masking of source style-associated tokens and b) Reconstruction of this source-style masked sentence conditioned with the target style. We follow a similar transduction method, in which we transpose the more difficult direct source to target TST task to a simpler Style-Masked Language Model (SMLM) Task, wherein, similar to BERT (CITATION), the goal of our model is now to reconstruct the source sentence from its style-masked version. We arrive at the SMLM mechanism naturally by formulating prototype editing/ transduction methods in a probabilistic framework, where TST resolves into estimating a hypothetical parallel dataset from a partially observed parallel dataset, wherein each domain is assumed to have a common latent style-masked prior. To generate this style-masked prior, we use “Explainable Attention” as our choice of attribution for a more precise style-masking step and also introduce a cost-effective and accurate “Attribution-Surplus” method of determining the position of masks from any arbitrary attribution model in O(1) time. We empirically show that this non-generational approach well suites the “content preserving” criteria for a task like TST, even for a complex style like Discourse Manipulation. Our model, the Style MLM, outperforms strong TST baselines and is on par with state-of-the-art TST models, which use complex architectures and orders of more parameters.- Anthology ID:
- 2023.inlg-main.25
- Volume:
- Proceedings of the 16th International Natural Language Generation Conference
- Month:
- September
- Year:
- 2023
- Address:
- Prague, Czechia
- Editors:
- C. Maria Keet, Hung-Yi Lee, Sina Zarrieß
- Venues:
- INLG | SIGDIAL
- SIG:
- SIGGEN
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 362–374
- Language:
- URL:
- https://aclanthology.org/2023.inlg-main.25
- DOI:
- 10.18653/v1/2023.inlg-main.25
- Cite (ACL):
- Sharan Narasimhan, Pooja H, Suvodip Dey, and Maunendra Sankar Desarkar. 2023. On Text Style Transfer via Style-Aware Masked Language Models. In Proceedings of the 16th International Natural Language Generation Conference, pages 362–374, Prague, Czechia. Association for Computational Linguistics.
- Cite (Informal):
- On Text Style Transfer via Style-Aware Masked Language Models (Narasimhan et al., INLG-SIGDIAL 2023)
- PDF:
- https://preview.aclanthology.org/fix-dup-bibkey/2023.inlg-main.25.pdf