Fiction-Writing Mode: An Effective Control for Human-Machine Collaborative Writing
Wenjie Zhong, Jason Naradowsky, Hiroya Takamura, Ichiro Kobayashi, Yusuke Miyao
Abstract
We explore the idea of incorporating concepts from writing skills curricula into human-machine collaborative writing scenarios, focusing on adding writing modes as a control for text generation models. Using crowd-sourced workers, we annotate a corpus of narrative text paragraphs with writing mode labels. Classifiers trained on this data achieve an average accuracy of ~87% on held-out data. We fine-tune a set of large language models to condition on writing mode labels, and show that the generated text is recognized as belonging to the specified mode with high accuracy. To study the ability of writing modes to provide fine-grained control over generated text, we devise a novel turn-based text reconstruction game to evaluate the difference between the generated text and the author’s intention. We show that authors prefer text suggestions made by writing mode-controlled models on average 61.1% of the time, with satisfaction scores 0.5 higher on a 5-point ordinal scale. When evaluated by humans, stories generated via collaboration with writing mode-controlled models achieve high similarity with the professionally written target story. We conclude by identifying the most common mistakes found in the generated stories.- Anthology ID:
- 2023.eacl-main.128
- Volume:
- Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics
- Month:
- May
- Year:
- 2023
- Address:
- Dubrovnik, Croatia
- Editors:
- Andreas Vlachos, Isabelle Augenstein
- Venue:
- EACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 1752–1765
- Language:
- URL:
- https://preview.aclanthology.org/add_missing_videos/2023.eacl-main.128/
- DOI:
- 10.18653/v1/2023.eacl-main.128
- Cite (ACL):
- Wenjie Zhong, Jason Naradowsky, Hiroya Takamura, Ichiro Kobayashi, and Yusuke Miyao. 2023. Fiction-Writing Mode: An Effective Control for Human-Machine Collaborative Writing. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1752–1765, Dubrovnik, Croatia. Association for Computational Linguistics.
- Cite (Informal):
- Fiction-Writing Mode: An Effective Control for Human-Machine Collaborative Writing (Zhong et al., EACL 2023)
- PDF:
- https://preview.aclanthology.org/add_missing_videos/2023.eacl-main.128.pdf