Uniform Complexity for Text Generation

Joseph Marvin Imperial, Harish Tayyar Madabushi

Abstract
Large language models (LLMs) have shown promising results in a wide array of generative NLP tasks, such as summarization and machine translation. In the context of narrative generation, however, existing models still do not capture factors that contribute to producing consistent text. For instance, it is logical that a piece of text or a story should be uniformly readable throughout and that this form of complexity should be controllable. As such, if the complexity of an input text prompt is rated first-grade reading level in the Flesch Reading Ease test, then the generated text continuing the plot should also be within this range of complexity. With this in mind, we introduce Uniform Complexity for Text Generation (UCTG), a new benchmark test which raises the challenge of making generative models observe uniform linguistic properties with respect to prompts. We experiment with over 150+ linguistically and cognitively motivated features for evaluating text complexity in humans and generative models. From our results, we find that models such as GPT-2 struggle to preserve the complexity of input prompts used in its generations, even if finetuned with professionally written texts.
Anthology ID:
2023.findings-emnlp.805
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2023
Month:
December
Year:
2023
Address:
Singapore
Editors:
Houda Bouamor, Juan Pino, Kalika Bali
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
12025–12046
Language:
URL:
https://aclanthology.org/2023.findings-emnlp.805
DOI:
10.18653/v1/2023.findings-emnlp.805
Bibkey:
Cite (ACL):
Joseph Marvin Imperial and Harish Tayyar Madabushi. 2023. Uniform Complexity for Text Generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12025–12046, Singapore. Association for Computational Linguistics.
Cite (Informal):
Uniform Complexity for Text Generation (Imperial & Madabushi, Findings 2023)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-5/2023.findings-emnlp.805.pdf
PDF Search