Human Bias in the Face of AI: Examining Human Judgment Against Text Labeled as AI Generated

Tiffany Zhu, Iain Weissburg, Kexun Zhang, William Yang Wang


Abstract
As Al advances in text generation, human trust in Al generated content remains constrained by biases that go beyond concerns of accuracy. This study explores how bias shapes the perception of AI versus human generated content. Through three experiments involving text rephrasing, news article summarization, and persuasive writing, we investigated how human raters respond to labeled and unlabeled content. While the raters could not differentiate the two types of texts in the blind test, they overwhelmingly favored content labeled as “Human Generated,” over those labeled “AI Generated,” by a preference score of over 30%. We observed the same pattern even when the labels were deliberately swapped. This human bias against AI has broader societal and cognitive implications, as it undervalues AI performance. This study highlights the limitations of human judgment in interacting with AI and offers a foundation for improving human-AI collaboration, especially in creative fields.
Anthology ID:
2025.findings-acl.1329
Volume:
Findings of the Association for Computational Linguistics: ACL 2025
Month:
July
Year:
2025
Address:
Vienna, Austria
Editors:
Wanxiang Che, Joyce Nabende, Ekaterina Shutova, Mohammad Taher Pilehvar
Venues:
Findings | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
25907–25914
Language:
URL:
https://preview.aclanthology.org/ingestion-acl-25/2025.findings-acl.1329/
DOI:
Bibkey:
Cite (ACL):
Tiffany Zhu, Iain Weissburg, Kexun Zhang, and William Yang Wang. 2025. Human Bias in the Face of AI: Examining Human Judgment Against Text Labeled as AI Generated. In Findings of the Association for Computational Linguistics: ACL 2025, pages 25907–25914, Vienna, Austria. Association for Computational Linguistics.
Cite (Informal):
Human Bias in the Face of AI: Examining Human Judgment Against Text Labeled as AI Generated (Zhu et al., Findings 2025)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-acl-25/2025.findings-acl.1329.pdf