Utterance-level Detection Framework for LLM-Involved Content Detection in Conversational Setting

Muyang Zhou, Huaxia Rui


Abstract
As Large Language Models(LLMs) increasingly power chatbots, social media, and other interactive platforms, the ability to detect AI in conversational settings is critical for ensuring transparency and preventing potential misuse. However, existing detection methods focus on static, document-level content, overlooking the dynamic nature of dialogues. To address this, we propose an utterance-level detection framework, which integrates features from individual and combined analysis of dialogue participants’ responses to detect LLM-generated text under conversational setting. Leveraging a transformer-based recurrent architecture and a curated dataset of human-human, human-LLM, and LLM-LLM dialogues, this framework achieves an accuracy of 98.14% with high inference speed, supported by extensive results of experiments on different models and settings. This work provides an effective solution for detecting LLM-generated text in real-time conversations, promoting transparency, and mitigating risks of misuse.
Anthology ID:
2026.eacl-long.63
Volume:
Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
March
Year:
2026
Address:
Rabat, Morocco
Editors:
Vera Demberg, Kentaro Inui, Lluís Marquez
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1350–1366
Language:
URL:
https://preview.aclanthology.org/ingest-eacl/2026.eacl-long.63/
DOI:
Bibkey:
Cite (ACL):
Muyang Zhou and Huaxia Rui. 2026. Utterance-level Detection Framework for LLM-Involved Content Detection in Conversational Setting. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1350–1366, Rabat, Morocco. Association for Computational Linguistics.
Cite (Informal):
Utterance-level Detection Framework for LLM-Involved Content Detection in Conversational Setting (Zhou & Rui, EACL 2026)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingest-eacl/2026.eacl-long.63.pdf