Linguistic Differences between AI and Human Comments in Weibo: Detect AI-Generated Text through Stylometric Features

Ziqi Li, Qi Zhang


Abstract
"LLM-enhanced social robots (LLM-Bots) generate responses similar to human interactions and pose risks to social media platforms. Distinguishing AI-generated texts (AIGTs) from human-written content is important for mitigating these threats. However, current AIGT detection technologies face limitations in social media contexts, including inadequate performance on shorttexts, poor interpretability, and a reliance on synthetic datasets. To address these challenges, this study first constructs a social media dataset composed of 463,382 Weibo comments to capture real-world interactions between LLM-Bots and human users. Second, a stylo metric feature set tailored to Chinese social media is developed. We conduct a comparative analysis of these features to reveal linguistic differences between human-written and AI-generated comments. Third,we propose a lightweight stylo metric feature-based self-attention classifier (SFSC). This model achieves a strong F1-score of 91.8% for detecting AI-generated short comments in Chinese while maintaining low computational overhead. Additionally, we provide interpretable criteria for the SFSC in AIGT detection through feature importance analysis. This study advances detection forAI-generated short texts in Chinese social media."
Anthology ID:
2025.ccl-1.64
Volume:
Proceedings of the 24th China National Conference on Computational Linguistics (CCL 2025)
Month:
August
Year:
2025
Address:
Jinan, China
Editors:
Maosong Sun, Peiyong Duan, Zhiyuan Liu, Ruifeng Xu, Weiwei Sun
Venue:
CCL
SIG:
Publisher:
Chinese Information Processing Society of China
Note:
Pages:
842–851
Language:
URL:
https://preview.aclanthology.org/ingest-ccl/2025.ccl-1.64/
DOI:
Bibkey:
Cite (ACL):
Ziqi Li and Qi Zhang. 2025. Linguistic Differences between AI and Human Comments in Weibo: Detect AI-Generated Text through Stylometric Features. In Proceedings of the 24th China National Conference on Computational Linguistics (CCL 2025), pages 842–851, Jinan, China. Chinese Information Processing Society of China.
Cite (Informal):
Linguistic Differences between AI and Human Comments in Weibo: Detect AI-Generated Text through Stylometric Features (Li & Zhang, CCL 2025)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingest-ccl/2025.ccl-1.64.pdf