Probing the Uniquely Identifiable Linguistic Patterns of Conversational AI Agents

Iqra Zahid, Tharindu Madusanka, Riza Batista-Navarro, Youcheng Sun

Abstract
The proliferation of Conversational AI agents (CAAs) has emphasised the need to distinguish between human and machine-generated texts, with implications spanning digital forensics and cybersecurity. While prior research primarily focussed on distinguishing human from machine-generated text, our study takes a more refined approach by analysing different CAAs. We construct linguistic profiles for five CAAs, aiming to identify Uniquely Identifiable Linguistic Patterns (UILPs) for each model using authorship attribution techniques. Authorship attribution (AA) is the task of identifying the author of an unknown text from a pool of known authors. Our research seeks to answer crucial questions about the existence of UILPs in CAAs, the linguistic overlap between various text types generated by these models, and the feasibility of Authorship Attribution (AA) for CAAs based on UILPs. Promisingly, we are able to attribute CAAs based on their original texts with a weighted F1-score of 96.94%. Further, we are able to attribute CAAs according to their writing style (as specified by prompts), yielding a weighted F1-score of 95.84%, which sets the baseline for this task. By employing principal component analysis (PCA), we identify the top 100 most informative linguistic features for each CAA, achieving a weighted F1-score ranging from 86.04% to 97.93%, and an overall weighted F1-score of 93.86%.
Anthology ID:
2024.findings-acl.274
Volume:
Findings of the Association for Computational Linguistics ACL 2024
Month:
August
Year:
2024
Address:
Bangkok, Thailand and virtual meeting
Editors:
Lun-Wei Ku, Andre Martins, Vivek Srikumar
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4612–4628
Language:
URL:
https://aclanthology.org/2024.findings-acl.274
DOI:
Bibkey:
Cite (ACL):
Iqra Zahid, Tharindu Madusanka, Riza Batista-Navarro, and Youcheng Sun. 2024. Probing the Uniquely Identifiable Linguistic Patterns of Conversational AI Agents. In Findings of the Association for Computational Linguistics ACL 2024, pages 4612–4628, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
Cite (Informal):
Probing the Uniquely Identifiable Linguistic Patterns of Conversational AI Agents (Zahid et al., Findings 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-4/2024.findings-acl.274.pdf
PDF Search