@inproceedings{qiu-etal-2025-emoagent,
    title = "{E}mo{A}gent: Assessing and Safeguarding Human-{AI} Interaction for Mental Health Safety",
    author = "Qiu, Jiahao  and
      He, Yinghui  and
      Juan, Xinzhe  and
      Wang, Yimin  and
      Liu, Yuhan  and
      Yao, Zixin  and
      Wu, Yue  and
      Jiang, Xun  and
      Yang, Ling  and
      Wang, Mengdi",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.594/",
    pages = "11752--11767",
    ISBN = "979-8-89176-332-6",
    abstract = "The rise of LLM-driven AI characters raises safety concerns, particularly for vulnerable human users with psychological disorders. To address these risks, we propose EmoAgent, a multi-agent AI framework designed to evaluate and mitigate mental health hazards in human-AI interactions. EmoAgent comprises two components: **EmoEval** simulates virtual users, including those portraying mentally vulnerable individuals, to assess mental health changes before and after interactions with AI characters. It uses clinically proven psychological and psychiatric assessment tools (PHQ-9, PDI, PANSS) to evaluate mental risks induced by LLM. **EmoGuard** serves as an intermediary, monitoring users' mental status, predicting potential harm, and providing corrective feedback to mitigate risks. Experiments conducted in popular character-based chatbots show that emotionally engaging dialogues can lead to psychological deterioration in vulnerable users, with mental state deterioration in more than 34.4{\%} of the simulations. EmoGuard significantly reduces these deterioration rates, underscoring its role in ensuring safer AI-human interactions."
}