Can I trust You? LLMs as conversational agents
Marc Döbler, Raghavendran Mahendravarman, Anna Moskvina, Nasrin Saef
Abstract
With the rising popularity of LLMs in the public sphere, they become more and more attractive as a tool for doing one’s own research without having to rely on search engines or specialized knowledge of a scientific field. But using LLMs as a source for factual information can lead one to fall prey to misinformation or hallucinations dreamed up by the model. In this paper we examine the gpt-4 LLM by simulating a large number of potential research queries and evaluate how many of the generated references are factually correct as well as existent.- Anthology ID:
- 2024.personalize-1.5
- Volume:
- Proceedings of the 1st Workshop on Personalization of Generative AI Systems (PERSONALIZE 2024)
- Month:
- March
- Year:
- 2024
- Address:
- St. Julians, Malta
- Editors:
- Ameet Deshpande, EunJeong Hwang, Vishvak Murahari, Joon Sung Park, Diyi Yang, Ashish Sabharwal, Karthik Narasimhan, Ashwin Kalyan
- Venues:
- PERSONALIZE | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 71–75
- Language:
- URL:
- https://aclanthology.org/2024.personalize-1.5
- DOI:
- Cite (ACL):
- Marc Döbler, Raghavendran Mahendravarman, Anna Moskvina, and Nasrin Saef. 2024. Can I trust You? LLMs as conversational agents. In Proceedings of the 1st Workshop on Personalization of Generative AI Systems (PERSONALIZE 2024), pages 71–75, St. Julians, Malta. Association for Computational Linguistics.
- Cite (Informal):
- Can I trust You? LLMs as conversational agents (Döbler et al., PERSONALIZE-WS 2024)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-2/2024.personalize-1.5.pdf