@inproceedings{addlesee-2024-grounding,
    title = "Grounding {LLM}s to In-prompt Instructions: Reducing Hallucinations Caused by Static Pre-training Knowledge",
    author = "Addlesee, Angus",
    editor = "Dinkar, Tanvi  and
      Attanasio, Giuseppe  and
      Cercas Curry, Amanda  and
      Konstas, Ioannis  and
      Hovy, Dirk  and
      Rieser, Verena",
    booktitle = "Proceedings of Safety4ConvAI: The Third Workshop on Safety for Conversational AI @ LREC-COLING 2024",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://preview.aclanthology.org/ingest-emnlp/2024.safety4convai-1.1/",
    pages = "1--7",
    abstract = "When deploying LLMs in certain commercial or research settings, domain specific knowledge must be explicitly provided within the prompt. This in-prompt knowledge can conflict with an LLM{'}s static world knowledge learned at pre-training, causing model hallucination (see examples in Table 1). In safety-critical settings, like healthcare and finance, these hallucinations can harm vulnerable users. We have curated a QA corpus containing information that LLMs could not have seen at pre-training. Using our corpus, we have probed various LLMs, manipulating both the prompt and the knowledge representation. We have found that our `Jodie' prompt consistently improves the model{'}s textual grounding to the given knowledge, and in-turn the overall answer accuracy. This is true in both the healthcare and finance domains - improving accuracy by up to 28{\%} (mean: 12{\%}). We have also identified that hierarchical and direct node-property graph structures could lead to more interpretable and controllable systems that provide a natural language interface with real-time in-domain knowledge. Our corpus will enable further work on this critical challenge."
}