@inproceedings{zimmerman-etal-2024-two,
    title = "Two-tiered Encoder-based Hallucination Detection for Retrieval-Augmented Generation in the Wild",
    author = "Zimmerman, Ilana  and
      Tredup, Jadin  and
      Selfridge, Ethan  and
      Bradley, Joseph",
    editor = "Dernoncourt, Franck  and
      Preo\c tiuc-Pietro, Daniel  and
      Shimorina, Anastasia",
    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track",
    month = nov,
    year = "2024",
    address = "Miami, Florida, US",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/build-pipeline-with-new-library/2024.emnlp-industry.2/",
    doi = "10.18653/v1/2024.emnlp-industry.2",
    pages = "8--22",
    abstract = "Detecting hallucinations, where Large Language Models (LLMs) are not factually consistent with a Knowledge Base (KB), is a challenge for Retrieval-Augmented Generation (RAG) systems. Current solutions rely on public datasets to develop prompts or fine-tune a Natural Language Inference (NLI) model. However, these approaches are not focused on developing an enterprise RAG system; they do not consider latency, train or evaluate on production data, nor do they handle non-verifiable statements such as small talk or questions. To address this, we leverage the customer service conversation data of four large brands to evaluate existing solutions and propose a set of small encoder models trained on a new dataset. We find the proposed models to outperform existing methods and highlight the value of combining a small amount of in-domain data with public datasets."
}