@inproceedings{nair-etal-2024-dera,
    title = "{DERA}: Enhancing Large Language Model Completions with Dialog-Enabled Resolving Agents",
    author = "Nair, Varun  and
      Schumacher, Elliot  and
      Tso, Geoffrey  and
      Kannan, Anitha",
    editor = "Naumann, Tristan  and
      Ben Abacha, Asma  and
      Bethard, Steven  and
      Roberts, Kirk  and
      Bitterman, Danielle",
    booktitle = "Proceedings of the 6th Clinical Natural Language Processing Workshop",
    month = jun,
    year = "2024",
    address = "Mexico City, Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2024.clinicalnlp-1.12/",
    doi = "10.18653/v1/2024.clinicalnlp-1.12",
    pages = "122--161",
    abstract = "Large language models (LLMs) have emerged as valuable tools for many natural language understanding tasks. In safety-critical applications such as healthcare, the utility of these models is governed by their ability to generate factually accurate and complete outputs. In this work, we present dialog-enabled resolving agents (DERA). DERA is a paradigm made possible by the increased conversational abilities of LLMs. It provides a simple, interpretable forum for models to communicate feedback and iteratively improve output. We frame our dialog as a discussion between two agent types {--} a Researcher, who processes information and identifies crucial problem components, and a Decider, who has the autonomy to integrate the Researcher{'}s information and makes judgments on the final output.We test DERA against three clinically-focused tasks, with GPT-4 serving as our LLM. DERA shows significant improvement over the base GPT-4 performance in both human expert preference evaluations and quantitative metrics for medical conversation summarization and care plan generation. In a new finding, we also show that GPT-4{'}s performance (70{\%}) on an open-ended version of the MedQA question-answering (QA) dataset (Jin 2021; USMLE) is well above the passing level (60{\%}), with DERA showing similar performance. We will release the open-ended MedQA dataset."
}