Conversational assistants are becoming increasingly popular, including in healthcare, partly due to the availability and capabilities of Large Language Models. There is a need for controlled, probing evaluations with real stakeholders, which can highlight the advantages and disadvantages of more traditional architectures and those based on generative AI. We present a within-group user study to compare two versions of a conversational assistant that allows patients with heart failure to ask about the salt content in food. One version of the system was developed with a neurosymbolic architecture, and another is based on GPT. Our objective in evaluating the two dialogue systems was not only to compare task performance but also to gain insights from real stakeholders. Results indicate that the two systems complement each other, highlighting the promise of a hybrid approach that leverages the strengths of both systems.

We propose a dialogue system that enables heart failure patients to inquire about salt content in foods and help them monitor and reduce salt intake. Addressing the lack of specific datasets for food-based salt content inquiries, we develop a template-based conversational dataset. The dataset is structured to ask clarification questions to identify food items and their salt content. Our findings indicate that while fine-tuning transformer-based models on the dataset yields limited performance, the integration of Neuro-Symbolic Rules significantly enhances the system’s performance. Our experiments show that by integrating neuro-symbolic rules, our system achieves an improvement in joint goal accuracy of over 20% across different data sizes compared to naively fine-tuning transformer-based models.