SubmissionNumber#=%=#11 FinalPaperTitle#=%=#Understanding the Impact of Confidence in Retrieval Augmented Generation: A Case Study in the Medical Domain ShortPaperTitle#=%=# NumberOfPages#=%=#17 CopyrightSigned#=%=#Shintaro Ozaki JobTitle#==# Organization#==#8916-5, Takayama-cho, Ikoma-shi, Nara, JAPAN Abstract#==#Retrieval Augmented Generation (RAG) complements the knowledge of Large Language Models (LLMs) by leveraging external information to enhance response accuracy for queries. This approach is widely applied in several fields by taking its advantage of injecting the most up-to-date information, and researchers are focusing on understanding and improving this aspect to unlock the full potential of RAG in such high-stakes applications. However, despite the potential of RAG to address these needs, the mechanisms behind the confidence levels of its outputs remain underexplored. Our study focuses on the impact of RAG, specifically examining whether RAG increases the confidence of LLM outputs in the medical domain. We conduct this analysis across various configurations and models. We evaluate confidence by treating the model's predicted probability as its output and calculating several evaluation metrics which include calibration error method, entropy, best probability, and accuracy. Experimental results across multiple datasets confirmed that certain models possess the capability to judge for themselves whether an inserted document relates to the correct answer. These results suggest that evaluating models based on their output probabilities determine whether they function as generators in the RAG framework. Our approach allows to evaluate whether the models handle retrieved documents. Author{1}{Firstname}#=%=#Shintaro Author{1}{Lastname}#=%=#Ozaki Author{1}{Username}#=%=#shintaro Author{1}{Email}#=%=#ozaki.shintaro.ou6@naist.ac.jp Author{1}{Affiliation}#=%=#Nara Institute of Science and Technology Author{2}{Firstname}#=%=#Yuta Author{2}{Lastname}#=%=#Kato Author{2}{Username}#=%=#yuta-k Author{2}{Email}#=%=#ukato6209@g.ecc.u-tokyo.ac.jp Author{2}{Affiliation}#=%=#The University of Tokyo Author{3}{Firstname}#=%=#Siyuan Author{3}{Lastname}#=%=#Feng Author{3}{Username}#=%=#feng_siyuan Author{3}{Email}#=%=#9445233883@g.ecc.u-tokyo.ac.jp Author{3}{Affiliation}#=%=#The University of Tokyo, Department of Language and Information Sciences Author{4}{Firstname}#=%=#Masayo Author{4}{Lastname}#=%=#Tomita Author{4}{Username}#=%=#mtomita Author{4}{Email}#=%=#tomita-masayo732@g.ecc.u-tokyo.ac.jp Author{4}{Affiliation}#=%=#The University of Tokyo Author{5}{Firstname}#=%=#Kazuki Author{5}{Lastname}#=%=#Hayashi Author{5}{Username}#=%=#kazuki-ha Author{5}{Email}#=%=#hayashi.kazuki.hl4@is.naist.jp Author{5}{Affiliation}#=%=#Nara Institute of Science and Technology Author{6}{Firstname}#=%=#Wataru Author{6}{Lastname}#=%=#Hashimoto Author{6}{Username}#=%=#wataru.hashimoto Author{6}{Email}#=%=#hashimoto.wataru.hq3@is.naist.jp Author{6}{Affiliation}#=%=#Nara Institute of Science and Technology Author{7}{Firstname}#=%=#Ryoma Author{7}{Lastname}#=%=#Obara Author{7}{Username}#=%=#ryoma-obara Author{7}{Email}#=%=#ryoma-obara@nec.com Author{7}{Affiliation}#=%=#NEC Author{8}{Firstname}#=%=#Masafumi Author{8}{Lastname}#=%=#Oyamada Author{8}{Username}#=%=#stillpedant Author{8}{Email}#=%=#oyamada@nec.com Author{8}{Affiliation}#=%=#NEC Author{9}{Firstname}#=%=#Katsuhiko Author{9}{Lastname}#=%=#Hayashi Author{9}{Username}#=%=#katsuhiko-h Author{9}{Email}#=%=#katsuhiko-hayashi@g.ecc.u-tokyo.ac.jp Author{9}{Affiliation}#=%=#The University of Tokyo Author{10}{Firstname}#=%=#Hidetaka Author{10}{Lastname}#=%=#Kamigaito Author{10}{Username}#=%=#gaito Author{10}{Email}#=%=#kamigaito.h@is.naist.jp Author{10}{Affiliation}#=%=#Nara Institute of Science and Technology Author{11}{Firstname}#=%=#Taro Author{11}{Lastname}#=%=#Watanabe Author{11}{Username}#=%=#taro.watanabe Author{11}{Email}#=%=#taro@is.naist.jp Author{11}{Affiliation}#=%=#Nara Institute of Science and Technology ========== èéáğö