Exploring Domain Robust Lightweight Reward Models based on Router Mechanism

Hyuk Namgoong, Jeesu Jung, Sangkeun Jung, YoonHyung Roh

Abstract
Recent advancements in large language models have heavily relied on the large reward model from reinforcement learning from human feedback for fine-tuning. However, the use of a single reward model across various domains may not always be optimal, often requiring retraining from scratch when new domain data is introduced. To address these challenges, we explore the utilization of small language models operating in a domain-specific manner based on router mechanisms. Our three approaches are: 1) utilize mixture of experts to form a single reward model by modularizing an internal router and experts, 2) employing external router to select the appropriate reward model from multiple domain-specific models, and 3) the framework reduces parameter size by loading reward models and router adapters onto a single small language model using adapters. Experimental validation underscores the effectiveness of our approach, demonstrating performance comparable to baseline methods while also reducing the total parameter size.
Anthology ID:
2024.findings-acl.511
Volume:
Findings of the Association for Computational Linguistics ACL 2024
Month:
August
Year:
2024
Address:
Bangkok, Thailand and virtual meeting
Editors:
Lun-Wei Ku, Andre Martins, Vivek Srikumar
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
8644–8652
Language:
URL:
https://aclanthology.org/2024.findings-acl.511
DOI:
10.18653/v1/2024.findings-acl.511
Bibkey:
Cite (ACL):
Hyuk Namgoong, Jeesu Jung, Sangkeun Jung, and YoonHyung Roh. 2024. Exploring Domain Robust Lightweight Reward Models based on Router Mechanism. In Findings of the Association for Computational Linguistics ACL 2024, pages 8644–8652, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
Cite (Informal):
Exploring Domain Robust Lightweight Reward Models based on Router Mechanism (Namgoong et al., Findings 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-5/2024.findings-acl.511.pdf
PDF Search