Evaluating the Generalization Property of Prefix-based Methods for Data-to-text Generation
Clarine Vongpaseut, Alberto Lumbreras, Mike Gartrell, Patrick Gallinari
Abstract
Fine-tuning is the prevalent paradigm to adapt pre-trained language models to downstream tasks. Lightweight fine-tuning methods, such as prefix-tuning, only tune a small set of parameters which alleviates cost. Such methods were shown to achieve results similar to fine-tuning; however, performance can decrease when the inputs get farther from the training domain. Moreover, latest works questioned the efficiency of recent lightweight fine-tuning techniques depending on the task and the size of the model. In this paper, we propose to evaluate the generalization property of prefix-based methods depending on the size of the pre-trained language model in the multi-domain setting on data-to-text generation. We found that their performance depends heavily on the size of the model.- Anthology ID:
- 2023.jeptalnrecital-short.8
- Volume:
- Actes de CORIA-TALN 2023. Actes de la 30e Conférence sur le Traitement Automatique des Langues Naturelles (TALN), volume 2 : travaux de recherche originaux -- articles courts
- Month:
- 6
- Year:
- 2023
- Address:
- Paris, France
- Editors:
- Christophe Servan, Anne Vilnat
- Venue:
- JEP/TALN/RECITAL
- SIG:
- Publisher:
- ATALA
- Note:
- Pages:
- 73–81
- Language:
- URL:
- https://aclanthology.org/2023.jeptalnrecital-short.8
- DOI:
- Cite (ACL):
- Clarine Vongpaseut, Alberto Lumbreras, Mike Gartrell, and Patrick Gallinari. 2023. Evaluating the Generalization Property of Prefix-based Methods for Data-to-text Generation. In Actes de CORIA-TALN 2023. Actes de la 30e Conférence sur le Traitement Automatique des Langues Naturelles (TALN), volume 2 : travaux de recherche originaux -- articles courts, pages 73–81, Paris, France. ATALA.
- Cite (Informal):
- Evaluating the Generalization Property of Prefix-based Methods for Data-to-text Generation (Vongpaseut et al., JEP/TALN/RECITAL 2023)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2023.jeptalnrecital-short.8.pdf