The Hidden Space of Transformer Language Adapters

Jesujoba Alabi, Marius Mosbach, Matan Eyal, Dietrich Klakow, Mor Geva

Abstract
We analyze the operation of transformer language adapters, which are small modules trained on top of a frozen language model to adapt its predictions to new target languages. We show that adapted predictions mostly evolve in the source language the model was trained on, while the target language becomes pronounced only in the very last layers of the model. Moreover, the adaptation process is gradual and distributed across layers, where it is possible to skip small groups of adapters without decreasing adaptation performance. Last, we show that adapters operate on top of the model’s frozen representation space while largely preserving its structure, rather than on an isolated subspace. Our findings provide a deeper view into the adaptation process of language models to new languages, showcasing the constraints imposed on it by the underlying model and introduces practical implications to enhance its efficiency.
Anthology ID:
2024.acl-long.356
Volume:
Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
August
Year:
2024
Address:
Bangkok, Thailand
Editors:
Lun-Wei Ku, Andre Martins, Vivek Srikumar
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
6588–6607
Language:
URL:
https://aclanthology.org/2024.acl-long.356
DOI:
10.18653/v1/2024.acl-long.356
Bibkey:
Cite (ACL):
Jesujoba Alabi, Marius Mosbach, Matan Eyal, Dietrich Klakow, and Mor Geva. 2024. The Hidden Space of Transformer Language Adapters. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6588–6607, Bangkok, Thailand. Association for Computational Linguistics.
Cite (Informal):
The Hidden Space of Transformer Language Adapters (Alabi et al., ACL 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/landing_page/2024.acl-long.356.pdf
PDF Search