@inproceedings{tavares-etal-2025-language,
    title = "Language Models Can be Efficiently Steered via Minimal Embedding Layer Transformations",
    author = "Tavares, Diogo  and
      Semedo, David  and
      Rudnicky, Alexander  and
      Magalhaes, Joao",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.1170/",
    pages = "22960--22978",
    ISBN = "979-8-89176-332-6",
    abstract = "Large Language Models (LLMs) are increasingly costly to fine-tune due to their size, with embedding layers alone accounting for up to 20{\%} of model parameters. While Parameter-Efficient Fine-Tuning (PEFT) methods exist, they largely overlook the embedding layer. In this paper, we introduce TinyTE, a novel PEFT approach that steers model behavior via minimal translational transformations in the embedding space. TinyTE modifies input embeddings without altering hidden layers, achieving competitive performance while requiring approximately 0.0001{\%} of the parameters needed for full fine-tuning. Experiments across architectures provide a new lens for understanding the relationship between input representations and model behavior{---}revealing them to be more flexible at their foundation than previously thought."
}