@inproceedings{alves-etal-2023-steering,
    title = "Steering Large Language Models for Machine Translation with Finetuning and In-Context Learning",
    author = "Alves, Duarte M.  and
      Guerreiro, Nuno M.  and
      Alves, Jo{\~a}o  and
      Pombal, Jos{\'e}  and
      Rei, Ricardo  and
      de Souza, Jos{\'e} G. C.  and
      Colombo, Pierre  and
      Martins, Andr{\'e} F. T.",
    editor = "Bouamor, Houda  and
      Pino, Juan  and
      Bali, Kalika",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2023",
    month = dec,
    year = "2023",
    address = "Singapore",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix___bootstrap-utility-classes/2023.findings-emnlp.744/",
    doi = "10.18653/v1/2023.findings-emnlp.744",
    pages = "11127--11148",
    abstract = "Large language models (LLMs) are a promising avenue for machine translation (MT). However, current LLM-based MT systems are brittle: their effectiveness highly depends on the choice of few-shot examples and they often require extra post-processing due to overgeneration. Alternatives such as finetuning on translation instructions are computationally expensive and may weaken in-context learning capabilities, due to overspecialization. In this paper, we provide a closer look at this problem. We start by showing that adapter-based finetuning with LoRA matches the performance of traditional finetuning while reducing the number of training parameters by a factor of 50. This method also outperforms few-shot prompting and eliminates the need for post-processing or in-context examples. However, we show that finetuning generally degrades few-shot performance, hindering adaptation capabilities. Finally, to obtain the best of both worlds, we propose a simple approach that incorporates few-shot examples during finetuning. Experiments on 10 language pairs show that our proposed approach recovers the original few-shot capabilities while keeping the added benefits of finetuning."
}