@inproceedings{inaba-etal-2025-bilingual,
    title = "How a Bilingual {LM} Becomes Bilingual: Tracing Internal Representations with Sparse Autoencoders",
    author = "Inaba, Tatsuro  and
      Kamoda, Go  and
      Inui, Kentaro  and
      Isonuma, Masaru  and
      Miyao, Yusuke  and
      Oseki, Yohei  and
      Takagi, Yu  and
      Heinzerling, Benjamin",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.findings-emnlp.725/",
    doi = "10.18653/v1/2025.findings-emnlp.725",
    pages = "13458--13470",
    ISBN = "979-8-89176-335-7",
    abstract = "This study explores how bilingual language models develop complex internal representations.We employ sparse autoencoders to analyze internal representations of bilingual language models with a focus on the effects of training steps, layers, and model sizes.Our analysis shows that language models first learn languages separately, and then gradually form bilingual alignments, particularly in the mid layers. We also found that this bilingual tendency is stronger in larger models.Building on these findings, we demonstrate the critical role of bilingual representations in model performance by employing a novel method that integrates decomposed representations from a fully trained model into a mid-training model.Our results provide insights into how language models acquire bilingual capabilities."
}