@inproceedings{ghosh-etal-2026-fine,
    title = "Fine-tuning vs. In-context Learning in Large Language Models: A Formal Language Learning Perspective",
    author = "Ghosh, Bishwamittra  and
      Das, Soumi  and
      Speicher, Till  and
      Wu, Qinyuan  and
      Khan, Mohammad Aflah  and
      Garg, Deepak  and
      Gummadi, Krishna P.  and
      Terzi, Evimaria",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.1932/",
    pages = "41651--41697",
    ISBN = "979-8-89176-390-6",
    abstract = "Large language models (LLMs) operate in two fundamental learning modes {--} fine-tuning (FT) and in-context learning (ICL) {--} raising key questions about which mode yields greater language proficiency and whether they differ in their inductive biases. Prior studies comparing FT and ICL have yielded mixed and inconclusive results due to inconsistent experimental setups. To enable a rigorous comparison, we propose a formal language learning task {--} offering precise language boundaries, controlled string sampling, and no data contamination {--} and introduce a discriminative test for language proficiency, where an LLM succeeds if it assigns higher generation probability to in-language strings than to out-of-language strings.Empirically, we find that: (a) FT has greater language proficiency than ICL on in-distribution generalization, but both perform equally well on out-of-distribution generalization. (b) Their inductive biases, measured by the correlation in string generation probabilities, are similar when both modes partially learn the language but diverge at higher proficiency levels. (c) Unlike FT, ICL performance differs substantially across models of varying sizes and families and is sensitive to the token vocabulary of the language. Thus, our work demonstrates the promise of formal languages as a controlled testbed for evaluating LLMs, behaviors that are difficult to isolate in natural language datasets. Our source code is available at https://github.com/bishwamittra/formallm."
}