@inproceedings{pawar-etal-2025-broken,
    title = "Broken Words, Broken Performance: Effect of Tokenization on Performance of {LLM}s",
    author = "Pawar, Sachin  and
      Apte, Manoj  and
      Jadhav, Kshitij  and
      Palshikar, Girish Keshav  and
      Ramrakhiyani, Nitin",
    editor = "Inui, Kentaro  and
      Sakti, Sakriani  and
      Wang, Haofen  and
      Wong, Derek F.  and
      Bhattacharyya, Pushpak  and
      Banerjee, Biplab  and
      Ekbal, Asif  and
      Chakraborty, Tanmoy  and
      Singh, Dhirendra Pratap",
    booktitle = "Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics",
    month = dec,
    year = "2025",
    address = "Mumbai, India",
    publisher = "The Asian Federation of Natural Language Processing and The Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-ijcnlp-aacl/2025.ijcnlp-short.31/",
    pages = "372--385",
    ISBN = "979-8-89176-299-2",
    abstract = "Tokenization is the first step in training any Large Language Model (LLM), where the text is split into a sequence of tokens as per the model{'}s fixed vocabulary. This tokenization in LLMs is different from the traditional tokenization in NLP where the text is split into a sequence of ``natural'' words. In LLMs, a natural word may also be broken into multiple tokens due to limited vocabulary size of the LLMs (e.g., Mistral{'}s tokenizer splits ``martial'' into ``mart'' and ``ial''). In this paper, we hypothesize that such breaking of natural words negatively impacts LLM performance on various NLP tasks. To quantify this effect, we propose a set of penalty functions that compute a tokenization penalty for a given text for a specific LLM, indicating how ``bad'' the tokenization is. We establish statistical significance of our hypothesis on multiple NLP tasks for a set of different LLMs."
}