@inproceedings{lesci-etal-2025-causal,
    title = "Causal Estimation of Tokenisation Bias",
    author = "Lesci, Pietro  and
      Meister, Clara  and
      Hofmann, Thomas  and
      Vlachos, Andreas  and
      Pimentel, Tiago",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.1374/",
    pages = "28325--28340",
    ISBN = "979-8-89176-251-0",
    abstract = "Modern language models are typically trained over subword sequences, but ultimately define probabilities over character-strings. Ideally, the choice of the tokeniser{---}which maps character-strings to subwords{---}should not affect the probability assigned to the underlying character-string; in practice, it does. We define this mismatch as **tokenisation bias**. In this work, we quantify one particular type of tokenisation bias: the effect of including or not a subword (e.g., $\langle$ hello $\rangle$) in a tokeniser{'}s vocabulary on the probability a trained model assigns to the corresponding characters (i.e., ``hello''). Estimating this effect is challenging because each model is trained with only one tokeniser. We address this by framing tokenisation bias as a causal effect and estimating it using the regression discontinuity design. Specifically, we exploit the fact that tokenisation algorithms rank subwords and add the first $K$ to a tokeniser{'}s vocabulary, where $K$ is an arbitrary cutoff point. As such, we can estimate a causal effect by comparing similar subwords around this cutoff. Experimentally, we find that tokenisation consistently affects models' outputs across scales, vocabularies, and tokenisers. Notably, a subword{'}s presence in a small model{'}s vocabulary may increase its characters' probability by up to 17 times, highlighting tokenisation as a key design choice in language modelling."
}