@inproceedings{kadlcik-etal-2025-pre,
    title = "Pre-trained Language Models Learn Remarkably Accurate Representations of Numbers",
    author = "Kadl{\v{c}}{\'i}k, Marek  and
      {\v{S}}tef{\'a}nik, Michal  and
      Mickus, Timothee  and
      Kucha{\v{r}}, Josef  and
      Spiegel, Michal",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.1356/",
    pages = "26693--26702",
    ISBN = "979-8-89176-332-6",
    abstract = "Pretrained language models (LMs) are prone to arithmetic errors. Existing work showed limited success in probing numeric values from models' representations, indicating that these errors can be attributed to the inherent unreliability of distributionally learned embeddings in representing exact quantities. However, we observe that previous probing methods are inadequate for the emergent structure of learned number embeddings with sinusoidal patterns.In response, we propose a novel probing technique that decodes numeric values from input embeddings with near-perfect accuracy across a range of open-source LMs. This proves that after the sole pre-training, LMs represent numbers with remarkable precision. Finally, we find that the embeddings' preciseness judged by our probe{'}s accuracy explains a large portion of LM{'}s errors in elementary arithmetic, and show that aligning the embeddings with the pattern discovered by our probe can mitigate these errors."
}