@inproceedings{yuchi-etal-2026-llms,
    title = "{LLM}s Know More About Numbers than They Can Say",
    author = "Yuchi, Fengting  and
      Du, Li  and
      Eisner, Jason",
    editor = "Demberg, Vera  and
      Inui, Kentaro  and
      Marquez, Llu{\'i}s",
    booktitle = "Proceedings of the 19th Conference of the {E}uropean Chapter of the {A}ssociation for {C}omputational {L}inguistics (Volume 2: Short Papers)",
    month = mar,
    year = "2026",
    address = "Rabat, Morocco",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-eacl/2026.eacl-short.47/",
    pages = "659--673",
    ISBN = "979-8-89176-381-4",
    abstract = "Although state-of-the-art LLMs can solve math problems, we find that they make errors on numerical comparisons with mixed notation: ``Which is larger, $5.7 \times 10^2$ or 580?{''}This raises a fundamental question: Do LLMs even know how big these numbers are?We probe the hidden states of several smaller open-source LLMs. A single linear projection of an appropriate hidden layer encodes the *log-magnitudes* of both kinds of numerals, allowing us to recover the numbers with relative error of about 2.3{\%} (on restricted synthetic text) or 19.06{\%} (on scientific papers).Furthermore, the hidden state after reading a *pair* of numerals encodes their *ranking*, with a linear classifier achieving over 90{\%} accuracy.Yet surprisingly, when explicitly asked to rank the same pairs of numerals, these LLMs achieve only 50-70{\%} accuracy, with worse performance for models whose probes are less effective.Finally, we show that incorporating the classifier probe{'}s log-loss as an auxiliary objective during finetuning brings an additional 3.22{\%} improvement in verbalized accuracy over base models, demonstrating that improving models' internal magnitude representations can enhance their numerical reasoning capabilities."
}