@inproceedings{william-lithgow-serrano-etal-2025-causal,
    title = "Causal Understanding by {LLM}s: The Role of Uncertainty",
    author = "William Lithgow-Serrano, Oscar  and
      Kanjirangat, Vani  and
      Antonucci, Alessandro",
    editor = "Noidea, Noidea",
    booktitle = "Proceedings of the 2nd Workshop on Uncertainty-Aware NLP (UncertaiNLP 2025)",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.uncertainlp-main.19/",
    pages = "208--228",
    ISBN = "979-8-89176-349-4",
    abstract = "Recent papers show LLMs achieve near-random accuracy in causal relation classification, raising questions about whether such failures arise from limited pretraining exposure or deeper representational gaps. We investigate this under uncertainty-based evaluation, testing whether pretraining exposure to causal examples improves causal understanding using {\ensuremath{>}}18K PubMed sentences{---}half from The Pile corpus, half post-2024{---}across seven models (Pythia-1.4B/7B/12B, GPT-J-6B, Dolly-7B/12B, Qwen-7B). We analyze model behavior through: (i) causal classification, where the model identifies causal relationships in text, and (ii) verbatim memorization probing, where we assess whether the model prefers previously seen causal statements over their paraphrases. Models perform four-way classification (direct/conditional/correlational/no-relationship) and select between originals and their generated paraphrases. Results show almost identical accuracy on seen/unseen sentences (p{\ensuremath{>}}0.05), no memorization bias (24.8{\%} original selection), output distribution over the possible options almost flat {---} with entropic values near the maximum (1.35/1.39), confirming random guessing. Instruction-tuned models show severe miscalibration (Qwen: {\ensuremath{>}}95{\%} confidence, 32.8{\%} accuracy, ECE=0.49). Conditional relations induce highest entropy (+11{\%} vs direct). These findings suggest that failures in causal understanding arise from the lack of structured causal representation, rather than insufficient exposure to causal examples during pretraining."
}