@inproceedings{tulchinskii-etal-2025-quantifying,
    title = "Quantifying Logical Consistency in Transformers via Query-Key Alignment",
    author = "Tulchinskii, Eduard  and
      Kushnareva, Laida  and
      Voznyuk, Anastasia  and
      Andriiainen, Andrei  and
      Piontkovskaya, Irina  and
      Burnaev, Evgeny  and
      Barannikov, Serguei",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.1785/",
    pages = "35184--35199",
    ISBN = "979-8-89176-332-6",
    abstract = "Large language models (LLMs) excel at many NLP tasks, yet their multi-step logical reasoning remains unreliable. Existing solutions such as Chain-of-Thought prompting generate intermediate steps but provide no internal check of their logical coherence. In this paper, we use the ``QK-score'', a lightweight metric based on query{--}key alignments within transformer attention heads, to evaluate the logical reasoning capabilities of LLMs. Our method automatically identifies attention heads that play a key role in distinguishing valid from invalid logical inferences, enabling efficient inference-time evaluation via a single forward pass. It reveals latent reasoning structure in LLMs and provides a scalable mechanistic alternative to ablation-based analysis. Across three benchmarks: ProntoQA-OOD, PARARULE-Plus, and MultiLogicEval, with models ranging from 1.5B to 70B parameters, the selected heads predict logical validity up to 14{\%} better than the model probabilities, and remain robust under distractors and increasing reasoning depth of $d\leq 6$."
}