@inproceedings{maskey-etal-2025-benchmarking,
    title = "Benchmarking Large Language Models for Cryptanalysis and Side-Channel Vulnerabilities",
    author = "Maskey, Utsav  and
      Zhu, Chencheng  and
      Naseem, Usman",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.findings-emnlp.1082/",
    doi = "10.18653/v1/2025.findings-emnlp.1082",
    pages = "19849--19865",
    ISBN = "979-8-89176-335-7",
    abstract = "Recent advancements in Large Language Models (LLMs) have transformed natural language understanding and generation, leading to extensive benchmarking across diverse tasks. However, cryptanalysis{---}a critical area for data security and its connection to LLMs' generalization abilities remains underexplored in LLM evaluations. To address this gap, we evaluate the cryptanalytic potential of state{-}of{-}the{-}art LLMs on ciphertexts produced by a range of cryptographic algorithms. We introduce a benchmark dataset of diverse plaintexts{---}spanning multiple domains, lengths, writing styles, and topics{---}paired with their encrypted versions. Using zero{-}shot and few{-}shot settings along with chain{-}of{-}thought prompting, we assess LLMs' decryption success rate and discuss their comprehension abilities. Our findings reveal key insights into LLMs' strengths and limitations in side{-}channel scenarios and raise concerns about their susceptibility to under-generalization related attacks. This research highlights the dual{-}use nature of LLMs in security contexts and contributes to the ongoing discussion on AI safety and security."
}