@inproceedings{yao-etal-2025-understanding,
    title = "Understanding the Repeat Curse in Large Language Models from a Feature Perspective",
    author = "Yao, Junchi  and
      Yang, Shu  and
      Xu, Jianhua  and
      Hu, Lijie  and
      Li, Mengdi  and
      Wang, Di",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/display_plenaries/2025.findings-acl.406/",
    pages = "7787--7815",
    ISBN = "979-8-89176-256-5",
    abstract = "Large language models (LLMs) have made remarkable progress in various domains, yet they often suffer from repetitive text generation, a phenomenon we refer to as the ``Repeat Curse''. While previous studies have proposed decoding strategies to mitigate repetition, the underlying mechanism behind this issue remains insufficiently explored. In this work, we investigate the root causes of repetition in LLMs through the lens of mechanistic interpretability. Inspired by recent advances in Sparse Autoencoders (SAEs), which enable monosemantic feature extraction, we propose a novel approach{---}{''}Duplicatus Charm''{---}to induce and analyze the Repeat Curse. Our method systematically identifies ``Repetition Features'' -the key model activations responsible for generating repetitive outputs. First, we locate the layers most involved in repetition through logit analysis. Next, we extract and stimulate relevant features using SAE-based activation manipulation. To validate our approach, we construct a repetition dataset covering token and paragraph level repetitions and introduce an evaluation pipeline to quantify the influence of identified repetition features. Furthermore, by deactivating these features, we have effectively mitigated the Repeat Curse."
}