@inproceedings{lee-etal-2025-mitigating,
    title = "Mitigating Attention Localization in Small Scale: Self-Attention Refinement via One-step Belief Propagation",
    author = "Lee, Nakyung  and
      Kim, Yeongoon  and
      Oh, Minhae  and
      Kim, Suhwan  and
      Koo, Jin Woo  and
      Jo, Hyewon  and
      Lee, Jungwoo",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.findings-emnlp.578/",
    doi = "10.18653/v1/2025.findings-emnlp.578",
    pages = "10897--10912",
    ISBN = "979-8-89176-335-7",
    abstract = "Transformer-based self-attention mechanism serves as the core of modern language models, yet it often suffers from *localization*, where attentions collapse onto a limited subset of tokens and fail to capture long-range dependencies. To address this issue, we propose **Self-Attention One-step Belief Propagation (SAOBP)**, a refinement framework that injects *multi-hop* relationships through a belief propagation process. To interpret and quantify these interactions, we introduce **Global Token Dependency (GTD)** that captures the relative contribution of multi-hop connections within the attention graph. Empirical results indicate that SAOBP helps prevent entropy collapse in deeper layers and adaptively maintains GTD at task-appropriate levels, thereby supporting improvements in model performance. Importantly, we observe competitive gains in small-scale models, highlighting its potential for improving inference quality in resource-constrained scenarios."
}