@inproceedings{kowsher-etal-2025-self,
    title = "Does Self-Attention Need Separate Weights in Transformers?",
    author = "Kowsher, Md  and
      Prottasha, Nusrat Jahan  and
      Yu, Chun-Nam  and
      Garibay, Ozlem  and
      Yousefi, Niloofar",
    editor = "Chen, Weizhu  and
      Yang, Yi  and
      Kachuee, Mohammad  and
      Fu, Xue-Yong",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: Industry Track)",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.naacl-industry.44/",
    pages = "535--543",
    ISBN = "979-8-89176-194-0",
    abstract = "Self-attention has revolutionized natural language processing by capturing long-range dependencies and improving context understanding. However, it comes with high computational costs and struggles with sequential data{'}s inherent directionality. This paper investigates and presents a simplified approach called ``shared weight self-attention,'' where a single weight matrix is used for Keys, Queries, and Values instead of separate matrices for each. This approach cuts training parameters by more than half and significantly reduces training time. Our method not only improves efficiency but also achieves strong performance on tasks from the GLUE benchmark, even outperforming the standard BERT baseline in handling noisy and out-of-domain data. Experimental results show a 66.53{\%} reduction in parameter size within the attention block and competitive accuracy improvements of 3.55{\%} and 0.89{\%} over symmetric and pairwise attention-based BERT models, respectively."
}