Voronova Anastasia
2025
Hypercomplex Transformer: Novel Attention Mechanism
Maxim Gordeev
|
Zuev Aleksandr
|
Mikhail Bakulin
|
Andrey Latyshev
|
Dmitry Kozlov
|
Yiwu Yao
|
Voronova Anastasia
Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics
Self-attention mechanisms have become foundational across modern deep learning architectures. Recent efforts focus on improving their efficiency, particularly for signal processing tasks. The existing approaches employ complex-valued representations for inputs and weights and achieve higher accuracy at the cost of increased model size and inference latency. Dual-numbered algebra offers a promising alternative that allows efficient multiplication and faster inference with the same representational capacity. Inspired by previous studies in the field of hypercomplex neural networks, we introduce a generalized hypercomplex attention block and integrate it into Transformer-based models for EEG classification. Our experiments include adaptation of the hypercomplex models, so that the number of parameters is equal to that of their real-valued counterparts. Across all scenarios, the dual- and complex-numbered models consistently outperform the real ones, demonstrating superior accuracy. This work presents hypercomplex attention as a competitive and computationally efficient strategy with potential value to solve multiple NLP tasks.
Search
Fix author
Co-authors
- Zuev Aleksandr 1
- Mikhail Bakulin 1
- Maxim Gordeev 1
- Dmitry Kozlov 1
- Andrey Latyshev 1
- show all...
- Yiwu Yao 1