Can Brain Signals Reveal Inner Alignment with Human Languages?
Jielin Qiu, William Han, Jiacheng Zhu, Mengdi Xu, Douglas Weber, Bo Li, Ding Zhao
Abstract
Brain Signals, such as Electroencephalography (EEG), and human languages have been widely explored independently for many downstream tasks, however, the connection between them has not been well explored. In this study, we explore the relationship and dependency between EEG and language. To study at the representation level, we introduced MTAM, a Multimodal Transformer Alignment Model, to observe coordinated representations between the two modalities. We used various relationship alignment-seeking techniques, such as Canonical Correlation Analysis and Wasserstein Distance, as loss functions to transfigure features. On downstream applications, sentiment analysis and relation detection, we achieved new state-of-the-art results on two datasets, ZuCo and K-EmoCon. Our method achieved an F1-score improvement of 1.7% on K-EmoCon and 9.3% on Zuco datasets for sentiment analysis, and 7.4% on ZuCo for relation detection. In addition, we provide interpretations of the performance improvement: (1) feature distribution shows the effectiveness of the alignment module for discovering and encoding the relationship between EEG and language; (2) alignment weights show the influence of different language semantics as well as EEG frequency features; (3) brain topographical maps provide an intuitive demonstration of the connectivity in the brain regions. Our code is available at https://github.com/Jason-Qiu/EEG_Language_Alignment.- Anthology ID:
- 2023.findings-emnlp.120
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2023
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 1789–1804
- Language:
- URL:
- https://aclanthology.org/2023.findings-emnlp.120
- DOI:
- 10.18653/v1/2023.findings-emnlp.120
- Cite (ACL):
- Jielin Qiu, William Han, Jiacheng Zhu, Mengdi Xu, Douglas Weber, Bo Li, and Ding Zhao. 2023. Can Brain Signals Reveal Inner Alignment with Human Languages?. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1789–1804, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- Can Brain Signals Reveal Inner Alignment with Human Languages? (Qiu et al., Findings 2023)
- PDF:
- https://preview.aclanthology.org/ingest-2024-clasp/2023.findings-emnlp.120.pdf