Abstract
Recent years have witnessed the tendency of neural encoding models on exploring brain language processing using naturalistic stimuli. Neural encoding models are data-driven methods that require an encoding model to investigate the mystery of brain mechanisms hidden in the data. As a data-driven method, the performance of encoding models is very sensitive to the experimental setting. However, it is unknown how the experimental setting further affects the conclusions of neural encoding models. This paper systematically investigated this problem and evaluated the influence of three experimental settings, i.e., the data size, the cross-validation training method, and the statistical testing method. Results demonstrate that inappropriate cross-validation training and small data size can substantially decrease the performance of encoding models, especially in the temporal lobe and the frontal lobe. And different null hypotheses in significance testing lead to highly different significant brain regions. Based on these results, we suggest a block-wise cross-validation training method and an adequate data size for increasing the performance of linear encoding models. We also propose two strict null hypotheses to control false positive discovery rates.- Anthology ID:
- 2022.lrec-1.687
- Volume:
- Proceedings of the Thirteenth Language Resources and Evaluation Conference
- Month:
- June
- Year:
- 2022
- Address:
- Marseille, France
- Editors:
- Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, Stelios Piperidis
- Venue:
- LREC
- SIG:
- Publisher:
- European Language Resources Association
- Note:
- Pages:
- 6397–6404
- Language:
- URL:
- https://aclanthology.org/2022.lrec-1.687
- DOI:
- Cite (ACL):
- Xiaohan Zhang, Shaonan Wang, and Chengqing Zong. 2022. How Does the Experimental Setting Affect the Conclusions of Neural Encoding Models?. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6397–6404, Marseille, France. European Language Resources Association.
- Cite (Informal):
- How Does the Experimental Setting Affect the Conclusions of Neural Encoding Models? (Zhang et al., LREC 2022)
- PDF:
- https://preview.aclanthology.org/ingest-2024-clasp/2022.lrec-1.687.pdf