Visualizing the Relationship Between Encoded Linguistic Information and Task Performance

Jiannan Xiang, Huayang Li, Defu Lian, Guoping Huang, Taro Watanabe, Lemao Liu

Abstract
Probing is popular to analyze whether linguistic information can be captured by a well-trained deep neural model, but it is hard to answer how the change of the encoded linguistic information will affect task performance. To this end, we study the dynamic relationship between the encoded linguistic information and task performance from the viewpoint of Pareto Optimality. Its key idea is to obtain a set of models which are Pareto-optimal in terms of both objectives. From this viewpoint, we propose a method to optimize the Pareto-optimal models by formalizing it as a multi-objective optimization problem. We conduct experiments on two popular NLP tasks, i.e., machine translation and language modeling, and investigate the relationship between several kinds of linguistic information and task performances. Experimental results demonstrate that the proposed method is better than a baseline method. Our empirical findings suggest that some syntactic information is helpful for NLP tasks whereas encoding more syntactic information does not necessarily lead to better performance, because the model architecture is also an important factor.
Anthology ID:
2022.findings-acl.35
Volume:
Findings of the Association for Computational Linguistics: ACL 2022
Month:
May
Year:
2022
Address:
Dublin, Ireland
Editors:
Smaranda Muresan, Preslav Nakov, Aline Villavicencio
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
410–422
Language:
URL:
https://aclanthology.org/2022.findings-acl.35
DOI:
10.18653/v1/2022.findings-acl.35
Bibkey:
Cite (ACL):
Jiannan Xiang, Huayang Li, Defu Lian, Guoping Huang, Taro Watanabe, and Lemao Liu. 2022. Visualizing the Relationship Between Encoded Linguistic Information and Task Performance. In Findings of the Association for Computational Linguistics: ACL 2022, pages 410–422, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
Visualizing the Relationship Between Encoded Linguistic Information and Task Performance (Xiang et al., Findings 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/naacl24-info/2022.findings-acl.35.pdf
PDF Search