Hard and Soft Evaluation of NLP models with BOOtSTrap SAmpling - BooStSa
Tommaso Fornaciari, Alexandra Uma, Massimo Poesio, Dirk Hovy
Abstract
Natural Language Processing (NLP) ‘s applied nature makes it necessary to select the most effective and robust models. Producing slightly higher performance is insufficient; we want to know whether this advantage will carry over to other data sets. Bootstrapped significance tests can indicate that ability. So while necessary, computing the significance of models’ performance differences has many levels of complexity. It can be tedious, especially when the experimental design has many conditions to compare and several runs of experiments. We present BooStSa, a tool that makes it easy to compute significance levels with the BOOtSTrap SAmpling procedure to evaluate models that predict not only standard hard labels but soft-labels (i.e., probability distributions over different classes) as well.- Anthology ID:
- 2022.acl-demo.12
- Volume:
- Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations
- Month:
- May
- Year:
- 2022
- Address:
- Dublin, Ireland
- Editors:
- Valerio Basile, Zornitsa Kozareva, Sanja Stajner
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 127–134
- Language:
- URL:
- https://aclanthology.org/2022.acl-demo.12
- DOI:
- 10.18653/v1/2022.acl-demo.12
- Cite (ACL):
- Tommaso Fornaciari, Alexandra Uma, Massimo Poesio, and Dirk Hovy. 2022. Hard and Soft Evaluation of NLP models with BOOtSTrap SAmpling - BooStSa. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 127–134, Dublin, Ireland. Association for Computational Linguistics.
- Cite (Informal):
- Hard and Soft Evaluation of NLP models with BOOtSTrap SAmpling - BooStSa (Fornaciari et al., ACL 2022)
- PDF:
- https://preview.aclanthology.org/naacl-24-ws-corrections/2022.acl-demo.12.pdf