Item Response Theory for Natural Language Processing
John P. Lalor, Pedro Rodriguez, João Sedoc, Jose Hernandez-Orallo
Abstract
This tutorial will introduce the NLP community to Item Response Theory (IRT; Baker 2001). IRT is a method from the field of psychometrics for model and dataset assessment. IRT has been used for decades to build test sets for human subjects and estimate latent characteristics of dataset examples. Recently, there has been an uptick in work applying IRT to tasks in NLP. It is our goal to introduce the wider NLP community to IRT and show its benefits for a number of NLP tasks. From this tutorial, we hope to encourage wider adoption of IRT among NLP researchers.- Anthology ID:
- 2024.eacl-tutorials.2
- Volume:
- Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Tutorial Abstracts
- Month:
- March
- Year:
- 2024
- Address:
- St. Julian’s, Malta
- Editors:
- Mohsen Mesgar, Sharid Loáiciga
- Venue:
- EACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 9–13
- Language:
- URL:
- https://aclanthology.org/2024.eacl-tutorials.2
- DOI:
- Cite (ACL):
- John P. Lalor, Pedro Rodriguez, João Sedoc, and Jose Hernandez-Orallo. 2024. Item Response Theory for Natural Language Processing. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Tutorial Abstracts, pages 9–13, St. Julian’s, Malta. Association for Computational Linguistics.
- Cite (Informal):
- Item Response Theory for Natural Language Processing (Lalor et al., EACL 2024)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2024.eacl-tutorials.2.pdf