Fine-grained Linguistic Evaluation of Question Answering Systems

Sarra El Ayari, Brigitte Grau, Anne-Laure Ligozat

[How to correct problems with metadata yourself]

Abstract
Question answering systems are complex systems using natural language processing. Some evaluation campaigns are organized to evaluate such systems in order to propose a classification of systems based on final results (number of correct answers). Nevertheless, teams need to evaluate more precisely the results obtained by their systems if they want to do a diagnostic evaluation. There are no tools or methods to do these evaluations systematically. We present REVISE, a tool for glass box evaluation based on diagnostic of question answering system results.
Anthology ID:
L10-1359
Volume:
Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)
Month:
May
Year:
2010
Address:
Valletta, Malta
Editors:
Nicoletta Calzolari, Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odijk, Stelios Piperidis, Mike Rosner, Daniel Tapias
Venue:
LREC
SIG:
Publisher:
European Language Resources Association (ELRA)
Note:
Pages:
Language:
URL:
http://www.lrec-conf.org/proceedings/lrec2010/pdf/520_Paper.pdf
DOI:
Bibkey:
Cite (ACL):
Sarra El Ayari, Brigitte Grau, and Anne-Laure Ligozat. 2010. Fine-grained Linguistic Evaluation of Question Answering Systems. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10), Valletta, Malta. European Language Resources Association (ELRA).
Cite (Informal):
Fine-grained Linguistic Evaluation of Question Answering Systems (Ayari et al., LREC 2010)
Copy Citation:
PDF:
http://www.lrec-conf.org/proceedings/lrec2010/pdf/520_Paper.pdf
PDF Search