Abstract
We conducted a human subject study of named entity recognition on a noisy corpus of conversational music recommendation queries, with many irregular and novel named entities. We evaluated the human NER linguistic behaviour in these challenging conditions and compared it with the most common NER systems nowadays, fine-tuned transformers. Our goal was to learn about the task to guide the design of better evaluation methods and NER algorithms. The results showed that NER in our context was quite hard for both human and algorithms under a strict evaluation schema; humans had higher precision, while the model higher recall because of entity exposure especially during pre-training; and entity types had different error patterns (e.g. frequent typing errors for artists). The released corpus goes beyond predefined frames of interaction and can support future work in conversational music recommendation.- Anthology ID:
- 2023.eacl-main.92
- Volume:
- Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics
- Month:
- May
- Year:
- 2023
- Address:
- Dubrovnik, Croatia
- Editors:
- Andreas Vlachos, Isabelle Augenstein
- Venue:
- EACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 1281–1296
- Language:
- URL:
- https://aclanthology.org/2023.eacl-main.92
- DOI:
- 10.18653/v1/2023.eacl-main.92
- Award:
- EACL Outstanding Paper
- Cite (ACL):
- Elena Epure and Romain Hennequin. 2023. A Human Subject Study of Named Entity Recognition in Conversational Music Recommendation Queries. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1281–1296, Dubrovnik, Croatia. Association for Computational Linguistics.
- Cite (Informal):
- A Human Subject Study of Named Entity Recognition in Conversational Music Recommendation Queries (Epure & Hennequin, EACL 2023)
- PDF:
- https://preview.aclanthology.org/emnlp22-frontmatter/2023.eacl-main.92.pdf