Abstract
The rapid advancement of natural language processing, information retrieval (IR), computer vision, and other technologies has presented significant challenges in evaluating the performance of these systems. One of the main challenges is the scarcity of human-labeled data, which hinders the fair and accurate assessment of these systems. In this work, we specifically focus on evaluating IR systems with sparse labels, borrowing from recent research on evaluating computer vision tasks.taking inspiration from the success of using Fréchet Inception Distance (FID) in assessing text-to-image generation systems. We propose leveraging the Fréchet Distance to measure the distance between the distributions of relevant judged items and retrieved results. Our experimental results on MS MARCO V1 dataset and TREC Deep Learning Tracks query sets demonstrate the effectiveness of the Fréchet Distance as a metric for evaluating IR systems, particularly in settings where a few labels are available.This approach contributes to the advancement of evaluation methodologies in real-world scenarios such as the assessment of generative IR systems.- Anthology ID:
- 2024.eacl-long.26
- Volume:
- Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
- Month:
- March
- Year:
- 2024
- Address:
- St. Julian’s, Malta
- Editors:
- Yvette Graham, Matthew Purver
- Venue:
- EACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 420–431
- Language:
- URL:
- https://aclanthology.org/2024.eacl-long.26
- DOI:
- Cite (ACL):
- Negar Arabzadeh and Charles Clarke. 2024. Fréchet Distance for Offline Evaluation of Information Retrieval Systems with Sparse Labels. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 420–431, St. Julian’s, Malta. Association for Computational Linguistics.
- Cite (Informal):
- Fréchet Distance for Offline Evaluation of Information Retrieval Systems with Sparse Labels (Arabzadeh & Clarke, EACL 2024)
- PDF:
- https://preview.aclanthology.org/fix-volume-bibkeys/2024.eacl-long.26.pdf