@inproceedings{correia-etal-2025-analysis,
    title = "Analysis of Automated Document Relevance Annotation for Information Retrieval in Oil and Gas Industry",
    author = "Correia, Jo{\~a}o Vitor Mariano  and
      Bell, Murilo Missano  and
      Amorim, Jo{\~a}o Vitor Robiatti  and
      Queiroz, Jonas  and
      Pedronette, Daniel  and
      Guilherme, Ivan Rizzo  and
      Lima de Oliveira, Felipe",
    editor = "Potdar, Saloni  and
      Rojas-Barahona, Lina  and
      Montella, Sebastien",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track",
    month = nov,
    year = "2025",
    address = "Suzhou (China)",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-industry.132/",
    pages = "1878--1889",
    ISBN = "979-8-89176-333-3",
    abstract = "The lack of high-quality test collections challenges Information Retrieval (IR) in specialized domains. This work addresses this issue by comparing supervised classifiers against zero-shot Large Language Models (LLMs) for automated relevance annotation in the oil and gas industry, using human expert judgments as a benchmark. A supervised classifier, trained on limited expert data, outperforms LLMs, achieving an F1-score that surpasses even a second human annotator. The study also empirically confirms that LLMs are susceptible to unfairly prefer technologically similar retrieval systems. While LLMs lack precision in this context, a well-engineered classifier offers an accurate and practical path to scaling evaluation datasets within a human-in-the-loop framework that empowers, not replaces, human expertise."
}