Adapting AutoARGUE for Automatic Report Evaluation under Missing Citation Annotations

Divrose Kaur, Jatin Bedi, Jasmeet Singh


Abstract
We adapt the AutoARGUE framework (Walden et al., 2026) for Task A.2 of RAG4Reports 2026, which requires ranking 57 report generation systems across 68 topics using automated evaluation. The RAGTIME-1 corpus poses a fundamental challenge: all nugget annotations use a no-reference-doc sentinel rather than ground-truth document citations, rendering the original citation-relevance gating inoperable. We address this with three adaptations: automatic sentinel detection with forced direct LLM-based nugget matching; a WEAK POSITIVE partial credit mechanism for sentences that correctly answer nuggets but lack attesting citations; and a report-level request alignment check. Our nugget_coverage_weighted metric achieves the highest topic-level Pearson correlation (r=0.599) of any non-coordinator submission, closely approaching the coordinator baseline (r=0.607).
Anthology ID:
2026.rag4reports-1.15
Volume:
Proceedings of the 1st Workshop on Multilingual Report Generation via Retrieval Augmented Generation (RAG4Reports 2026)
Month:
July
Year:
2026
Address:
San Diego, CA, USA
Editors:
Eugene Yang, Dawn Lawrie, Sean MacAvaney, James Mayfield, Luca Soldaini, Andrew Yates
Venues:
RAG4Reports | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
103–107
Language:
URL:
https://preview.aclanthology.org/ingest-acl-workshops/2026.rag4reports-1.15/
DOI:
Bibkey:
Cite (ACL):
Divrose Kaur, Jatin Bedi, and Jasmeet Singh. 2026. Adapting AutoARGUE for Automatic Report Evaluation under Missing Citation Annotations. In Proceedings of the 1st Workshop on Multilingual Report Generation via Retrieval Augmented Generation (RAG4Reports 2026), pages 103–107, San Diego, CA, USA. Association for Computational Linguistics.
Cite (Informal):
Adapting AutoARGUE for Automatic Report Evaluation under Missing Citation Annotations (Kaur et al., RAG4Reports 2026)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingest-acl-workshops/2026.rag4reports-1.15.pdf