@inproceedings{wang-etal-2025-measuring,
    title = "Measuring Risk of Bias in Biomedical Reports: The {R}o{BBR} Benchmark",
    author = "Wang, Jianyou  and
      Cao, Weili  and
      Bao, Longtian  and
      Zheng, Youze  and
      Pasternak, Gil  and
      Wang, Kaicheng  and
      Wang, Xiaoyue  and
      Paturi, Ramamohan  and
      Bergen, Leon",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.160/",
    pages = "3220--3248",
    ISBN = "979-8-89176-332-6",
    abstract = "Systems that answer questions by reviewing the scientific literature are becoming increasingly feasible. To draw reliable conclusions, these systems should take into account the quality of available evidence from different studies, placing more weight on studies that use a valid methodology. We present a benchmark for measuring the methodological strength of biomedical papers, drawing on the risk-of-bias framework used for systematic reviews. Derived from over 500 biomedical studies, the three benchmark tasks encompass expert reviewers' judgments of studies' research methodologies, including the assessments of risk of bias within these studies. The benchmark contains a human-validated annotation pipeline for fine-grained alignment of reviewers' judgments with research paper sentences. Our analyses show that large language models' reasoning and retrieval capabilities impact their effectiveness with risk-of-bias assessment. The dataset is available at https://github.com/RoBBR-Benchmark/RoBBR."
}