@inproceedings{gu-etal-2026-scicustom,
    title = "{S}ci{C}ustom: A Framework for Custom Evaluation of Scientific Capabilities in Large Language Models",
    author = "Gu, Yiyang  and
      Yang, Junwei  and
      Luo, Junyu  and
      Yuan, Ye  and
      Feng, Bin  and
      Xia, Yingce  and
      Xie, Shufang  and
      Liu, Kaili  and
      Wu, Bohan  and
      Shi, Qi  and
      Li, Haoran  and
      Xiao, Beier  and
      Xiao, Zhiping  and
      Luo, Xiao  and
      Zhang, Weizhi  and
      Yu, Philip S.  and
      Liu, Zequn  and
      Zhang, Ming",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.2117/",
    pages = "45661--45678",
    ISBN = "979-8-89176-390-6",
    abstract = "Large language models (LLMs) are increasingly applied to scientific research, yet existing evaluations often fail to reflect the fine-grained capabilities required in practice. Most benchmarks are manually curated or domain-generic, limiting scalability and alignment with real scientific use cases. In this paper, we propose a new framework named SciCustom to address the problem. It enables the custom construction of benchmarks from large-scale scientific data to evaluate application-specific scientific capabilities in LLMs. SciCustom first organizes scientific knowledge into ontology-grounded knowledge units with controlled granularity and trains a tagger to map large-scale data instances into this knowledge space. Given a custom requirement, relevant knowledge units are identified via voting-based multi-model consensus. These units enable relevance-aware benchmark retrieval via binary search, followed by proxy subset selection and data-grounded benchmark generation for efficient evaluation. Experiments in chemistry and healthcare demonstrate that SciCustom reveals fine-grained differences in LLM scientific capabilities that standard benchmarks overlook, while requiring neither expert annotation nor synthetic question generation. This work provides a scalable and application-aware foundation for benchmarking scientific capabilities in LLMs."
}