@inproceedings{yuan-etal-2025-beyond-one,
    title = "Beyond One-Size-Fits-All: Tailored Benchmarks for Efficient Evaluation",
    author = "Yuan, Peiwen  and
      Zhang, Yueqi  and
      Feng, Shaoxiong  and
      Li, Yiwei  and
      Wang, Xinglin  and
      Shi, Jiayi  and
      Tan, Chuyi  and
      Pan, Boyuan  and
      Hu, Yao  and
      Li, Kan",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.759/",
    pages = "15591--15615",
    ISBN = "979-8-89176-251-0",
    abstract = "Evaluating models on large benchmarks can be very resource-intensive, especially during a period of rapid model evolution. Existing efficient evaluation methods estimate the performance of target models by testing them on a small, static coreset derived from the publicly available evaluation results of source models, which are separate from the target models. However, these approaches rely on the assumption that target models have high prediction consistency with source models, which doesn{'}t generalize well in practice. To fill this gap, we propose TailoredBench, a method that conducts customized evaluation tailored to each target model. Specifically, a Global-coreset is first constructed as a probe to identify the most consistent source models for each target model with an adaptive source model selection strategy. Afterwards, a scalable K-Medoids clustering algorithm is proposed to extend the Global-coreset to a tailored Native-coreset for each target model. According to the predictions on respective Native-coreset, we estimate the overall performance of target models with a calibrated estimation strategy. Comprehensive experiments on five benchmarks across over 300 models demonstrate that compared to best performing baselines, TailoredBench achieves an average reduction of 31.4{\%} in MAE of accuracy estimates under the same inference budgets, showcasing strong effectiveness and generalizability."
}