@inproceedings{li-etal-2026-crowdselect,
    title = "{C}rowd{S}elect: {S}ynthetic{I}nstruction Data Selection with Multi-{LLM} Wisdom",
    author = "Li, Yisen  and
      Yang, Lingfeng  and
      Shen, Wenxuan  and
      Zhou, Pan  and
      Wan, Yao  and
      Lin, Weiwei  and
      Chen, Dongping",
    editor = "Demberg, Vera  and
      Inui, Kentaro  and
      Marquez, Llu{\'i}s",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {EACL} 2026",
    month = mar,
    year = "2026",
    address = "Rabat, Morocco",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/manual-author-scripts/2026.findings-eacl.79/",
    pages = "1542--1569",
    ISBN = "979-8-89176-386-9",
    abstract = "Distilling advanced Large Language Models' instruction-following capabilities into smaller models using a selected subset has become a mainstream approach in model training. While existing synthetic instruction data selection strategies rely mainly on single-dimensional signals (i.e., reward scores, model perplexity), they fail to capture the complexity of instruction-following across diverse fields. Therefore, we investigate more diverse signals to capture comprehensive instruction-response pair characteristics and propose three foundational metrics that leverage Multi-LLMs wisdom, informed by (1) diverse LLM responses and (2) reward model assessment. Building upon base metrics, we propose CrowdSelect, an integrated metric incorporating a clustering-based approach to maintain response diversity. Our comprehensive experiments demonstrate that our foundation metrics consistently improve performance across 4 base models on MT-bench and Arena-Hard. CrowdSelect, efficiently incorporating all metrics, achieves state-of-the-art performance in both Full and LoRA fine-tuning, showing improvements of 4.81{\%} on Arena-Hard and 11.1{\%} on MT-bench with Llama-3.2-3b-instruct. We hope our findings will bring valuable insights for future research in this direction."
}