@inproceedings{wang-etal-2025-pmpo,
    title = "{PMPO}: A Self-Optimizing Framework for Creating High-Fidelity Measurement Tools for Social Bias in Large Language Models",
    author = "Wang, Zeqiang  and
      Wang, Yuqi  and
      Wu, Xinyue  and
      Li, Chenxi  and
      Liu, Yiran  and
      Ge, Linghan  and
      Yu, Zhan  and
      Shi, Jiaxin  and
      De, Suparna",
    editor = "Inui, Kentaro  and
      Sakti, Sakriani  and
      Wang, Haofen  and
      Wong, Derek F.  and
      Bhattacharyya, Pushpak  and
      Banerjee, Biplab  and
      Ekbal, Asif  and
      Chakraborty, Tanmoy  and
      Singh, Dhirendra Pratap",
    booktitle = "Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics",
    month = dec,
    year = "2025",
    address = "Mumbai, India",
    publisher = "The Asian Federation of Natural Language Processing and The Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-ijcnlp-aacl/2025.ijcnlp-long.130/",
    pages = "2427--2440",
    ISBN = "979-8-89176-298-5",
    abstract = "The potential of Large Language Models (LLMs) as instruments for measuring social phenomena is constrained by the methodological limitations of current probing techniques. Prevailing methods rely on static, handcrafted probe sets whose quality is highly dependent on their authors' subjective expertise. This results in measurement tools with inconsistent statistical reliability that defy systematic optimization. Such an ``artisanal'' approach, akin to using an ``uneven ruler,'' undermines the scientific rigor of its findings and severely limits the applicability of LLMs in the social sciences. To elevate bias measurement from a craft to a science, we introduce the Psychometric-driven Probe Optimization (PMPO) framework. This framework treats a probe set as an optimizable scientific instrument and, for the first time, utilizes a Neural Genetic Algorithm that leverages a powerful LLM as a ``neural genetic operator.'' Through a hybrid strategy of gradient-guided mutation and creative rephrasing, PMPO automatically enhances the probe set{'}s reliability, sensitivity, and diversity. We first establish the external validity of our foundational measurement method (PLC), demonstrating a high correlation between its measurement of occupational gender bias and real-world U.S. Bureau of Labor Statistics data (average Pearson{'}s r=0.83, p{\ensuremath{<}}.001). Building on this, we show that the PMPO framework can elevate a standard probe set{'}s internal consistency (Cronbach{'}s Alpha) from 0.90 to an exceptional 0.96 within 10 generations. Critically, in a rigorous, double-blind ``Turing Test,'' probes evolved by PMPO from non-expert seeds were judged by sociology experts to have achieved a level of quality, sophistication, and nuance that is comparable to, and even indistinguishable from, those handcrafted by domain experts. This work provides a systematic pathway to upgrade LLM measurement tools from artisanal artifacts to automated scientific instruments, offering an unprecedented and trustworthy tool for AI safety auditing and computational social science."
}