@inproceedings{singh-etal-2025-tfdp,
    title = "{TFDP}: Token-Efficient Disparity Audits for Autoregressive {LLM}s via Single-Token Masked Evaluation",
    author = "Singh, Inderjeet  and
      Srinivasan, Ramya  and
      Vainshtein, Roman  and
      Kojima, Hisashi",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.1250/",
    pages = "24609--24626",
    ISBN = "979-8-89176-332-6",
    abstract = "Auditing autoregressive Large Language Models (LLMs) for disparities is often impeded by high token costs and limited precision. We introduce Token-Focused Disparity Probing (TFDP), a novel methodology overcoming these challenges by adapting single-token masked prediction to autoregressive architectures via targeted token querying. Disparities between minimally contrastive sentence pairs are quantified through a multi-scale semantic alignment score that integrates sentence, local-context, and token embeddings with adaptive weighting. We propose three disparity metrics: Preference Score ($\mathcal{PS}$), Prediction Set Divergence ($\mathcal{PSD}$), and Weighted Final Score ($\mathcal{WFS}$), for comprehensive assessment. Evaluated on our customized Proverbs Disparity Dataset (PDD) with controlled attribute toggles (e.g., gender bias, misinformation susceptibility), TFDP precisely detects disparities while achieving up to 42 times fewer output tokens than minimal n-token continuations, offering a scalable tool for responsible LLM evaluation."
}