@inproceedings{brown-etal-2025-adaptively,
    title = "Adaptively profiling models with task elicitation",
    author = "Brown, Davis  and
      Balehannina, Prithvi  and
      Jin, Helen  and
      Havaldar, Shreya  and
      Hassani, Hamed  and
      Wong, Eric",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.1270/",
    pages = "24996--25031",
    ISBN = "979-8-89176-332-6",
    abstract = "Language model evaluations often fail to characterize consequential failure modes, forcing experts to inspect outputs and build new benchmarks. We introduce task elicitation, a method that automatically builds new evaluations to profile model behavior. Task elicitation finds hundreds of natural-language tasks{---}an order of magnitude more than prior work{---}where frontier models exhibit systematic failures, in domains ranging from forecasting to online harassment. For example, we find that Sonnet 3.5 over-associates quantum computing and AGI and that o3-mini is prone to hallucination when fabrications are repeated in-context."
}