@inproceedings{santillan-cooper-etal-2025-synthetic,
    title = "Synthetic Data for Evaluation: Supporting {LLM}-as-a-Judge Workflows with {E}val{A}ssist",
    author = "Santill{\'a}n Cooper, Mart{\'i}n  and
      Ashktorab, Zahra  and
      Do, Hyo Jin  and
      Miehling, Erik  and
      Geyer, Werner  and
      Gajcin, Jasmina  and
      Daly, Elizabeth M.  and
      Pan, Qian  and
      Desmond, Michael",
    editor = {Habernal, Ivan  and
      Schulam, Peter  and
      Tiedemann, J{\"o}rg},
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-demos.1/",
    pages = "1--11",
    ISBN = "979-8-89176-334-0",
    abstract = "We present a synthetic data generation tool integrated into EvalAssist. EvalAssist is a web-based application designed to assist human-centered evaluation of language model outputs by allowing users to refine LLM-as-a-Judge evaluation criteria. The synthetic data generation tool in EvalAssist is tailored for evaluation contexts and informed by findings from user studies with AI practitioners. Participants identified key pain points in current workflows including circularity risks (where models are judged by criteria derived by themselves), compounded bias (amplification of biases across multiple stages of a pipeline), and poor support for edge cases, and expressed a strong preference for real-world grounding and fine-grained control. In response, our tool supports flexible prompting, RAG-based grounding, persona diversity, and iterative generation workflows. We also incorporate features for quality assurance and edge case discovery."
}