@inproceedings{karim-etal-2025-beyond,
    title = "Beyond the Score: Uncertainty-Calibrated {LLM}s for Automated Essay Assessment",
    author = "Karim, Ahmed  and
      Wang, Qiao  and
      Yuan, Zheng",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.992/",
    pages = "19642--19647",
    ISBN = "979-8-89176-332-6",
    abstract = "Automated Essay Scoring (AES) systems now attain near{--}human agreement on some public benchmarks, yet real-world adoption{---}especially in high-stakes examinations{---}remains limited. A principal obstacle is that most models output a single score without any accompanying measure of confidence or explanation. We address this gap with conformal prediction, a distribution-free wrapper that equips any classifier with set-valued outputs enjoying formal coverage guarantees. Two open-source Large Language Models{---}Llama-3 8B and Qwen-2.5 3B{---}are fine-tuned on three diverse corpora (ASAP, TOEFL11, Cambridge-FCE) and calibrated at a 90{\%} risk level. Reliability is assessed with UAcc, an uncertainty-aware accuracy that rewards models for being both correct and concise. To our knowledge, this is the first work to combine conformal prediction and UAcc for essay scoring. The calibrated models consistently meet the coverage target while keeping prediction sets compact, indicating that open-source, mid-sized LLMs can already support teacher-in-the-loop AES; we discuss scaling and broader user studies as future work."
}