@inproceedings{chakravarty-etal-2025-enhancing,
    title = "Enhancing Marker Scoring Accuracy through Ordinal Confidence Modelling in Educational Assessments",
    author = "Chakravarty, Abhirup  and
      Brenchley, Mark  and
      Breakspear, Trevor  and
      Lewin, Ian  and
      Huang, Yan",
    editor = "Rehm, Georg  and
      Li, Yunyao",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/display_plenaries/2025.acl-industry.106/",
    pages = "1498--1507",
    ISBN = "979-8-89176-288-6",
    abstract = "A key ethical challenge in Automated Essay Scoring (AES) is ensuring that scores are only released when they meet high reliability standards. Confidence modelling addresses this by assigning a reliability estimate measure, in the form of a confidence score, to each automated score. In this study, we frame confidence estimation as a classification task: predicting whether an AES-generated score correctly places a candidate in the appropriate CEFR level. While this is a binary decision, we leverage the inherent granularity of the scoring domain in two ways. First, we reformulate the task as an $n$-ary classification problem using score binning. Second, we introduce a set of novel Kernel Weighted Ordinal Categorical Cross Entropy (KWOCCE) loss functions that incorporate the ordinal structure of CEFR labels. Our best-performing model achieves an F1 score of 0.97, and enables the system to release 47{\%} of scores with 100{\%} CEFR agreement and 99{\%} with at least 95{\%} CEFR agreement {---} compared to {\ensuremath{\approx}} 92 {\%} CEFR agreement from the standalone AES model where we release all AM predicted scores."
}