@inproceedings{thomas-etal-2025-beyond,
    title = "Beyond Agreement: Rethinking Ground Truth in Educational {AI} Annotation",
    author = "Thomas, Danielle R  and
      Borchers, Conrad  and
      Koedinger, Ken",
    editor = "Wilson, Joshua  and
      Ormerod, Christopher  and
      Beiting Parrish, Magdalen",
    booktitle = "Proceedings of the Artificial Intelligence in Measurement and Education Conference (AIME-Con): Full Papers",
    month = oct,
    year = "2025",
    address = "Wyndham Grand Pittsburgh, Downtown, Pittsburgh, Pennsylvania, United States",
    publisher = "National Council on Measurement in Education (NCME)",
    url = "https://preview.aclanthology.org/name-variant-enfa-fane/2025.aimecon-main.37/",
    pages = "345--351",
    ISBN = "979-8-218-84228-4",
    abstract = "Humans are biased, inconsistent, and yet we keep trusting them to define ``ground truth.'' This paper questions the overreliance on inter-rater reliability in educational AI and proposes a multidimensional approach leveraging expert-based approaches and close-the-loop validity to build annotations that reflect impact, not just agreement. It{'}s time we do better."
}