@inproceedings{massoud-etal-2026-designing,
    title = "Designing Annotation Guidelines for Trait-Based {A}rabic Automated Essay Scoring: A Systematic Methodology",
    author = "Massoud, Walid  and
      Bouamor, Houda  and
      Hussein, Abdelrahman Abdel Latif  and
      Zekri, Abdullah Mohamed Mohamed",
    editor = "Liu, Yang Janet  and
      Gessler, Luke",
    booktitle = "Proceedings of the 20th Linguistic Annotation Workshop ({LAW} {XX})",
    month = jul,
    year = "2026",
    address = "San Diego, California, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl-workshops/2026.law-main.11/",
    pages = "146--157",
    ISBN = "979-8-89176-404-0",
    abstract = "Automated Essay Scoring (AES) fundamentally depends on high-quality annotated data, yet systematic approaches to developing annotation guidelines remain largely undocumented, especially for Arabic. We present a comprehensive methodology for trait-based Arabic AES annotation, applied to build a dataset of 7,859 essays by high school students annotated across seven writing traits, achieving substantial inter-annotator agreement (QWK: 0.66{--}0.75). Our methodology encompasses: (1) a seven-dimensional scoring framework grounded in Arabic linguistic and rhetorical conventions; (2) over 25 pages of Arabic-language guidelines with terminology unification, text-type-specific scoring descriptors, and annotated student examples; (3) a multi-stage training protocol that raised annotator agreement before production began; and (4) quality assurance mechanisms, including dual annotation and supervisor adjudication. We release all materials publicly, providing both a validated foundation for Arabic AES research and a replicable template for annotation guideline development in other morphologically complex, under-resourced languages."
}