@inproceedings{smirnova-etal-2026-aggregate,
    title = "What Aggregate Scores Hide: Per-Rule Evaluation of {R}ussian Grammatical Error Correction",
    author = "Smirnova, Anna  and
      Kopan, Artyom  and
      Makeev, Vladislav  and
      Chernishev, George",
    editor = "Kochmar, Ekaterina  and
      Alhafni, Bashar  and
      Bann{\`o}, Stefano  and
      Bexte, Marie  and
      Burstein, Jill  and
      Horbach, Andrea  and
      Laarmann-Quante, Ronja  and
      Tack, Anais  and
      Yaneva, Victoria  and
      Yuan, Zheng",
    booktitle = "Proceedings of the 21st Workshop on Innovative Use of {NLP} for Building Educational Applications ({BEA} 2026)",
    month = jul,
    year = "2026",
    address = "San Diego, California, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl-workshops/2026.bea-1.32/",
    pages = "463--478",
    ISBN = "979-8-89176-409-5",
    abstract = "Russian grammar correction models can improveon aggregate benchmarkswhile getting worse at specific grammar rules.We show this through per-rule evaluationon a diagnostic benchmark of 48 prescriptive rules:finetuning on synthetic data improves overall F$_{0.5}$while driving subordinate-clause comma accuracyfrom 14{\%} to 1{\%}.The suppression is invisible under corpus-level metricsand undetectable with existing coarse, corpus-specific tagsets;it is recoverable only when diagnosed at rule granularity.To enable this analysis,we develop a 98-category error taxonomygrounded in Rozental{'}s reference grammarand SyntErr, an open-source synthetic data generatorwhose per-rule distribution is an explicit parameter,designed to support arbitrary rule sets and languages.Finetuning eight open models (0.8B{--}12B)on 39K synthetic examplesyields up to 75.3 F$_{0.5}$,approaching frontier API modelswith models small enough to run on device.We release the taxonomy, generator,per-rule evaluation data, and all training artifacts."
}