@inproceedings{aranberri-2026-register,
    title = "Register Sensitivity in Scalar {MT} Evaluation: Evidence from {S}panish{--}{B}asque Informal Discourse",
    author = "Aranberri, Nora",
    editor = "Ojha, Atul Kr.  and
      Sakti, Sakriani  and
      Soria, Claudia  and
      Melero, Maite  and
      McCrae, John P.  and
      Lignos, Constantine  and
      Liu, Chao-Hong  and
      Claramunt, German Rigau  and
      Rehm, Georg",
    booktitle = "Proceedings of the {SIGUL} 2026 Joint Workshop with {ELE}, {EURALI}, and {DCLRL}: Towards Inclusivity and Equality: Language Resources and Technologies for Under-Resourced and Endangered Languages",
    month = may,
    year = "2026",
    address = "Palma, Mallorca, Spain",
    publisher = "ELRA Language Resources Association (ELRA)",
    url = "https://preview.aclanthology.org/revision-workflow/2026.sigul-1.2/",
    doi = "10.63317/4ig5743iz2r3",
    pages = "19--32",
    abstract = "Automatic scalar metrics are widely used for machine translation (MT) evaluation, yet their behavior under sociolinguistic variation remains underexplored, particularly in under-resourced and minority-language contexts. We present a small, controlled empirical analysis of reference-based evaluation in Spanish{--}Basque informal discourse. Register is operationalized as indexical density, capturing dialectal forms, informal lexicon, code-switching and orthographic stylization. Across two MT systems and prompting conditions, sentence-level scores from chrF++, COMET-DA, and XCOMET-XL show a consistent negative association with indexical density under the original informal reference. In a reference-perturbation design that holds MT outputs constant while replacing the informal reference with a standardized Batua version, scores increase systematically, particularly for high-density items, and the density{--}score association weakens. These results provide controlled evidence that evaluation outcomes in this setting depend in part on reference register configuration. In minority-language and informal domains, reference design choices may influence how translation quality is measured and interpreted."
}