@inproceedings{vieira-etal-2026-alba,
    title = "{ALBA}: A {E}uropean {P}ortuguese Benchmark for Evaluating Language and Linguistic Dimensions in Generative {LLM}s",
    author = "Vieira, In{\^e}s  and
      Calvo, In{\^e}s  and
      Paulo, Iago  and
      Furtado, James  and
      Ferreira, Rafael  and
      Tavares, Diogo  and
      Gl{\'o}ria-Silva, Diogo  and
      Semedo, David  and
      Magalh{\~a}es, Jo{\~a}o",
    editor = "Souza, Marlo  and
      de-Dios-Flores, Iria  and
      Santos, Diana  and
      Freitas, Larissa  and
      Souza, Jackson Wilke da Cruz  and
      Ribeiro, Eug{\'e}nio",
    booktitle = "Proceedings of the 17th International Conference on Computational Processing of {P}ortuguese ({PROPOR} 2026) - Vol. 1",
    month = apr,
    year = "2026",
    address = "Salvador, Brazil",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-dnd/2026.propor-1.69/",
    pages = "697--707",
    ISBN = "979-8-89176-387-6",
    abstract = "As Large Language Models (LLMs) expand across multilingual domains, evaluating their performance in under-represented languages becomes increasingly important. European Portuguese (pt-PT) is particularly affected, as existing training data and benchmarks are mainly in Brazilian Portuguese (pt-BR). To address this, we introduce ALBA, a linguistically grounded benchmark designed from the ground up to assess LLM proficiency in linguistic-related tasks in pt-PT across eight linguistic dimensions, including Language Variety, Culture-bound Semantics, Discourse Analysis, Word Plays, Syntax, Morphology, Lexicology, and Phonetics and Phonology. ALBA is manually constructed by language experts and paired with an LLM-as-a-judge framework for scalable evaluation of pt-PT generated language. Experiments on a diverse set of models reveal performance variability across linguistic dimensions, highlighting the need for comprehensive, variety-sensitive benchmarks that support further development of tools in pt-PT."
}