@inproceedings{niloy-etal-2026-meaning,
    title = "Meaning Over Morphology: A Multi-Metric Benchmark of {LLM}s for {B}angla Dialect Translation",
    author = "Niloy, Soumik Deb  and
      Rahman, Subhey Sadi  and
      E Sobhani, Mahbub  and
      Alam, Md. Golam Rabiul  and
      Sadeque, Farig Yousuf  and
      Hassan, Md. Rezuwan",
    editor = "Anastasopoulos, Antonis  and
      Markantonatou, Stella  and
      Ralli, Angela  and
      Zampieri, Marcos  and
      Bompolas, Stavros  and
      Stamou, Vivian",
    booktitle = "Proceedings of the First Workshop on Dialects in {NLP} {---} A Resource Perspective",
    month = may,
    year = "2026",
    address = "Palma de Mallorca",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/test-year-match/2026.dialres-1.24/",
    doi = "10.63317/3h8y7fs2zbq4",
    pages = "238--255",
    abstract = "Regional dialects of Bangla, such as Sylheti and Chittagonian, pose significant challenges for natural language processing due to their low-resource nature and substantial linguistic variation from standard Bangla. In this work, we present a systematic evaluation of eight open-source LLMs for translating fifteen distinct Bangla dialects into standard Bangla. To achieve this comprehensive coverage, we utilize a combination of established benchmarks and a novel dataset curated from an ongoing regional linguistic project. We assess model performance using a multi-metric framework that combines exact-match and error-rate evaluations such as, Averaged BLEU, WER, and CER with embedding-based semantic metrics including BERTScore, METEOR, and COMET. Additionally, we perform a detailed dialect-level linguistic analysis to identify the deep-seated structural, orthographic, and semantic barriers inherent to dialectal translation. Our study highlights the strengths and limitations of current open-source models, provides empirical insights for future dialect-aware fine-tuning, and contributes a reproducible benchmark for the research community."
}