@inproceedings{pistotti-etal-2025-benefits,
    title = "The Benefits of Being Uncertain: Perplexity as a Signal for Naturalness in Multilingual Machine Translation",
    author = "Pistotti, Timothy  and
      J. Witbrock, Michael  and
      Padriac Amato Tahua O{'}Leary, Dr  and
      Brown, Jason",
    editor = "Noidea, Noidea",
    booktitle = "Proceedings of the 2nd Workshop on Uncertainty-Aware NLP (UncertaiNLP 2025)",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.uncertainlp-main.7/",
    pages = "61--65",
    ISBN = "979-8-89176-349-4",
    abstract = "Model-internal uncertainty metrics like perplexity potentially offer low-cost signals for Machine Translation Quality Estimation (TQE). This paper analyses perplexity in the No Language Left Behind (NLLB) multilingual model. We quantify a significant model-human perplexity gap, where the model is consistently more confident in its own, often literal, machine-generated translation than in diverse, high-quality human versions. We then demonstrate that the utility of perplexity as a TQE signal is highly context-dependent, being strongest for low-resource pairs. Finally, we present an illustrative case study where a flawed translation is refined by providing potentially useful information in a targeted prompt, simulating a knowledge-based repair. We show that as the translation{'}s quality and naturalness improve (a +0.15 COMET score increase), its perplexity also increases, challenging the simple assumption that lower perplexity indicates higher quality and motivating a more nuanced view of uncertainty as signalling a text{'}s departure from rigid translationese."
}