Jason Katz-Brown

Also published as: Jason Brown


2025

pdf bib
The Benefits of Being Uncertain: Perplexity as a Signal for Naturalness in Multilingual Machine Translation
Timothy Pistotti | Michael J. Witbrock | Dr Padriac Amato Tahua O’Leary | Jason Brown
Proceedings of the 2nd Workshop on Uncertainty-Aware NLP (UncertaiNLP 2025)

Model-internal uncertainty metrics like perplexity potentially offer low-cost signals for Machine Translation Quality Estimation (TQE). This paper analyses perplexity in the No Language Left Behind (NLLB) multilingual model. We quantify a significant model-human perplexity gap, where the model is consistently more confident in its own, often literal, machine-generated translation than in diverse, high-quality human versions. We then demonstrate that the utility of perplexity as a TQE signal is highly context-dependent, being strongest for low-resource pairs. Finally, we present an illustrative case study where a flawed translation is refined by providing potentially useful information in a targeted prompt, simulating a knowledge-based repair. We show that as the translation’s quality and naturalness improve (a +0.15 COMET score increase), its perplexity also increases, challenging the simple assumption that lower perplexity indicates higher quality and motivating a more nuanced view of uncertainty as signalling a text’s departure from rigid translationese.

2013

pdf bib
Rhythm, Metrics, and the Link to Phonology
Jason Brown | Sam Mandal
Proceedings of the Australasian Language Technology Association Workshop 2013 (ALTA 2013)

2011

pdf bib
Training a Parser for Machine Translation Reordering
Jason Katz-Brown | Slav Petrov | Ryan McDonald | Franz Och | David Talbot | Hiroshi Ichikawa | Masakazu Seno | Hideto Kazawa
Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing

pdf bib
Training dependency parsers by jointly optimizing multiple objectives
Keith Hall | Ryan McDonald | Jason Katz-Brown | Michael Ringgaard
Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing

pdf bib
A Lightweight Evaluation Framework for Machine Translation Reordering
David Talbot | Hideto Kazawa | Hiroshi Ichikawa | Jason Katz-Brown | Masakazu Seno | Franz Och
Proceedings of the Sixth Workshop on Statistical Machine Translation