@inproceedings{elnokrashy-kocmi-2023-ebleu,
    title = "e{BLEU}: Unexpectedly Good Machine Translation Evaluation Using Simple Word Embeddings",
    author = "ElNokrashy, Muhammad  and
      Kocmi, Tom",
    editor = "Koehn, Philipp  and
      Haddow, Barry  and
      Kocmi, Tom  and
      Monz, Christof",
    booktitle = "Proceedings of the Eighth Conference on Machine Translation",
    month = dec,
    year = "2023",
    address = "Singapore",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/Ingest-2025-COMPUTEL/2023.wmt-1.61/",
    doi = "10.18653/v1/2023.wmt-1.61",
    pages = "746--750",
    abstract = "We propose eBLEU, a metric inspired by BLEU metric that uses embedding similarities instead of string matches. We introduce meaning diffusion vectors to enable matching n-grams of semantically similar words in a BLEU-like algorithm, using efficient, non-contextual word embeddings like fastText. On WMT23 data, eBLEU beats BLEU and ChrF by around 3.8{\%} system-level score, approaching BERTScore at {\ensuremath{-}}0.9{\%} absolute difference. In WMT22 scenarios, eBLEU outperforms f101spBLEU and ChrF in MQM by 2.2{\%}{\ensuremath{-}}3.6{\%}. Curiously, on MTurk evaluations, eBLEU surpasses past methods by 3.9{\%}{\ensuremath{-}}8.2{\%} (f200spBLEU, COMET-22). eBLEU presents an interesting middle-ground between traditional metrics and pretrained metrics."
}