@inproceedings{staiano-etal-2025-italert,
    title = "{ITALERT}: Assessing the Quality of {LLM}s and {NMT} in Translating {I}talian Emergency Response Text",
    author = "Staiano, Maria Carmen  and
      Han, Lifeng  and
      Monti, Johanna  and
      Chiusaroli, Francesca",
    editor = "Bouillon, Pierrette  and
      Gerlach, Johanna  and
      Girletti, Sabrina  and
      Volkart, Lise  and
      Rubino, Raphael  and
      Sennrich, Rico  and
      Farinha, Ana C.  and
      Gaido, Marco  and
      Daems, Joke  and
      Kenny, Dorothy  and
      Moniz, Helena  and
      Szoc, Sara",
    booktitle = "Proceedings of Machine Translation Summit XX: Volume 1",
    month = jun,
    year = "2025",
    address = "Geneva, Switzerland",
    publisher = "European Association for Machine Translation",
    url = "https://preview.aclanthology.org/mtsummit-25-ingestion/2025.mtsummit-1.43/",
    pages = "566--577",
    ISBN = "978-2-9701897-0-1",
    abstract = "This paper presents the outcomes of an initial investigation into the performance of Large Language Models (LLMs) and Neural Machine Translation (NMT) systems in translating high-stakes messages. The research employed a novel bilingual corpus, ITALERT (Italian Emergency Response Text) and applied a human-centric post-editing based metric (HOPE) to assess translation quality systematically. The initial dataset contains eleven texts in Italian and their corresponding English translations, both extracted from the national communication campaign website of the Italian Civil Protection Department. The texts deal with eight crisis scenarios: flooding, earthquake, forest fire, volcanic eruption, tsunami, industrial accident, nuclear risk, and dam failure. The dataset has been carefully compiled to ensure usability and clarity for evaluating machine translation (MT) systems in crisis settings. Our findings show that current LLMs and NMT models, such as ChatGPT (OpenAI{'}s GPT-4o model) and Google MT, face limitations in translating emergency texts, particularly in maintaining the appropriate register, resolving context ambiguities, and managing domain-specific terminology."
}