Abstract
In this paper we evaluate the utility of large language models (LLMs) for translation of text with markup in which the most important and challenging aspect is to correctly transfer markup tags while ensuring that the content, both, inside and outside tags is correctly translated. While LLMs have been shown to be effective for plain text translation, their effectiveness for structured document translation is not well understood. To this end, we experiment with BLOOM and BLOOMZ, which are open-source multilingual LLMs, using zero, one and few-shot prompting, and compare with a domain-specific in-house NMT system using a detag-and-project approach for markup tags. We observe that LLMs with in-context learning exhibit poorer translation quality compared to the domain-specific NMT system, however, they are effective in transferring markup tags, especially the large BLOOM model (176 billion parameters). This is further confirmed by our human evaluation which also reveals the types of errors of the different tag transfer techniques. While LLM-based approaches come with the risk of losing, hallucinating and corrupting tags, they excel at placing them correctly in the translation.- Anthology ID:
- 2023.mtsummit-research.13
- Volume:
- Proceedings of Machine Translation Summit XIX, Vol. 1: Research Track
- Month:
- September
- Year:
- 2023
- Address:
- Macau SAR, China
- Editors:
- Masao Utiyama, Rui Wang
- Venue:
- MTSummit
- SIG:
- Publisher:
- Asia-Pacific Association for Machine Translation
- Note:
- Pages:
- 148–159
- Language:
- URL:
- https://aclanthology.org/2023.mtsummit-research.13
- DOI:
- Cite (ACL):
- Raj Dabre, Bianka Buschbeck, Miriam Exel, and Hideki Tanaka. 2023. A Study on the Effectiveness of Large Language Models for Translation with Markup. In Proceedings of Machine Translation Summit XIX, Vol. 1: Research Track, pages 148–159, Macau SAR, China. Asia-Pacific Association for Machine Translation.
- Cite (Informal):
- A Study on the Effectiveness of Large Language Models for Translation with Markup (Dabre et al., MTSummit 2023)
- PDF:
- https://preview.aclanthology.org/ijclclp-past-ingestion/2023.mtsummit-research.13.pdf