A new evaluation method for speech translation systems and a case study on ATR-MATRIX from Japanese to English
Toshiyuki Takezawa, Fumiaki Sugaya, Akio Yokoo, Seiichi Yamamoto
Abstract
ATR-MATRIX is a multi-lingual speech-to-speech translation system designed to facilitate communications between two parties of different languages engaged in a spontaneous conversation in a travel arrangement domain. In this paper, we propose a new evaluation method for speech translation systems. Our current focus is on measuring the robustness of a language translation sub-system, with quick calculation and low cost. Therefore, we calculate the difference between the translation output from transcription texts and the translation output from input speech by a dynamic programming method. We present the first trial experiment of this method applied to our Japanese-to-English speech translation system. We also provide related discussions on such points as error analysis and the relationship between the proposed method and translation quality evaluation manually done by humans.- Anthology ID:
- 1999.mtsummit-1.44
- Volume:
- Proceedings of Machine Translation Summit VII
- Month:
- September 13-17
- Year:
- 1999
- Address:
- Singapore, Singapore
- Venue:
- MTSummit
- SIG:
- Publisher:
- Note:
- Pages:
- 299–307
- Language:
- URL:
- https://aclanthology.org/1999.mtsummit-1.44
- DOI:
- Cite (ACL):
- Toshiyuki Takezawa, Fumiaki Sugaya, Akio Yokoo, and Seiichi Yamamoto. 1999. A new evaluation method for speech translation systems and a case study on ATR-MATRIX from Japanese to English. In Proceedings of Machine Translation Summit VII, pages 299–307, Singapore, Singapore.
- Cite (Informal):
- A new evaluation method for speech translation systems and a case study on ATR-MATRIX from Japanese to English (Takezawa et al., MTSummit 1999)
- PDF:
- https://preview.aclanthology.org/ingestion-script-update/1999.mtsummit-1.44.pdf