Precise measurement method of a speech translation system’s capability with a paired comparison method between the system and humans
Fumiaki Sugaya, Keiji Yasuda, Toshiyuki Takezawa, Seiichi Yamamoto
Abstract
The main goal of the present paper is to propose new schemes for the overall evaluation of a speech translation system. These schemes are expected to support and improve the design of the target application system, and precisely determine its performance. Experiments are conducted on the Japanese-to-English speech translation system ATR-MATRIX, which was developed at ATR Interpreting Telecommunications Research Laboratories. In the proposed schemes, the system’s translations are compared with those of a native Japanese taking the Test of English for International Communication (TOEIC), which is used as a measure of one’s speech translation capability. Subjective and automatic comparisons are made and the results are compared. A regression analysis on the subjective results shows that the speech translation capability of ATR-MATRIX matches a Japanese person scoring around 500 on the TOEIC. The automatic comparisons also show promising results.- Anthology ID:
- 2001.mtsummit-papers.62
- Volume:
- Proceedings of Machine Translation Summit VIII
- Month:
- September 18-22
- Year:
- 2001
- Address:
- Santiago de Compostela, Spain
- Editor:
- Bente Maegaard
- Venue:
- MTSummit
- SIG:
- Publisher:
- Note:
- Pages:
- Language:
- URL:
- https://aclanthology.org/2001.mtsummit-papers.62
- DOI:
- Cite (ACL):
- Fumiaki Sugaya, Keiji Yasuda, Toshiyuki Takezawa, and Seiichi Yamamoto. 2001. Precise measurement method of a speech translation system’s capability with a paired comparison method between the system and humans. In Proceedings of Machine Translation Summit VIII, Santiago de Compostela, Spain.
- Cite (Informal):
- Precise measurement method of a speech translation system’s capability with a paired comparison method between the system and humans (Sugaya et al., MTSummit 2001)
- PDF:
- https://preview.aclanthology.org/landing_page/2001.mtsummit-papers.62.pdf