Evaluation of a simultaneous interpretation system and analysis of speech log for user experience assessment

Akiko Sakamoto, Kazuhiko Abe, Kazuo Sumita, Satoshi Kamatani


Abstract
This paper focuses on the user experience (UX) of a simultaneous interpretation system for face-to-face conversation between two users. To assess the UX of the system, we first made a transcript of the speech of users recorded during a task-based evaluation experiment and then analyzed user speech from the viewpoint of UX. In a task-based evaluation experiment, 44 tasks out of 45 tasks were solved. The solved task ratio was 97.8%. This indicates that the system can effectively provide interpretation to enable users to solve tasks. However, we found that users repeated speech due to errors in automatic speech recognition (ASR) or machine translation (MT). Users repeated clauses 1.8 times on average. Users seemed to repeat themselves until they received a response from their partner users. In addition, we found that after approximately 3.6 repetitions, users would change their words to avoid errors in ASR or MT and to evoke a response from their partner users.
Anthology ID:
2013.iwslt-papers.18
Volume:
Proceedings of the 10th International Workshop on Spoken Language Translation: Papers
Month:
December 5-6
Year:
2013
Address:
Heidelberg, Germany
Venue:
IWSLT
SIG:
SIGSLT
Publisher:
Note:
Pages:
Language:
URL:
https://aclanthology.org/2013.iwslt-papers.18
DOI:
Bibkey:
Cite (ACL):
Akiko Sakamoto, Kazuhiko Abe, Kazuo Sumita, and Satoshi Kamatani. 2013. Evaluation of a simultaneous interpretation system and analysis of speech log for user experience assessment. In Proceedings of the 10th International Workshop on Spoken Language Translation: Papers, Heidelberg, Germany.
Cite (Informal):
Evaluation of a simultaneous interpretation system and analysis of speech log for user experience assessment (Sakamoto et al., IWSLT 2013)
Copy Citation:
PDF:
https://preview.aclanthology.org/emnlp-22-attachments/2013.iwslt-papers.18.pdf