Barriers to Effective Evaluation of Simultaneous Interpretation

Shira Wein, Te I, Colin Cherry, Juraj Juraska, Dirk Padfield, Wolfgang Macherey


Abstract
Simultaneous interpretation is an especially challenging form of translation because it requires converting speech from one language to another in real-time. Though prior work has relied on out-of-the-box machine translation metrics to evaluate interpretation data, we hypothesize that strategies common in high-quality human interpretations, such as summarization, may not be handled well by standard machine translation metrics. In this work, we examine both qualitatively and quantitatively four potential barriers to evaluation of interpretation: disfluency, summarization, paraphrasing, and segmentation. Our experiments reveal that, while some machine translation metrics correlate fairly well with human judgments of interpretation quality, much work is still needed to account for strategies of interpretation during evaluation. As a first step to address this, we develop a fine-tuned model for interpretation evaluation, and achieve better correlation with human judgments than the state-of-the-art machine translation metrics.
Anthology ID:
2024.findings-eacl.15
Volume:
Findings of the Association for Computational Linguistics: EACL 2024
Month:
March
Year:
2024
Address:
St. Julian’s, Malta
Editors:
Yvette Graham, Matthew Purver
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
209–219
Language:
URL:
https://aclanthology.org/2024.findings-eacl.15
DOI:
Bibkey:
Cite (ACL):
Shira Wein, Te I, Colin Cherry, Juraj Juraska, Dirk Padfield, and Wolfgang Macherey. 2024. Barriers to Effective Evaluation of Simultaneous Interpretation. In Findings of the Association for Computational Linguistics: EACL 2024, pages 209–219, St. Julian’s, Malta. Association for Computational Linguistics.
Cite (Informal):
Barriers to Effective Evaluation of Simultaneous Interpretation (Wein et al., Findings 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.findings-eacl.15.pdf
Video:
 https://aclanthology.org/2024.findings-eacl.15.mp4