FlowEval: A Consensus-Based Dialogue Evaluation Framework Using Segment Act Flows
Jianqiao Zhao, Yanyang Li, Wanyu Du, Yangfeng Ji, Dong Yu, Michael Lyu, Liwei Wang
Abstract
Despite recent progress in open-domain dialogue evaluation, how to develop automatic metrics remains an open problem. We explore the potential of dialogue evaluation featuring dialog act information, which was hardly explicitly modeled in previous methods. However, defined at the utterance level in general, dialog act is of coarse granularity, as an utterance can contain multiple segments possessing different functions. Hence, we propose segment act, an extension of dialog act from utterance level to segment level, and crowdsource a large-scale dataset for it. To utilize segment act flows, sequences of segment acts, for evaluation, we develop the first consensus-based dialogue evaluation framework, FlowEval. This framework provides a reference-free approach for dialog evaluation by finding pseudo-references. Extensive experiments against strong baselines on three benchmark datasets demonstrate the effectiveness and other desirable characteristics of our FlowEval, pointing out a potential path for better dialogue evaluation.- Anthology ID:
- 2022.emnlp-main.715
- Volume:
- Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
- Month:
- December
- Year:
- 2022
- Address:
- Abu Dhabi, United Arab Emirates
- Editors:
- Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 10469–10483
- Language:
- URL:
- https://preview.aclanthology.org/build-pipeline-with-new-library/2022.emnlp-main.715/
- DOI:
- 10.18653/v1/2022.emnlp-main.715
- Cite (ACL):
- Jianqiao Zhao, Yanyang Li, Wanyu Du, Yangfeng Ji, Dong Yu, Michael Lyu, and Liwei Wang. 2022. FlowEval: A Consensus-Based Dialogue Evaluation Framework Using Segment Act Flows. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10469–10483, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Cite (Informal):
- FlowEval: A Consensus-Based Dialogue Evaluation Framework Using Segment Act Flows (Zhao et al., EMNLP 2022)
- PDF:
- https://preview.aclanthology.org/build-pipeline-with-new-library/2022.emnlp-main.715.pdf