Unlocking Structure Measuring: Introducing PDD, an Automatic Metric for Positional Discourse Coherence

Yinhong Liu, Yixuan Su, Ehsan Shareghi, Nigel Collier

Abstract
Recent large language models (LLMs) have shown remarkable performance in aligning generated text with user intentions across various tasks. When it comes to long-form text generation, there has been a growing interest in generation from a discourse coherence perspective.However, existing lexical or semantic metrics such as BLEU, ROUGE, BertScore cannot effectively capture the discourse coherence.The development of discourse-specific automatic evaluation methods for assessing the output of LLMs warrants greater focus and exploration. In this paper, we present a novel automatic metric designed to quantify the discourse divergence between two long-form articles.Extensive experiments on three datasets from representative domains demonstrate that our metric aligns more closely with human preferences and GPT-4 coherence evaluation, outperforming existing evaluation methods.
Anthology ID:
2024.naacl-short.9
Volume:
Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Kevin Duh, Helena Gomez, Steven Bethard
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
92–100
Language:
URL:
https://aclanthology.org/2024.naacl-short.9
DOI:
Bibkey:
Cite (ACL):
Yinhong Liu, Yixuan Su, Ehsan Shareghi, and Nigel Collier. 2024. Unlocking Structure Measuring: Introducing PDD, an Automatic Metric for Positional Discourse Coherence. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 92–100, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
Unlocking Structure Measuring: Introducing PDD, an Automatic Metric for Positional Discourse Coherence (Liu et al., NAACL 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-checklist/2024.naacl-short.9.pdf
PDF Search