X-PARADE: Cross-Lingual Textual Entailment and Information Divergence across Paragraphs

Juan Rodriguez, Katrin Erk, Greg Durrett

Abstract
Understanding when two pieces of text convey the same information is a goal touching many subproblems in NLP, including textual entailment and fact-checking. This problem becomes more complex when those two pieces of text are in different languages. Here, we introduce X-PARADE (Cross-lingual Paragraph-level Analysis of Divergences and Entailments), the first cross-lingual dataset of paragraph-level information divergences. Annotators label a paragraph in a target language at the span level and evaluate it with respect to a corresponding paragraph in a source language, indicating whether a given piece of information is the same, new, or new but can be inferred. This last notion establishes a link with cross-language NLI. Aligned paragraphs are sourced from Wikipedia pages in different languages, reflecting real information divergences observed in the wild. Armed with our dataset, we investigate a diverse set of approaches for this problem, including classic token alignment from machine translation, textual entailment methods that localize their decisions, and prompting LLMs. Our results show that these methods vary in their capability to handle inferable information, but they all fall short of human performance.
Anthology ID:
2024.naacl-long.66
Volume:
Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Kevin Duh, Helena Gomez, Steven Bethard
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1198–1222
Language:
URL:
https://aclanthology.org/2024.naacl-long.66
DOI:
10.18653/v1/2024.naacl-long.66
Bibkey:
Cite (ACL):
Juan Rodriguez, Katrin Erk, and Greg Durrett. 2024. X-PARADE: Cross-Lingual Textual Entailment and Information Divergence across Paragraphs. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1198–1222, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
X-PARADE: Cross-Lingual Textual Entailment and Information Divergence across Paragraphs (Rodriguez et al., NAACL 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/landing_page/2024.naacl-long.66.pdf
PDF Search