Runqi Liu
2025
Multilingual Retrieval Augmented Generation for Culturally-Sensitive Tasks: A Benchmark for Cross-lingual Robustness
Bryan Li
|
Fiona Luo
|
Samar Haider
|
Adwait Agashe
|
Siyu Li
|
Runqi Liu
|
Miranda Muqing Miao
|
Shriya Ramakrishnan
|
Yuan Yuan
|
Chris Callison-Burch
Findings of the Association for Computational Linguistics: ACL 2025
The paradigm of retrieval-augmented generated (RAG) helps mitigate hallucinations of large language models (LLMs). However, RAG also introduces biases contained within the retrieved documents. These biases can be amplified in scenarios which are multilingual and culturally-sensitive, such as territorial disputes. We thus introduce BordIRLines, a dataset of territorial disputes paired with retrieved Wikipedia documents, across 49 languages. We evaluate the cross-lingual robustness of this RAG setting by formalizing several modes for multilingual retrieval. Our experiments on several LLMs show that incorporating perspectives from diverse languages can in fact improve robustness; retrieving multilingual documents best improves response consistency and decreases geopolitical bias over RAG with purely in-language documents. We also consider how RAG responses utilize presented documents, finding a much wider variance in the linguistic distribution of response citations, when querying in low-resource languages. Our further analyses investigate the various aspects of a cross-lingual RAG pipeline, from retrieval to document contents. We release our benchmark to support continued research towards equitable information access across languages, at https://huggingface.co/datasets/borderlines/bordirlines.
Search
Fix author
Co-authors
- Adwait Agashe 1
- Chris Callison-Burch 1
- Samar Haider 1
- Bryan Li 1
- Siyu Li 1
- show all...