Noah Flynn
2026
DREAM: Deep Research Evaluation with Agentic Metrics
Elad Ben Avraham | ChangHao Li | Ron Dorfman | Roy Ganz | Oren Nuriel | Amir Dudai | Aviad Aberdam | Noah Flynn | Elman Mansimov | Aditya Kalyanpur | Ron Litman
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Elad Ben Avraham | ChangHao Li | Ron Dorfman | Roy Ganz | Oren Nuriel | Amir Dudai | Aviad Aberdam | Noah Flynn | Elman Mansimov | Aditya Kalyanpur | Ron Litman
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Deep Research Agents generate analyst-grade reports, yet evaluating them remains challenging due to the absence of a single ground truth and the multidimensional nature of research quality. Recent benchmarks propose distinct methodologies, yet they suffer from the Mirage of Synthesis, where strong surface-level fluency and citation alignment can obscure underlying factual and reasoning defects. We characterize this gap by introducing a taxonomy across four verticals that exposes a critical capability mismatch: static evaluators inherently lack the tool-use capabilities required to assess temporal validity and factual correctness. To address this, we propose **DREAM** (Deep Research Evaluation with Agentic Metrics), a framework that instantiates the principle of capability parity by making evaluation itself agentic. DREAM structures assessment through an evaluation protocol combining query-agnostic metrics with adaptive metrics generated by a tool-calling agent, enabling temporally aware coverage, grounded verification, and systematic reasoning probes. Controlled evaluations demonstrate DREAM is significantly more sensitive to factual and temporal decay than existing benchmarks, offering a scalable, reference-free evaluation paradigm.
2024
SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages
Holy Lovenia | Rahmad Mahendra | Salsabil Maulana Akbar | Lester James V. Miranda | Jennifer Santoso | Elyanah Aco | Akhdan Fadhilah | Jonibek Mansurov | Joseph Marvin Imperial | Onno P. Kampman | Joel Ruben Antony Moniz | Muhammad Ravi Shulthan Habibi | Frederikus Hudi | Railey Montalan | Ryan Ignatius | Joanito Agili Lopo | William Nixon | Börje F. Karlsson | James Jaya | Ryandito Diandaru | Yuze Gao | Patrick Amadeus | Bin Wang | Jan Christian Blaise Cruz | Chenxi Whitehouse | Ivan Halim Parmonangan | Maria Khelli | Wenyu Zhang | Lucky Susanto | Reynard Adha Ryanda | Sonny Lazuardi Hermawan | Dan John Velasco | Muhammad Dehan Al Kautsar | Willy Fitra Hendria | Yasmin Moslem | Noah Flynn | Muhammad Farid Adilazuarda | Haochen Li | Johanes Lee | R. Damanhuri | Shuo Sun | Muhammad Reza Qorib | Amirbek Djanibekov | Wei Qi Leong | Quyet V. Do | Niklas Muennighoff | Tanrada Pansuwan | Ilham Firdausi Putra | Yan Xu | Tai Ngee Chia | Ayu Purwarianti | Sebastian Ruder | William Tjhi | Peerat Limkonchotiwat | Alham Fikri Aji | Sedrick Keh | Genta Indra Winata | Ruochen Zhang | Fajri Koto | Zheng-Xin Yong | Samuel Cahyawijaya
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
Holy Lovenia | Rahmad Mahendra | Salsabil Maulana Akbar | Lester James V. Miranda | Jennifer Santoso | Elyanah Aco | Akhdan Fadhilah | Jonibek Mansurov | Joseph Marvin Imperial | Onno P. Kampman | Joel Ruben Antony Moniz | Muhammad Ravi Shulthan Habibi | Frederikus Hudi | Railey Montalan | Ryan Ignatius | Joanito Agili Lopo | William Nixon | Börje F. Karlsson | James Jaya | Ryandito Diandaru | Yuze Gao | Patrick Amadeus | Bin Wang | Jan Christian Blaise Cruz | Chenxi Whitehouse | Ivan Halim Parmonangan | Maria Khelli | Wenyu Zhang | Lucky Susanto | Reynard Adha Ryanda | Sonny Lazuardi Hermawan | Dan John Velasco | Muhammad Dehan Al Kautsar | Willy Fitra Hendria | Yasmin Moslem | Noah Flynn | Muhammad Farid Adilazuarda | Haochen Li | Johanes Lee | R. Damanhuri | Shuo Sun | Muhammad Reza Qorib | Amirbek Djanibekov | Wei Qi Leong | Quyet V. Do | Niklas Muennighoff | Tanrada Pansuwan | Ilham Firdausi Putra | Yan Xu | Tai Ngee Chia | Ayu Purwarianti | Sebastian Ruder | William Tjhi | Peerat Limkonchotiwat | Alham Fikri Aji | Sedrick Keh | Genta Indra Winata | Ruochen Zhang | Fajri Koto | Zheng-Xin Yong | Samuel Cahyawijaya
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
Southeast Asia (SEA) is a region rich in linguistic diversity and cultural variety, with over 1,300 indigenous languages and a population of 671 million people. However, prevailing AI models suffer from a significant lack of representation of texts, images, and audio datasets from SEA, compromising the quality of AI models for SEA languages. Evaluating models for SEA languages is challenging due to the scarcity of high-quality datasets, compounded by the dominance of English training data, raising concerns about potential cultural misrepresentation. To address these challenges, through a collaborative movement, we introduce SEACrowd, a comprehensive resource center that fills the resource gap by providing standardized corpora in nearly 1,000 SEA languages across three modalities. Through our SEACrowd benchmarks, we assess the quality of AI models on 36 indigenous languages across 13 tasks, offering valuable insights into the current AI landscape in SEA. Furthermore, we propose strategies to facilitate greater AI advancements, maximizing potential utility and resource equity for the future of AI in Southeast Asia.
Search
Fix author
Co-authors
- Aviad Aberdam 1
- Elyanah Aco 1
- Muhammad Farid Adilazuarda 1
- Alham Fikri Aji 1
- Salsabil Maulana Akbar 1
- Muhammad Dehan Al Kautsar 1
- Patrick Amadeus 1
- Elad Ben Avraham 1
- Samuel Cahyawijaya 1
- Tai Ngee Chia 1
- Jan Christian Blaise Cruz 1
- R. Damanhuri 1
- Ryandito Diandaru 1
- Amirbek Djanibekov 1
- Quyet V. Do 1
- Ron Dorfman 1
- Amir Dudai 1
- Akhdan Fadhilah 1
- Roy Ganz 1
- Yuze Gao 1
- Muhammad Ravi Shulthan Habibi 1
- Willy Fitra Hendria 1
- Sonny Lazuardi Hermawan 1
- Frederikus Hudi 1
- Ryan Ignatius 1
- Joseph Marvin Imperial 1
- James Jaya 1
- Aditya Kalyanpur 1
- Onno P. Kampman 1
- Börje F. Karlsson 1
- Sedrick Keh 1
- Maria Khelli 1
- Fajri Koto 1
- Johanes Lee 1
- Wei Qi Leong 1
- ChangHao Li 1
- Haochen Li 1
- Peerat Limkonchotiwat 1
- Ron Litman 1
- Joanito Agili Lopo 1
- Holy Lovenia 1
- Rahmad Mahendra 1
- Elman Mansimov 1
- Jonibek Mansurov 1
- Lester James Validad Miranda 1
- Joel Ruben Antony Moniz 1
- Jann Railey Montalan 1
- Yasmin Moslem 1
- Niklas Muennighoff 1
- William Nixon 1
- Oren Nuriel 1
- Tanrada Pansuwan 1
- Ivan Halim Parmonangan 1
- Ayu Purwarianti 1
- Ilham Firdausi Putra 1
- Muhammad Reza Qorib 1
- Sebastian Ruder 1
- Reynard Adha Ryanda 1
- Jennifer Santoso 1
- Shuo Sun 1
- Lucky Susanto 1
- William Tjhi 1
- Dan John Velasco 1
- Bin Wang 1
- Chenxi Whitehouse 1
- Genta Indra Winata 1
- Yan Xu 1
- Zheng Xin Yong 1
- Wenyu Zhang 1
- Ruochen Zhang 1