Andrew Cohen


2024

pdf
The ART of LLM Refinement: Ask, Refine, and Trust
Kumar Shridhar | Koustuv Sinha | Andrew Cohen | Tianlu Wang | Ping Yu | Ramakanth Pasunuru | Mrinmaya Sachan | Jason Weston | Asli Celikyilmaz
Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)

Large Language Models (LLMs) have demonstrated remarkable generative abilities, but can they judge the quality of their own generations and self-improve?A popular concept, referred to as *self-refinement*, postulates that LLMs can detect and correct the errors in their generations when asked to do so. However, recent empirical evidence points in the opposite direction, suggesting that LLMs often struggle to accurately identify errors when reasoning is involved. To address this, we propose a reasoning with a refinement strategy called *ART: Ask, Refine, and Trust*, which *asks* necessary questions to decide when an LLM should *refine* its output, and uses it to affirm or deny *trust* in its refinement by ranking the refinement and the initial prediction. On two multistep reasoning tasks of mathematical word problems (GSM8K) and question answering (StrategyQA), *ART* achieves a performance gain of +5 points over self-refinement baselines, while using a much smaller model as the decision maker. We believe that *ART* with smaller models, making refinement decisions can be a cost-effective alternative to fine-tuning LLMs.

1995

pdf
Developing a Nonsymbolic Phonetic Notation for Speech Synthesis
Andrew Cohen
Computational Linguistics, Volume 21, Number 4, December 1995