Abstract
Recent developments in the quality and accessibility of large language models have precipitated a surge in user-facing tools for content generation. Motivated by a necessity for human quality control of these systems, we introduce ReportGPT: a pipeline framework for verifiable human-in-the-loop table-to-text generation. ReportGPT is based on a domain specific language, which acts as a proof mechanism for generating verifiable commentary. This allows users to quickly check the relevancy and factuality of model outputs. User selections then become few-shot examples for improving the performance of the pipeline. We configure 3 approaches to our pipeline, and find that usage of language models in ReportGPT’s components trade off precision for more insightful downstream commentary. Furthermore, ReportGPT learns from human feedback in real-time, needing only a few samples to improve performance.- Anthology ID:
- 2024.emnlp-industry.39
- Volume:
- Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track
- Month:
- November
- Year:
- 2024
- Address:
- Miami, Florida, US
- Editors:
- Franck Dernoncourt, Daniel Preoţiuc-Pietro, Anastasia Shimorina
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 529–537
- Language:
- URL:
- https://aclanthology.org/2024.emnlp-industry.39
- DOI:
- 10.18653/v1/2024.emnlp-industry.39
- Cite (ACL):
- Lucas Cecchi and Petr Babkin. 2024. ReportGPT: Human-in-the-loop Verifiable Table-to-Text Generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 529–537, Miami, Florida, US. Association for Computational Linguistics.
- Cite (Informal):
- ReportGPT: Human-in-the-loop Verifiable Table-to-Text Generation (Cecchi & Babkin, EMNLP 2024)
- PDF:
- https://preview.aclanthology.org/dois-2013-emnlp/2024.emnlp-industry.39.pdf