factgenie: A Framework for Span-based Evaluation of Generated Texts
Zdeněk Kasner, Ondrej Platek, Patricia Schmidtova, Simone Balloccu, Ondrej Dusek
Abstract
We present ‘factgenie‘: a framework for annotating and visualizing word spans in textual model outputs. Annotations can capture various span-based phenomena such as semantic inaccuracies or irrelevant text. With ‘factgenie‘, the annotations can be collected both from human crowdworkers and large language models. Our framework consists of a web interface for data visualization and gathering text annotations, powered by an easily extensible codebase.- Anthology ID:
- 2024.inlg-demos.5
- Volume:
- Proceedings of the 17th International Natural Language Generation Conference: System Demonstrations
- Month:
- September
- Year:
- 2024
- Address:
- Tokyo, Japan
- Editors:
- Saad Mahamood, Nguyen Le Minh, Daphne Ippolito
- Venue:
- INLG
- SIG:
- SIGGEN
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 13–15
- Language:
- URL:
- https://preview.aclanthology.org/jlcl-multiple-ingestion/2024.inlg-demos.5/
- DOI:
- Cite (ACL):
- Zdeněk Kasner, Ondrej Platek, Patricia Schmidtova, Simone Balloccu, and Ondrej Dusek. 2024. factgenie: A Framework for Span-based Evaluation of Generated Texts. In Proceedings of the 17th International Natural Language Generation Conference: System Demonstrations, pages 13–15, Tokyo, Japan. Association for Computational Linguistics.
- Cite (Informal):
- factgenie: A Framework for Span-based Evaluation of Generated Texts (Kasner et al., INLG 2024)
- PDF:
- https://preview.aclanthology.org/jlcl-multiple-ingestion/2024.inlg-demos.5.pdf