First-AID: the first Annotation Interface for grounded Dialogues

Stefano Menini, Daniel Russo, Alessio Palmero Aprosio, Marco Guerini


Abstract
The swift advancement of Large Language Models (LLMs) has led to their widespread use across various tasks and domains, demonstrating remarkable generalization capabilities. However, achieving optimal performance in specialized tasks often requires fine-tuning LLMs with task-specific resources. The creation of high-quality, human-annotated datasets for this purpose is challenging due to financial constraints and the limited availability of human experts. To address these limitations, we propose First-AID, a novel human-in-the-loop (HITL) data collection framework for the knowledge-driven generation of synthetic dialogues using LLM prompting. In particular, our framework implements different strategies of data collection that require different user intervention during dialogue generation to reduce post-editing efforts and enhance the quality of generated dialogues. We also evaluated First-AID on misinformation and hate countering dialogues collection, demonstrating (1) its potential for efficient and high-quality data generation and (2) its adaptability to different practical constraints thanks to the three data collection strategies.
Anthology ID:
2025.acl-demo.54
Volume:
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)
Month:
July
Year:
2025
Address:
Vienna, Austria
Editors:
Pushkar Mishra, Smaranda Muresan, Tao Yu
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
563–571
Language:
URL:
https://preview.aclanthology.org/ingestion-acl-25/2025.acl-demo.54/
DOI:
Bibkey:
Cite (ACL):
Stefano Menini, Daniel Russo, Alessio Palmero Aprosio, and Marco Guerini. 2025. First-AID: the first Annotation Interface for grounded Dialogues. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 563–571, Vienna, Austria. Association for Computational Linguistics.
Cite (Informal):
First-AID: the first Annotation Interface for grounded Dialogues (Menini et al., ACL 2025)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-acl-25/2025.acl-demo.54.pdf
Copyright agreement:
 2025.acl-demo.54.copyright_agreement.pdf