Abstract
The rapid advancement of natural language processing (NLP) research has led to various applications spanning a wide range of domains that require models to interact with humans – e.g., chatbots responding to human inquiries, machine translation systems assisting human translators, designers prompting Large Language Models for co-creation or prototyping AI-infused applications, etc. In these cases, humans interaction is key to the success of NLP applications; any potential misconceptions or differences might lead to error cascades at the subsequent stages. Such interaction involves a lot of design choices around models, e.g. the sensitivity of interfaces, the impact of design choice and evaluation questions, etc. This tutorial aims to provide a systematic and up-to-date overview of key considerations and effective approaches for studying human-NLP model interactions. Our tutorial will focus specifically on the scenario where end users – lay people and domain experts who have access to NLP models but are less familiar with NLP techniques – use or collaborate with deployed models. Throughout the tutorial, we will use five case studies (on classifier-assisted decision making, machine-aided translation, dialog systems, and prompting) to cover three major themes: (1) how to conduct human-in-the-loop usability evaluations to ensure that models are capable of interacting with humans; (2) how to design user interfaces (UIs) and interaction mechanisms that provide end users with easy access to NLP models; (3) how to learn and improve NLP models through the human interactions. We will use best practices from HCI to ground our discussion, and will highlight current challenges and future directions.- Anthology ID:
- 2023.emnlp-tutorial.3
- Volume:
- Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Qi Zhang, Hassan Sajjad
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 13–18
- Language:
- URL:
- https://aclanthology.org/2023.emnlp-tutorial.3
- DOI:
- 10.18653/v1/2023.emnlp-tutorial.3
- Cite (ACL):
- Tongshuang Wu, Diyi Yang, and Sebastin Santy. 2023. Designing, Evaluating, and Learning from Humans Interacting with NLP Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts, pages 13–18, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- Designing, Evaluating, and Learning from Humans Interacting with NLP Models (Wu et al., EMNLP 2023)
- PDF:
- https://preview.aclanthology.org/naacl24-info/2023.emnlp-tutorial.3.pdf