@inproceedings{wu-etal-2023-designing,
    title = "Designing, Evaluating, and Learning from Humans Interacting with {NLP} Models",
    author = "Wu, Tongshuang  and
      Yang, Diyi  and
      Santy, Sebastin",
    editor = "Zhang, Qi  and
      Sajjad, Hassan",
    booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts",
    month = dec,
    year = "2023",
    address = "Singapore",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-yuqing-yang-usc/2023.emnlp-tutorial.3/",
    doi = "10.18653/v1/2023.emnlp-tutorial.3",
    pages = "13--18",
    abstract = "The rapid advancement of natural language processing (NLP) research has led to various applications spanning a wide range of domains that require models to interact with humans {--} e.g., chatbots responding to human inquiries, machine translation systems assisting human translators, designers prompting Large Language Models for co-creation or prototyping AI-infused applications, etc. In these cases, humans interaction is key to the success of NLP applications; any potential misconceptions or differences might lead to error cascades at the subsequent stages. Such interaction involves a lot of design choices around models, e.g. the sensitivity of interfaces, the impact of design choice and evaluation questions, etc. This tutorial aims to provide a systematic and up-to-date overview of key considerations and effective approaches for studying human-NLP model interactions. Our tutorial will focus specifically on the scenario where end users {--} lay people and domain experts who have access to NLP models but are less familiar with NLP techniques {--} use or collaborate with deployed models. Throughout the tutorial, we will use five case studies (on classifier-assisted decision making, machine-aided translation, dialog systems, and prompting) to cover three major themes: (1) how to conduct human-in-the-loop usability evaluations to ensure that models are capable of interacting with humans; (2) how to design user interfaces (UIs) and interaction mechanisms that provide end users with easy access to NLP models; (3) how to learn and improve NLP models through the human interactions. We will use best practices from HCI to ground our discussion, and will highlight current challenges and future directions."
}