Abstract
Building a natural language processing (NLP) model can be challenging for end-users such as analysts, journalists, investigators, etc., especially given that they will likely apply existing tools out of the box. In this article, we take a closer look at how two complementary approaches, a state-of-the-art human-in-the-loop (HITL) tool and a generative language model (GPT-3) perform out of the box, that is, without fine-tuning. Concretely, we compare these approaches when end-users with little technical background are given pattern extraction tasks from text. We discover that the HITL tool performs with higher precision, while GPT-3 requires some level of engineering in its input prompts as well as post-processing on its output before it can achieve comparable results. Future work in this space should look further into the advantages and disadvantages of the two approaches, HITL and generative language model, as well as into ways to optimally combine them.- Anthology ID:
- 2022.dash-1.7
- Volume:
- Proceedings of the Fourth Workshop on Data Science with Human-in-the-Loop (Language Advances)
- Month:
- December
- Year:
- 2022
- Address:
- Abu Dhabi, United Arab Emirates (Hybrid)
- Venue:
- DaSH
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 43–50
- Language:
- URL:
- https://aclanthology.org/2022.dash-1.7
- DOI:
- Cite (ACL):
- Maeda Hanafi, Yannis Katsis, Ishan Jindal, and Lucian Popa. 2022. A Comparative Analysis between Human-in-the-loop Systems and Large Language Models for Pattern Extraction Tasks. In Proceedings of the Fourth Workshop on Data Science with Human-in-the-Loop (Language Advances), pages 43–50, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
- Cite (Informal):
- A Comparative Analysis between Human-in-the-loop Systems and Large Language Models for Pattern Extraction Tasks (Hanafi et al., DaSH 2022)
- PDF:
- https://preview.aclanthology.org/ingestion-script-update/2022.dash-1.7.pdf