CPS-TaskForge: Generating Collaborative Problem Solving Environments for Diverse Communication Tasks
Nikita Haduong, Irene Wang, Bo-Ru Lu, Prithviraj Ammanabrolu, Noah A. Smith
Abstract
Teams can outperform individuals; could adding AI teammates further bolster performance of teams solving problems collaboratively? Collaborative problem solving (CPS) research commonly studies teams with two agents (human-human or human-AI), but team research literature finds that, for complex tasks, larger teams are more effective. Progress in studying collaboration with more than two agents, through textual records of team interactions, is hindered by a major data challenge: available CPS corpora are predominantly dyadic, and adapting pre-existing CPS tasks to more agents is non-trivial. We address this data challenge by developing a CPS task generator, CPS-TaskForge, that can produce environments for studying CPS under a wide array of conditions, and releasing a CPS task design checklist grounded in the theoretical PISA 2015 CPS framework to help facilitate the development of CPS corpora with more agents. CPS-TaskForge takes the form of a resource management (tower defense) game, and different CPS tasks can be studied by manipulating game design parameters. We conduct a case study with groups of 3–4 humans to validate production of diverse natural language CPS communication in a game instance produced by CPS-TaskForge. We discuss opportunities for advancing research in CPS (both with human-only and human-AI teams) using different task configurations. We release all data and code.- Anthology ID:
- 2024.customnlp4u-1.9
- Volume:
- Proceedings of the 1st Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual (CustomNLP4U)
- Month:
- November
- Year:
- 2024
- Address:
- Miami, Florida, USA
- Editors:
- Sachin Kumar, Vidhisha Balachandran, Chan Young Park, Weijia Shi, Shirley Anugrah Hayati, Yulia Tsvetkov, Noah Smith, Hannaneh Hajishirzi, Dongyeop Kang, David Jurgens
- Venue:
- CustomNLP4U
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 86–112
- Language:
- URL:
- https://preview.aclanthology.org/build-pipeline-with-new-library/2024.customnlp4u-1.9/
- DOI:
- 10.18653/v1/2024.customnlp4u-1.9
- Cite (ACL):
- Nikita Haduong, Irene Wang, Bo-Ru Lu, Prithviraj Ammanabrolu, and Noah A. Smith. 2024. CPS-TaskForge: Generating Collaborative Problem Solving Environments for Diverse Communication Tasks. In Proceedings of the 1st Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual (CustomNLP4U), pages 86–112, Miami, Florida, USA. Association for Computational Linguistics.
- Cite (Informal):
- CPS-TaskForge: Generating Collaborative Problem Solving Environments for Diverse Communication Tasks (Haduong et al., CustomNLP4U 2024)
- PDF:
- https://preview.aclanthology.org/build-pipeline-with-new-library/2024.customnlp4u-1.9.pdf