Would you Rather? A New Benchmark for Learning Machine Alignment with Cultural Values and Social Preferences
Yi Tay, Donovan Ong, Jie Fu, Alvin Chan, Nancy Chen, Anh Tuan Luu, Chris Pal
Abstract
Understanding human preferences, along with cultural and social nuances, lives at the heart of natural language understanding. Concretely, we present a new task and corpus for learning alignments between machine and human preferences. Our newly introduced problem is concerned with predicting the preferable options from two sentences describing scenarios that may involve social and cultural situations. Our problem is framed as a natural language inference task with crowd-sourced preference votes by human players, obtained from a gamified voting platform. We benchmark several state-of-the-art neural models, along with BERT and friends on this task. Our experimental results show that current state-of-the-art NLP models still leave much room for improvement.- Anthology ID:
- 2020.acl-main.477
- Original:
- 2020.acl-main.477v1
- Version 2:
- 2020.acl-main.477v2
- Volume:
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Month:
- July
- Year:
- 2020
- Address:
- Online
- Editors:
- Dan Jurafsky, Joyce Chai, Natalie Schluter, Joel Tetreault
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 5369–5373
- Language:
- URL:
- https://aclanthology.org/2020.acl-main.477
- DOI:
- 10.18653/v1/2020.acl-main.477
- Cite (ACL):
- Yi Tay, Donovan Ong, Jie Fu, Alvin Chan, Nancy Chen, Anh Tuan Luu, and Chris Pal. 2020. Would you Rather? A New Benchmark for Learning Machine Alignment with Cultural Values and Social Preferences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5369–5373, Online. Association for Computational Linguistics.
- Cite (Informal):
- Would you Rather? A New Benchmark for Learning Machine Alignment with Cultural Values and Social Preferences (Tay et al., ACL 2020)
- PDF:
- https://preview.aclanthology.org/landing_page/2020.acl-main.477.pdf