Abstract
Masked language models (MLMs) are pre-trained with a denoising objective that is in a mismatch with the objective of downstream fine-tuning. We propose pragmatic masking and surrogate fine-tuning as two complementing strategies that exploit social cues to drive pre-trained representations toward a broad set of concepts useful for a wide class of social meaning tasks. We test our models on 15 different Twitter datasets for social meaning detection. Our methods achieve 2.34% F1 over a competitive baseline, while outperforming domain-specific language models pre-trained on large datasets. Our methods also excel in few-shot learning: with only 5% of training data (severely few-shot), our methods enable an impressive 68.54% average F1. The methods are also language agnostic, as we show in a zero-shot setting involving six datasets from three different languages.- Anthology ID:
- 2022.wassa-1.14
- Volume:
- Proceedings of the 12th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis
- Month:
- May
- Year:
- 2022
- Address:
- Dublin, Ireland
- Venue:
- WASSA
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 141–156
- Language:
- URL:
- https://aclanthology.org/2022.wassa-1.14
- DOI:
- 10.18653/v1/2022.wassa-1.14
- Cite (ACL):
- Chiyu Zhang and Muhammad Abdul-Mageed. 2022. Improving Social Meaning Detection with Pragmatic Masking and Surrogate Fine-Tuning. In Proceedings of the 12th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis, pages 141–156, Dublin, Ireland. Association for Computational Linguistics.
- Cite (Informal):
- Improving Social Meaning Detection with Pragmatic Masking and Surrogate Fine-Tuning (Zhang & Abdul-Mageed, WASSA 2022)
- PDF:
- https://preview.aclanthology.org/ingestion-script-update/2022.wassa-1.14.pdf
- Code
- chiyuzhang94/pmlm-sft
- Data
- TweetEval