On the Distribution of Lexical Features at Multiple Levels of Analysis
Fatemeh Almodaresi, Lyle Ungar, Vivek Kulkarni, Mohsen Zakeri, Salvatore Giorgi, H. Andrew Schwartz
Abstract
Natural language processing has increasingly moved from modeling documents and words toward studying the people behind the language. This move to working with data at the user or community level has presented the field with different characteristics of linguistic data. In this paper, we empirically characterize various lexical distributions at different levels of analysis, showing that, while most features are decidedly sparse and non-normal at the message-level (as with traditional NLP), they follow the central limit theorem to become much more Log-normal or even Normal at the user- and county-levels. Finally, we demonstrate that modeling lexical features for the correct level of analysis leads to marked improvements in common social scientific prediction tasks.- Anthology ID:
- P17-2013
- Volume:
- Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
- Month:
- July
- Year:
- 2017
- Address:
- Vancouver, Canada
- Editors:
- Regina Barzilay, Min-Yen Kan
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 79–84
- Language:
- URL:
- https://aclanthology.org/P17-2013
- DOI:
- 10.18653/v1/P17-2013
- Cite (ACL):
- Fatemeh Almodaresi, Lyle Ungar, Vivek Kulkarni, Mohsen Zakeri, Salvatore Giorgi, and H. Andrew Schwartz. 2017. On the Distribution of Lexical Features at Multiple Levels of Analysis. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 79–84, Vancouver, Canada. Association for Computational Linguistics.
- Cite (Informal):
- On the Distribution of Lexical Features at Multiple Levels of Analysis (Almodaresi et al., ACL 2017)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/P17-2013.pdf