Abstract
In unsupervised grammar induction, data likelihood is known to be only weakly correlated with parsing accuracy, especially at convergence after multiple runs. In order to find a better indicator for quality of induced grammars, this paper correlates several linguistically- and psycholinguistically-motivated predictors to parsing accuracy on a large multilingual grammar induction evaluation data set. Results show that variance of average surprisal (VAS) better correlates with parsing accuracy than data likelihood and that using VAS instead of data likelihood for model selection provides a significant accuracy boost. Further evidence shows VAS to be a better candidate than data likelihood for predicting word order typology classification. Analyses show that VAS seems to separate content words from function words in natural language grammars, and to better arrange words with different frequencies into separate classes that are more consistent with linguistic theory.- Anthology ID:
- P19-1235
- Volume:
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Month:
- July
- Year:
- 2019
- Address:
- Florence, Italy
- Editors:
- Anna Korhonen, David Traum, Lluís Màrquez
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 2453–2463
- Language:
- URL:
- https://aclanthology.org/P19-1235
- DOI:
- 10.18653/v1/P19-1235
- Cite (ACL):
- Lifeng Jin and William Schuler. 2019. Variance of Average Surprisal: A Better Predictor for Quality of Grammar from Unsupervised PCFG Induction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2453–2463, Florence, Italy. Association for Computational Linguistics.
- Cite (Informal):
- Variance of Average Surprisal: A Better Predictor for Quality of Grammar from Unsupervised PCFG Induction (Jin & Schuler, ACL 2019)
- PDF:
- https://preview.aclanthology.org/naacl24-info/P19-1235.pdf
- Data
- Penn Treebank