Abstract
Quantitatively measuring the impact-related aspects of scientific, engineering, and technological (SET) innovations is a fundamental problem with broad applications. Traditional citation-based measures for assessing the impact of innovations and related entities do not take into account the content of the publications. This limits their ability to provide rigorous quality-related metrics because they cannot account for the reasons that led to a citation. We present approaches to estimate content-aware bibliometrics to quantitatively measure the scholarly impact of a publication. Our approaches assess the impact of a cited publication by the extent to which the cited publication informs the citing publication. We introduce a new metric, called “Content Informed Index” (CII), that uses the content of the paper as a source of distant-supervision, to quantify how much the cited-node informs the citing-node. We evaluate the weights estimated by our approach on three manually annotated datasets, where the annotations quantify the extent of information in the citation. Particularly, we evaluate how well the ranking imposed by our approach associates with the ranking imposed by the manual annotations. CII achieves up to 103% improvement in performance as compared to the second-best performing approach.- Anthology ID:
- 2021.emnlp-main.488
- Volume:
- Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
- Month:
- November
- Year:
- 2021
- Address:
- Online and Punta Cana, Dominican Republic
- Editors:
- Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 6041–6053
- Language:
- URL:
- https://aclanthology.org/2021.emnlp-main.488
- DOI:
- 10.18653/v1/2021.emnlp-main.488
- Cite (ACL):
- Saurav Manchanda and George Karypis. 2021. Evaluating Scholarly Impact: Towards Content-Aware Bibliometrics. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6041–6053, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
- Cite (Informal):
- Evaluating Scholarly Impact: Towards Content-Aware Bibliometrics (Manchanda & Karypis, EMNLP 2021)
- PDF:
- https://preview.aclanthology.org/naacl24-info/2021.emnlp-main.488.pdf
- Code
- gurdaspuriya/evaluating-scholarly-impact
- Data
- S2ORC